A metallurgist ( as I understood his quals) in another thread somewhere suggested in the purest sense that using current process we weren't actually annealing but stress relieving.
We're still annealing, we just want a "partial" anneal as opposed to a "full" anneal. Yes those are actually correct terms

The reason why we don't heat the brass too hot is because we want to keep the brass in the "recovery" range. There's three phases of annealing, recovery, recrystalization, and grain growth. In the recovery phase, enough heat energy is applied to allow the dislocations to rearrange themselves back into a lower energy state. That's the "stress relieving" that the metalurgist you were refering to was talking about.
If the brass gets too hot, it goes beyond the recovery phase into recrystalization and grain growth. That's when we get a completely new grain structure and becomes "dead soft". Another possibility is called "liquation" which is when you just reach the melting temperature and the grain boundaries become liquid. Also not good.