I'm going to challenge the premise of your question. I don't think it's a given that annealing will automatically help.
I'm in the camp of "don't do [inset reloading task] unless you have a good, specific reason to do it". I don't really enjoy reloading and want to do as little of it as possible. As a result, I skip a lot of steps people say you "have to do". (Sometimes I find out the hard way why I can't skip a certain step). I ask "are these loads good enough to do what I need them to do? If not, what is the easiest thing I can do to make them good enough?". Annealing, for me, has never been the answer to that question.
The only concrete, take it to the bank reason to anneal is that it will help prevent split necks, espeically if you're aggressively forming wildcats. I think you're making a big leap by assuming groups will improve when you anneal. At best, that's going to be situationally dependent. It *may* have desirable effects for *some* types of shooting depending on how you do it. The majority of shooters will never see the difference from behind the gun.