My personal experience is wide ranging, un-scientific and anecdotal at best. My shooting skills and the rifles I use prevent me from proving to what degree what I think is true, is actually true. With that said, if you look at the big truths the question becomes the title of the thread.
Annealing is a must.
The question is how much of the case and how often. This depends on the pressure of the loads and fit in the chamber before firing.
I shoot cases that are more than 100 years old, probably shot 1000's of times in multiple chambers, probably never over 35-40,000 psi. Who knows how many or the last time they were annealed before I got them. I have only split a few.
Compare that with new brass that is only partially or inadequately annealed that split on the first firing. Or "basic" cases that are drawn to length but not formed that split in the final forming process. Many wildcat cartridges need to need annealed a couple of times in the reforming process as well as some being fire formed. A 20% failure is common.
Somewhere between being able to form the case, have it survive between 10-1000 firings, or being able to shrink group sizes .1 MOA, annealing stops being critical, or in my case the shooter and his equipment aren't able to prove the difference.
300 Blackout sub-sonic.
Loaded neck diameter .328", Chamber .335"
This is an interesting study because using the same case and bullet combination, I can vary the peak pressure more than 50,000 psi without changing velocity more than the ES/SD of either combination.
The low pressure low load can be wiped down, re primed and charged, the bullet seated with finger pressure and a light crimp applied. Then ran through an AR multiple times.
Change powders, run the pressure up to 60,000+ and it's a race between the primer pocket and neck that has to be sized .010" each time, to see which gives out first. If the neck has been annealed, it will be the primer pocket.
The high pressure round has a much higher starting pressure, a cleaner release from the neck and most likely a much straighter start down the bore after a .250" jump as shown by the 70% reduction in group size.
Change the bullet from monolithic copper to cast and the group size is reversed. Jacketed bullets fall some place in between. Matching neck tension to powder and bullet can be critical and is probably over looked often.
There seem to be a lot of factors to consider answering the question of if and when to anneal, and how much neck interference to use, how much difference does it make.
My personal thought or answer to the original question is that the benefit of annealing or neck sizing adjustment stop, when it is no longer the weak link.
At my skill level, with the equipment I load and shoot with, at the distances I shoot, the benefits can almost be judged by case failure instead of group size measured in .001's on the target.
That in no way means having an understanding of what goes on and how to control that, is of no benefit. I just see it in .100's instead of .001's. Maybe someday I will play with the big kids.