I’ve found annealing to be absolutely critical to some tasks, and virtually irrelevant for a great many others. I still use it, sometimes, if I’m doing case reforming such as making 7mm TCUs out of 223/5.56 brass. As for accuracy, I’ve run cases as many as 25-30 firings without any detectable loss of accuracy, and that was firing from strictly match-grade barrels in a return to battery machine rest fixture. I tended to lose cases due to primer pockets loosening, and almost never to case splits at the neck. If you’re losing necks or cracking, you’re most likely just st overworking the cases due to your die I.D. and your expander O.D., which is a totally separate problem.
Anneal if you wish, but many of the benefits being touted today are largely imaginary, and the process share as hell isn't the necessity that many are making it out to be.