Before you go out and spend a ton of money on more equipment, I've using standard RCBS die with expander balls for 50+years without any issue whatsoever.
Expander Ball Discussion:
Are you getting any significant drag on the expander button? If so, are you lubing the inside of the case neck? Most expander balls are oversized and have a rough surface. I polished mine with very fine emery cloth to produce a "glass like finish" which significantly improves expander the neck without lengthening it. Also, by reducing it .001" often produces better fit.
Sizing Discussion
What are the fired datum line measurements versus the sized measurements? If the die is not touching the shoulder, then you could be extruding (lengthening) the case. You don't need to purchase competition shell holders. You can adjust your sizing die in small increments to find the optimum shoulder set back.
Although you don't need these either, I use Ship Shims which fit under the lock ring of the die to simplify making sizing corrections. These range from .003" to .010" and provide a full range of easy adjustment without having to reset your FL die. Available at Brownell's and are much cheaper than competition shell holders and quite frankly work better, at least for me.
I cannot comment on annealing since I don't use this process, but I wonder if any inconsistency in the process could be causing a problem.
The only thing I can think of that's causing the first sizing to be off is that you have a lot of flex in the press, or the sizing die in working loose, or you do not have sufficient cam over so that the die is at least touching the shoulder, or somehow the annealing is affecting the consistency of the brass but as I said, I do not anneal so this comment may not be applicable. It would be interesting to see the fired datum line measurement and the sized measurements. Is the shoulder being pushed back enough and consistently?