Anything like this is a summation of uncertainties (tolerances in this case). If everything is *exactly the same, you're going to get exactly the same velocity. But that's never hte case. Listing all the things I can think of.
Powder: If you've ever looked closely at powder, the kernels are not all exactly the same size and color. There will be slight variations in burn rate as a result of the normal manufacturing tolerances and coatings. There is also the obvious variation in charge weight. Heating barrels will heat the powder slightly.
Case: Case volume varies, and case sizing varies. Case volume impacts pressure. Also case heat capacity varies because case weight varies. Case hardness varies. All of the above impact the amount of energy absorbed by the case. case neck consistency impacts starting force.
Barrel: Each shot down the bore changes the dimensions of the chamber and rifling. Each shot also increases the temperature, and the barrel may distort, and will certainly grow slightly after each shot. This alters friction, which impacts velocity. Temperature also impacts the stiffness of the bore, and how it responds to pressure. Powder residue left behind will alter friction and possibly heat.
Primer: Priming compound is not exactly the same from primer to primer, and neither is the cup/anvil geometry. They will vary in energy content. Firing spring hysteresis, lubrication, grit, dirt, and temperature changes will impact the force of impact on the primer.
Bullet: jacket thickness and hardness vary, as does weight, volume, and hardness. Same with the core. Seating depth impacts case volume and start pressure.
Some of this we can control, but some of it we can't. It's also tough to tell which ones really matter the most. It seems like powder charge, case volume, consistent sizing are good places to start, as is using high quality bullets. I don't want to get into the primer weighing thing, but some study there might be illuminating. Heavy, long barrels with large heat capacities would be helpful. And it wouldn't surprise me if some powders are better than others due to the sort of mess they leave behind.
One thing to look at is how the velocity trend changes as the barrel heats. The true variance is often a little less than we think if you account for the rising mean velocity. In other words, if you know your barrel walks a 1/4 minute up over 20 shots, then you can deal with that. The random part, you can't.