You say that like it’s wrong to take pride in what you do, or that 3-4fps ES is a frivolous goal to strive for? It’s hyper accurate, but the ES starts to trip it up at longer ranges.
That isn't necessarily true. Nobody shoots smaller groups at 1000 yds than the discipline of long range benchrest. Period. We do not obsess over low ES. We obsess over group size.
More often than not, the smallest group doesn't have the smallest ES. Even when tuning at short range (200-300 yds), we never pick a load with a smaller ES over a load with a smaller group. Further, when we are shooting our tuning ladders at 1000 yds we often don't even bother to get out the chrono.
ES is simply an indicator. If it is under 20 fps it is fine, and under 30 fps is probably okay too. The goal is to have a tuning node where the bullets will impact at the same POI with a 20-30 FPS velocity spread. (We don't always get that wide if a node.) Now if the ES is 50 or 60--while it might shoot a good 100 yd group--it probably won't shoot a good long range group.
So there are some things ES can tell us. When I occasionally chrono my 1000 yd BR loads, they are usually in the single digits to low teens. This tells me my loading practices are solid. It tells me that things like weighing primers and seating them to an exact crush with a K&M Primer Gauge does make a positive difference.
If the ES is small but the group is big, then my loading practices are fine but the barrel hates that powder/bullet/seating combo.
If the ES is big but the group is small, it might be an indication of needing more neck tension. The theory here is that the bullet needs to be delayed a tiny bit more before it starts down the barrel to help the powder burn more uniformly.
I recently worked up a load for a customer rifle. It was a hunting rifle in 338 Lapua. The load I found produces a .4" 3-shot group at 100 yds and had an ES of 22. That was the best I could get this rifle to shoot.
The customer took it to a long range shooting course and hit every target out to 2600 yds. He said he still wasn't really happy with the load because it's ES was 22. I pointed out that if he was consistently making hits to 2600 yds then the ES was fine. Then I explained the whole deal with ES and he is now happy.