In my view, 0.003" is cutting it too close with unturned brass, although just barely. Even good brass can vary in thickness by 0.001" within a lot. If you turn it, you'll KNOW you have 0.003", which is fine. But unturned, you might have 0.001" on some cases, and you might once in a while get one with no clearance. It sort of depends on which cases you measured to figure the 0.003". You might be fine, but you might be going to close.
I've found that .007" of clearance on a 6mm is plenty to give case neck life that will outlast the primer pockets without annealing, but I shoot for 0.005". I've done 0.001" before on a benchrest setup, but you really have to be careful, and turning is obviously required. I don't know if it helps accuracy any, but the cases last forever.