I run o-rings and die bushings (Lock-N-Load quick change) and have no concerns about movement. It can allow the die to self align.
Mine is a quick change as well. Set it and forget it (sort of). I'm getting a consistent 0.002" now, so I was hoping a little bit of play (lateral, not vertical) would be OK. Now I'm on to learning how to trim.