I am by no means claim to be anywhere close to a 'professional' but will take a stab at this. Sure others will also. Everything I have read says to set cut off tool dead nuts center line. Too low and it can or will cause the 'dig in and raise hell with everything' you are experiencing.lol Too high and it will chatter. Too fast and it might chatter. Too slow and them SOB's might chatter. Too dull and they might chatter. Not exactly at 90* to material being cut and it might chatter etc etc. Not my favorite tool but I have found it easier to use a cut off tool on my SB Hvy 10 then my Atlas/Craftsman 36". Rigidity and mass helps to prevent flex which can cause all of what you are mentioning in my experience! Also using a good cutting oil will usually make it cut smoother. However, I have had times when running it dry and slow worked better, just don't cut much at a time and allow tool and material to get very hot.
Edit to add...take your time and make darn sure you have your tool at 90*(well, not yours but you know what I mean). A shallow cut isn't real issue but get into a piece of 1" or bigger and you will get a lot of heat build up and cutting edge deflection if you are cutting on the front and friction/heat on both sides. I have a VFD on my SB and turn speed to a crawl and slowly increase speed and or inward pressure until I get the optimum cut, no chatter etc. One of the great benefits of variable speed is with a cutoff tool in my experience.
My worthless amateur 2 cents worth!
Respectfully,
Dennis
Suppose when I hit post there will be 40 replies with my slow typing that will all be opposite of my experiences.lol