Just my 2 cents worth, but we set a 60,000psi, 5000 degree bomb off inside of our barrels every time we pull the trigger. The bullet is compressed to the point of very tight obturation to the bore, which expands to some degree as well as bends and flexes while and after the bullet is in the bore, packing grit and debris with it.
It's amazing what all is going on and how consistent and repeatably we can harness all those goings-on while shooting one hole groups. Amazing! All with lands that are about as tall as a human hair is thick. There's lots going on and into it that doesn't readily meet the eye.
Over time, a very long time...a CLEAN phosphor bronze brush may day the crown. In a much shorter time, a DIRTY brush will do the same, as the dirt acts as an abrasive. That said, yes, it's inevitable that you will eventually wear or dull the nice sharp crown that we painstakingly indicate true and machine...for about 50 bucks.
The question is not if it will damage the crown, but how long will it take to do it to a point where it will degrade accuracy. To some degree we could likely calculate that time. But, based on several years of experience, I can say that IMO, with reasonable care, it's long enough that the throat will need maintenance before the crown will.
I very much believe in the use of a bronze brush in my cleaning regimen, mostly for the purpose of cleaning hard carbon from the bore. I'm yet to find a carbon cleaner that is effective enough on carbon, without the aid of agitation with a brush. There are cleaners that will soften it and the brush can score the carbon enough to let the cleaners penetrate it, making removal much easier and requiring fewer strokes of a rod through the bore. That's where I feel we can see the most benefit from a brush, as the same dirt and grit that makes the brush abrasive to the crown also winds up on the rod while in the bore. No matter the rod, and no matter the guide, the rod and this grit will come into contact with the lands. This is true, whether we're brushing or not, as long as we're "pushing" the rod through the bore
So, the fewer times we have to push a rod through, the better off we are, IMHO. A brush simply reduces the strokes necessary. The rod WILL hit the lands near the middle of the bore and the only way to stop it is to pull the rod rather than push it. Bill Calfee even states as fact that a ss rod hits the bore hard enough to gall in a ss barrel. I don't buy that for a minute, but it does come in contact and with a fair amount of pressure, depending largely upon how tight the brush or patch is. My point is, as easy and inexpensive as it is to re-crown...and as much of a pain it is to remove the brush with each stroke...and the amount of time or strokes it will take to damage a crown...I don't see it worth it. We have bigger concerns when it comes to cleaning than pulling the brush back across the crown. Yes, eventually it will dull it, but what will the throat look like by that time and how about the lands near the middle of the bore that have been abraded by the rod with every stroke? I believe in being diligent in regard to careful cleaning, but barrels are just like powder and bullets...expendable. The crown can be touched up at any time if need be. Shooting the small 30's for years, I have seen dulled crowns before the barrel laid down, but they are exceptional in regard to barrel life. A simple touch up of the crown made me feel better, but in all honesty, didn't make it shoot better.
All of that said, you certainly aren't going to hurt anything by removing the brush rather than pulling it back across the crown, so if that's what you want to do, do it. I'll be shooting while you're cleaning your gun.