Change powders or change bullets.
Things may not get any better with jacketed and Trail Boss.
A leading cause in accuracy problems is gas cutting. The bullet not sealing the bore and high pressure gasses blowing by the bullet and deforming the base. Trail Boss was designed for low pressure, low velocity loads. That means soft lead. If you’re talking BHN, that would be under 16, more like 12. The pressure needs to be enough to expand the bullet base to seal the bore. With jacketed and hard cast bullets, the pressure required exceeds what is safe in some firearms chambered in black powder gesigned cartridges. 45 Colt is one on those cartridges.
There is no reason to make a cast bullet in 24 BHN for loading and shooting. The reason the bullet is that hard is so you can load 50 pounds in a box and throw it on and off trucks without damaging the bullets.
All this compounds. If the bullet is too hard, not only will it not seal the bore, it won’t compress. Part of what makes lube in grooves work is compression. This helps push the lube out into the rifling.
The type of lube needs to match the bullet hardness, velocity and barrel length. I shoot smaller caliber cartridges.generally 30 caliber and under. An 85 grain 25 caliber bullet does not have much area to carry lube. My bullets are generally somewhere between 22/1 and 25/1 lead to tin ratio. That puts them in the 9-10 BHN range. 15-1700 fps from a 34” barrel is not a problem for accuracy or leading. It’s all about bullet fit, lube used, powder choice. Probably in that order of importance. If you don’t have a line build up on the muzzle after shooting. It’s clearly the wrong lube, or not enough.
You checked the bore size at the muzzle end. You also need to check the chamber end, between might help in accuracy remains an issue. What can happen is with an over size chamber end, the base of the bullet is basically ruined by gas cutting since it didn’t seal. A smaller muzzle end won’t fix that. The opposite happens with a tight breech end. The bullet diameter is reduced to less than the muzzle diameter. A loose or tight spot any where in the barrel causes problems.
There really is a lot to making a cast bullet fly right. But no reason without just a little bit of thought and getting it all working together that you can’t have a very consistent 2 MOA set up. It just takes some time of trial and error.
There’s always the easy button and shoot jacketed or powder coated.
Here’s the bore on a solid 4-5 MOA rifle shooting soft lead bullet. 32 S&W cartridge, 26” barrel.

. Its job is 2” stieel at 50 yards with iron sights. With powder coated bullets, hits more often than not on a 1” spinner.