A family member asked me to watch over her place while calving season was going on - ranges were looooong against coyotes and feral dogs.
I spoke with Hornady about 30 cal A-Maxs, and the "rep" assured me that they had the same jackets as V-Maxs, so I took my 1,000 yard 300 Win Mag target rifle, loaded with 178 A-Maxs.
The 30 Cal A-Maxes do NOT workie like V-Maxs... they slipped right through. 'Yotes spun and ran, neighboring ranchers complained
Plan "B"... For the following season, family member ponies up for a spankin' new Rem M-700 Sendero-II in 264 Win Mag, with a Nikon 6.5-20x44 Monarch.
Gun is very accurate, and loads with 95 V-Maxs actually work like V-Maxs are supposed to. The smash factor on feral dogs and coyotes must be seen to believed - a major terminal ickie!! She doesn't have the ranch anymore, but I still have the rifle... for bigger canines.
I don't think there is one caliber that will do what you want, there are a bunch. But if shots are out in the 800 to 1K area or a bit further, I would recommend something bigger than a 243, and don't waste your time with long needle bullets - they sip right through at long range, and it is amazing how resilient yote and ferals are... if you don't break them, they spin twice, and they are gone!
.
I think a 6mm-06, or 6mmx284, with 87gr V-Maxs would be a minimum place to start, up through the large cased 6.5 - 264 calibers,, like the 6.5-06, the 6.5x284, 264 Win Mag, etc... in a 11 to 15 pound rifle that is (maybe) a no turn chamber, but NOT a benchrest or F-Class rifle... it needs to be a true field rifle.