I would make the argument that a rifle capable of 600 yard effective range is not compatible with a home defense weapon. Optics, barrel length, attachments, and ammo for both of those criteria are significantly different. One of the fun parts of ar15s is having different uppers.
You want 600 yard effective range? Get a low to mid power variable optic and put an 16-20 inch 6 or 6.5 grendel variant upper with a bipod mounted. I like the 6mm better than 6.5 for grendel.
You want home defense with a mag restriction? take your pick of high energy close range ar 15 compatible rounds. Slap in a stronger spring and an adjustable gas block and your ar15 can chew anything you want and get more than double the energy per shot of 223 (450 bushmaster can get over 2.5k ftlbs).
Or you can use a shotgun. Which is better. Anything between nickel plated 4 buck and soft lead 00 works really well. I started out reloading for shotgun so I have done a lot of it. A lee loadall is like 50 bucks. Grab a couple hundred fgm hulls from a local trap club, some 12s4 wads (cut off petals off), some blue dot, and some bpi buckshot and you got cheap-ish home defense 12 gauge ammo you can tailor for recoil or function in any 12g. Its fun too. bpi has a good deal on number 1 buck which is the smallest I suggest for unplated lead. 16 pellet should be easy (4x4). Use 1 3/8 load data.
The first Nickel-Plated Buckshot available to American handloaders. This is the finest buckshot we've ever seen! Antimonied lead core with a copper wash and then a thick nickel plating. Coyotes have met their match!
www.ballisticproducts.com
Should cost about .50 per shell. Which is about what the junkiest 9 pelllet 00 costs right now. 16#1 has 50% more cutting area per shot. Thats a ton more damage in home defense range. And number 1 buck is the smallest pellet size that meets fbi minimum penetration testing. Best home defense ammo on the planet in my opinion, that's why I made a ton of it. Doesn't kick bad with blue dot either.