One could assemble with anti-rotation pins to mitigate that concern.If you're going to leave bare, I'd say 7075 might be a better option for bare aluminum wear resistance. The difference being seen only after what I'd estimate at 20k+ rounds and I'd guess that at that point the 6061 might have looser pivot pin holes.
Even if the pin holes do wear, there are stainless steel bushings available that remedy that problem. You can also send your completed lower out for anodizing. The go-to guy is U.S. Anodizing.One could assemble with anti-rotation pins to mitigate that concern.