yes doing that bolt up would put the charger on the piss. BUT that bolt has to be tight!!! that's what holds the charger still and stops it moving and vibrating. if you drive round without that bolt tight then no wonder your belts arnt lasting long. I'd expect your pulley to be worn aswell.
I'd make up a plate to remove most of that gap and use the shims I'm sending to you 2moro to get the gap to zero