Question... See if your attack waves is unsuccessfully, like you would not get through his defense, would the loyalty still goes down? 250 guardians... That's not much troops at all to attack an city is it?
Loyalty is only lowered if the attack is successful AND the scholar in the attack wave survives.
If the attack is successful but the scholar is killed (or no scholar was sent) then the city's loyalty is unaffected. If the attack is unsuccessful then the scholar with the attack wave will be killed.