I didn't realize at first what prompted the attacks on my towers, until I paid attention to one of the notes on a load screen. Once I realized that, it was all good. I usually don't hesitate to move around the city however I like, since I'm by now capable of wiping out whole platoons and barely taking any damage, so I'm often hovering around three-fourths notorious. Paying off heralds and such is easy enough, though, especially with all the revenue I was generating after renovating most of the shops from an early point in the game. :-D
"Too often we enjoy the comfort of opinion without the discomfort of thought." - John F. Kennedy on reality
"What if everything you see is more than what you see--the person next to you is a warrior and the space that appears empty is a secret door to another world? What if something appears that shouldn't? You either dismiss it, or you accept that there is much more to the world than you think. Perhaps it really is a doorway, and if you choose to go inside, you'll find many unexpected things." - Shigeru Miyamoto on secret doors to another world