See, everyone saw him mow down those soldiers. They know what he's about. he uses fear to control everyone, and appeals to sociopaths like Merle. He's essentially imprisoned everyone in order to make them 'secure'. Wonder how many men that aren't on his team actually live in Woodbury.