Not abusing Leadership

So I'm playing a Wizard/Cleric in a Kingmaker campaign and when and If I make it to 7th level I want a group of fawning hangers-on to polish my armor and cook my meals. That said, I totally do not want to undermine the DM or rock the game. My Dm has been excellent and seems to enjoy running the game and I do not want his job to become harder or overshadow the other players. Still its in character for me to have a crowd of kept artists, servants and personal men at arms as well as a sworn sword to do my bidding.

Currently I have a 12 Charisma and the Natural Born Leader trait so my starting leadership score should be at least 9, I should have a castle/fortress/mansion/guild-house by then because its kingmaker and I'll be 7th level and building a city is part of the AP and my character's personal ambition.

For my Cohort I'm thinking a Cavalier of the Order of Lion (possibly standard bearer) to act as my personal attendant/messenger/right hand. I don't want someone to really take up a role in most combats so much as I want a guy I can send to make proclamations, manage projects for me and make me look important. The way I see it The leadership feat gives me a cohort and followers with the standard array of attributes and the clothes on their backs.

My plan is to assume that two thirds of my Followers are Standard Servants/craftsmen and one third is warriors (my men at arms) and healers (adepts with the self sufficient feat).