My chi growls at me when I take off his leash?

When I take Scooby out to play it's great but if I want to come in before he does he growls at me when I try to remove his leash. any ideas on how to stop him?