My friends are becoming Dad's friends. Any advice?

I hate to sound trite here but it is becoming annoying. I know he is lonely (he just moved in). We have a large house, I try to give my dd space but when the subject comes up to go to a friends house hes in the car ready to go (so to speak). My wife and I cant talk at dinner as he retains every thing on what and when we are doing or chime in gin on the next event. We do take him to cookouts with our friends when he is invited. The last one had a few seniors there so it was good for him to make connections. Its just when my buddies say to come over to the game hes thinking it includes him. At times I just take off and leave (now I feel bad). I do tell him we are doing our time and going out (we try once a week). I am trying to get him to go out on his own to civic events (lodge meetings) but he wont go without me. I hate to lie or not tell him what I am doing and now my wife and I have to play quiet when we are around (very frustrating). This weekend we will get some time away (way over due). My daughter will come to the house to watch him and the dog (he wont care for our dog while we are away but we have to watch his another pet peeve). So how do I get him to realize my friends are mine, he has been good at times to give us our space and we do make sure we include him. When my friends come over the conversation becomes taken over with the stories I have had for years. I don't want to exclude him or give him a plate of food and send him back to his room but......