While my extended family usually keeps their opinions on the subject to themselves, we know they do not fully support our homsechooling. I have met the people they base their thoughts on, and lets just say I dont blame them :) I will just smile inwardly when my kid is the smartest kid they know! :)
ANYwho ...boundaries are really important for a family with small ones. As moms we are responsible for using our children's time well, and sometimes that says saying no to things.
As far as the current situation ..maybe explain that because of the baby we are needing to do school this summer, if you would like to keep him/her for a few days then we are off this week/few days. First come first serve, but he/she will only be able to be away from home for this time frame.
Depending on how old the child is, maybe they fit more than one visit into one week or so off. I know that I have wonderful memories of spending time with my grandparents for a week one summer.
I think the main thing is doign what is right for your kid, and they will have to accept it. It may hurt them, but it wont harm them in the long run.