I wouldn't (and don't) hesitate to drop activities if they have a negative impact on the rest of the family. It can just be temporary after all.