Maybe then just remind the kids of this? We have bedroom toys and playroom toys. Inside toys and outside toys. When it all gets muddled up I just nicely remind the kids "can you please play with that in the playroom?" Or "sand stays outside". Or something like that.
I'd rather know someone's rules at the start and always ask if it's ok for the kids to be doing X y z. Honestly it sounds like your guests aren't terribly respectful if they're not.