I think you can refuse anything you like if you're being polite. You can refuse rudely also, it's your choice but you may not ever be offered anything ever again from that person if you're rude. Being polite always is the better option.
I think it can become an issue if you always refuse though. Eg. I have a certain couple of friends who visit periodically but always refuse anything I offer - a drink (any drink!), snacks, cake, anything!) and I find that rude tbh. It comes across as unfriendly and snobby to me. Maybe they don't intend that but I think it's kind and nice to appreciate being offered drinks/food when visiting. Indeed I always turn up with a plate or something myself when I visit others.