I see what your saying op. you went to the track when no one else was there so there was no risk to other kids on bmx bikes so people can't push that issue.
I grew up with motor bikes and I rode on the school oval. It was wrong and we had the ranger called on us heaps and we would ride off. Your daughter is 4 and not on a loud bike tearing up the soil so I see no issue.
U will always have some people get on their high horse about it and call the police tho.
I now like in a new estate and we have motorbikes riding around the blocks that aren't built on yet through the bushes and who cares!! I'm sure that's illegal but I don't mind it at all. But then my neighbour next door might hate it. So everyone is different.
If I saw ur 4yo riding I would be jealous cuz I can't wait for my 1yo boy to be old enough to have his own bike!
Just try find somewhere more secluded. Or if someone near u has a big property as if u can go in and let your daughter ride there.. I wouldn't mind if I had a bush property!