I would have been peeved about the fence and the stuff being left in the yard but not really about the track and stuff. All the suburbs I have grown up in, (and we moved a fair bit,) it was fairly normal to go and play on the block next door. We had BMX tracks and even a treehouse on one. And it wasn't just my family, there was kids from all over the neighborhood. I guess if they were planning on using it as crop or grazing land it may have been different because soil quality would be considered a bit but if there was a building that was going to be built on it then it was all just going to be bulldozed anyway.
Having said that, I grew up in the wonderful day and age where parents took responsibility for their children and didn't go sue the rest of the world for little Johnny breaking his arm or leg when he fell off his bike, monkey bars etc. (what's sad is I am only 26 and the world has changed so much in this respect, but that's another topic altogether )