I think if the owners 'needed' to do stuff inside the property, they should have done it before the very day the tenant takes possession of the property.
If I turned up at my new rental property ready to move in and the owners were swanning around inside I'd be completely thinking WTF?
They may own the property, but the day the tenant takes possession should be the day they no longer have the right to be there without the tenants permission.
The PM sounds like a d!ck.
OP, if I was you I'd probably send an email to the PM to smooth things over whilst remaining firm on the fact that you do not want this happening again. It would suck for this unpleasantness to affect the rest of your tenancy, so I would try to put this behind you and focus on settling in.