we tried doing this in our old house. I didn't want someone coming through without one of us home. dh emailed and proposed an alternative time but she said that was the only day she could do so we just let it go. I'm also in Vic.
not sure if you can refuse them entry so long as they've given the notice in the required amount of time (forget what that is but isn't it 7 days??).