For me it's never ok. I used to know someone that was a serial sleeper with married men. I firmly believe she did it as an ego boost. She felt that if a partnered man was willing to put his family on the line to sleep with her, it must mean she was really desirable/interesting/worth the risk.
As to blame to the 3rd party, they may not be in a relationship with them, but what happened to common decency? I have given this example before - you are in a car park and crash into someone else's car. Is it ok to drive away bc you don't know that person intimately or have a connection to them in some way? Surely we should treat others with respect and decency without having to be in a relationship with them? That argument doesn't work with any other situation I can think of, where you need to have a close connection to someone to feel remorse or take responsibility, and it really doesn't hold weight here imo.