There is a product called a Dricast - it's like a latex bag or something that you put on and it seals to your skin. All the photos on the website show people swimming with them on. My husband got one when he was in a lower arm cast - his Dricast went all the way above the elbow. It comes with all the directions as to how to get it underwater (something about submerging it a bit at a time) - well worth a read, because the aforementioned husband did NOT take much notice of it and ended up with a wet cast after taking our little one to his swimming lesson. That was less than a week after the cast went on. He spent a lot of the next few days with a hairdryer, vacuum cleaner and a bag of rice.
If it is just your finger, could you get fitted with a thermoplastic splint? Not long after the broken hand story above, my husband then managed to break another finger - and he ended up needing a surgical procedure to pin it. A week after that, they fitted him with a thermoplastic splint and he was swimming and snorkelling on our holiday shortly after