The fact that the wife is threatening legal action already and that they're trying to move on this so quickly bothers me. Why now, after 2.5 years, is it so important that DD spend so much time with them? In my opinion they should be letting you go at a pace you feel you and DD would be comfortable with. DD is the most important person in all this, and as her father he should realise things need to be what's best for her.
Go at your pace, enlist what help you need and best of luck! 😃