Little ones can be like that- totally normal.
I used to wake DD if I HAD to be somewhere (doctors, CFHN, Imunisations etc)
If it was a lunch outting, or going to a friends house/shopping, I'd call & move the time/date or ask them to come visit us insted.
Do you have a capsule/stridder?
We had a capsule, so if I fed and DD fell asleep & I knew we had to leave soon, I'd put her down in the capsule(insted of cot) then slot the capsule into the car when we left.
3 weeks is still so little. I don't think I left the house with DD till she was 6 weeks old.