Definitely have something planned with friends or family to help keep you distracted. Personally though if your son has never spent any time with his dad then him just coming and taking him for a couple of hours would probably be really hard and upsetting for your son. I know it may not be the nest option for you as you would have to chaperone, but could you start with his dad spending time with him while you're still there (such as at a park if you don't want him in your house). So that your son gets to know this person a little bit before he takes him.

Can dad come to your house and spend some time with him in his own environment?
Please don’t be a mum that denies access. If his father is wanting to be in his life please let him.
Have a discussion with him about boundaries etc and you should be able to successfully compare t together.

Change is hard, being away from your child especially when they are your world is tougher. My advice from personal experience is to do a slow transition if the toddler struggles. Don't rush anything of you aren't 100% sure its safe. But if you know your child is safe plan to so some "me" time whilst your child is gone. Catch up with friends do something you cant usually do with a toddler around. Keep busy essentially. Its hard at first but gets easier especially when you see how happy your child is spending time with both parents seperately. All the best.