you have refused to let him go

"you have refused to let my son go". The word "him" refers to the people of Israel as God's son.

I will certainly kill your son, your firstborn

The word "son" here refers to the actual son of Pharaoh.