Jesus continues speaking at the Pharisee's house, but addresses his host directly.
# the man who had invited him
"the Pharisee who had invited him to his house for a meal"
# When you give
"you" is singular because Jesus is speaking directly to the Pharisee that invited him. (See: [[rc://en/ta/man/translate/figs-you]])
# do not invite
This probably does not mean they can never invite these people. More likely it means they should invite others as well. AT: "do not only invite" or "do not always invite"