Here "the house" represents those who live in the house. Alternate translation: "If the people in that house receive you well" or "If the people who live in that house treat you well"
This could mean 1) that God would hold back peace or blessings from that household or 2) the apostles were supposed to do something, such as asking God not to honor their greeting of peace. Alternate Translation: "take back your greeting of peace"