went down

This means that they traveled from a higher place to a lower place. Cana is southwest of Capernaum and is at a higher elevation.

his brothers

The word "brothers" includes both brothers and sisters. All Jesus' brothers and sisters were younger than he was.