There was an elevation change from Jerusalem to Antioch. There is also an idea of Jerusalem, especially the Temple, being at the height of importance. Regardless of elevation, people always go up to Jerusalem and up to the Temple, and down when going away.