# we bought

The word "we" refers to Nehemiah and his servants.

# all my servants were gathered

"I gathered all of my servants there" 

# for the work

"to work on the wall"