all the men who were with him

"all the men accompanying Abimelech" or "all the men fighting for Abimelech"

dividing into four units