Two things combine, here:
1) structures
2) vocabulary
The structure with the preposition before the relative pronoun – with whom you work – is more formal than the one with the preposition staying with the verb – ø / that / who / whom you work with.
The relative pronouns 'who' or 'whom' are more formal than 'that' or dropping the object relative pronoun altogether.
A sensible thing to do is not to combine a formal structure with informal vocabulary or formal vocabulary with an informal structure.
So here, the sentence which would be the least likely is the fourth one, "The people whom you work with are your 'colleagues'." The informal structure, preposition staying with the verb, does not go well with the formal relative pronoun 'whom'.
Another area of 'grey grammar', sentences which cannot be deemed 'wrong' but which you are not likely to hear or read.
I would call
1) informal (informal or standard structure + informal zero relative pronoun)
3) standard (informal or standard structure + standard relative pronoun)
5) formal (formal structure + formal relative pronoun)
The reason 2) is 'not quite right' is that if you go for informality, you might as well go the whole hog rather than stop half-way through. Why stop at 'that' if you can further simplify by leaving the relative pronoun out altogether?!
Same reason for 4), for formality and complexifying instead of simplifying.