I assume you are talking about three particular shapes, and not just these three shapes in general. If you are just comparing ALL triangles, squares, and circles, then you would most likely want to to use an "s" as I just did in this sentence.
Now let's look at your sentences.
1 The triangle, square, and circle have a different shape.
Incorrect. There are three subjects, but the second half of your sentence is singular.
2 The triangle, square, and circle have different shapes.
Correct.
3 The triangle, square, and circle are of different sizes.
Correct. But you could simplify it to "the triangle, square, and circle are different sizes."
4 The triangle, square, and circle are of a (can I ignore 'a' here?)
different size.
Incorrect for the same reason as #1. And if the subject was, in fact, singular, you could not ignore the 'a.'
5 The triangle, square, and circle have ... VS The triangle, the square, and the circle have ...
While it is grammatically correct to repeat "the" each time, there is no need for it. And in English we like to keep things as simple as possible, while still maintaining the same meaning. It is best to only use "the" once in this sentence.
I'm not sure what the purpose of this is, but just as a comment, both statements can easily be combined into one short sentence: The triangle, square, and circle all have different shapes and sizes.