Language is not all about grammar, semantics play a big part too.
A sentence like this:
I was worried about you.
You could grammatically try to parse as a passive construction, parallel to, say
I was asked about my work.
However, that second sentence can be easily reread as an active construction:
(Someone) asked me about my work.
Whereas the original sentence struggles a bit:
(Something) worried me about you.
Nothing is really off, though.
Now, let's compare the original sentence to a different construction:
I was happy for you.
Here, happy can not be parsed as a verb in any way, and we are simply looking at a subject+(past copula)+adjective. If we interpret the first sentence in the same way, we can see the obvious parallel:
I | was | worried | about you.
I | was | happy | for you.
Grammatically we can parse the sentence as a passive construction, or as an active construction where worried functions attributively ("like an adjective"). Semantically, I think the sentence is usually parsed as the latter, active, construction.