"Subject" here means two different things, and comes into English by two different paths. "Subject to" comes from the general sense of subjection, that is to say the overlordship of one over another. Much as people who were under the rulership of a king were called his subjects. This usage is attested very early in the English language.
"Subject" in the sense of the underlying meaning of a thing (the subject or an email or conversation for example) is really a shortened form of "subject matter" and is related to the grammatical term "subject" meaning, roughly speaking, the doer of the (active) verb.
So they came into English in by slightly different routes, even though they have largely the same derivation. In both cases they convey the sense of something that lies underneath. In the first case "subject to" it is underneath in the sense of domination, in the later "subject of" it is underneath in the sense of underlying meaning, or underlying subject matter.
So whether to use "of" or "to" depends entirely on if you are intending domination or subject matter.
In the particular examples you have:
This procedure is subject to syntactic analysis.
Here the procedure is ruled by syntactic analysis, which is to say, if syntactic analysis is applied, there is an obligation to accept it.
This procedure is subject of syntactic analysis.
This sentence is actually syntactically incorrect. "Subject" needs a determiner to be correct here. "this procedure is a subject of syntactic analysis" for example. This indicates that one of the underlying meanings, or one of the subject matters of syntactic analysis is this particular procedure.
See Etymology Online for more on this matter.