There's nothing "fake" about the subject "This formula." It's a perfectly valid subject of the passive verb "is established." In the transposition from the active voice, the subject is ordinarily moved to the object of the preposition "by." Thus your active-voice version might be
The reader established this formula by observing that A+B=C.
In the passive, this becomes
This formula is established (by the reader) by observing that A+B=C.
When the active-voice subject is unknown or unimportant, it is often elided, as indicated by the parenthesis. This leaves the second "by" introducing an adverbial prepositional phrase of manner to modify the verb. Perhaps you feel a small bit of unease at that "by" echoing the missing one.
A dangling modifier would be one that can only modify the missing active-voice subject:
Having observed that A+B=C, the formula is established.