I have a feeling that what bothers you about "First, Second, Third, and Finally" may be an apparent lack of parallel structure: The first 3 items don't end it "ly" and the first three seem on the surface to be nouns or adjectives while the last is an adverb.
But don't pay too much attention to surface form alone. If you check a quality dictionary, you'll find that all four words can be used as adverbs. So there is no lack of parallel structure.
For example, in my print volume of Random House Webster's College Dictionary (2001), there are these definitions:
first. Definition #9. adv. In the first place; firstly.
second. Definition #21. adv. In the second place; secondly.
third. Definition #11. adv. In the third place; thirdly.
If you're writing poetry, then you may want to pay more attention to the structural forms of the words in addition to the part of speech. But for ordinary prose, matching the "grammatical form" (as opposed to the surface form) is perfectly fine:
Every element of a parallel series must be a functional match...and
serve the same grammatical function. (Chicago Manual of Style, 16th
edition (print) section 5.212, p. 259, emphasis added)
Express parallel ideas in parallel form. Adjectives should be
paralleled by adjectives, nouns by nouns, infinitives by infinitives,
subordinate clauses by subordinate clauses, etc. (The Gregg Reference
Manual, 6th edition (print), section 1081, p. 215)