I think this was an etymological error by the French geologist Jules Desnoyer in Annales des sciences naturelles in 1829. He considered the five periods described by Charles Lyell, then proposed new names for just four periods (now roughly the Jurassic, Cretaceous, Paleogene+Neogene, and Quaternary). Shortly thereafter, he reconsidered the Quaternary and declined to put real support behind it. He wrote in French and called the period the "Quaternaires." This had some influence, though apparently not a lot immediately. But being a textbook, it surely influenced later French geologists.
For instance, Frenchman Émile Haug used the term in the third fascicule of vol. 2 of his textbook Traité de Geologie published in 1911. I know I'm skipping ahead 82 years, but this is hard to research, OK? Haug divided post-Cretaceous time (i.e. the Cenozoic Era) into three periods and renamed them and adjusted boundaries. Apart from the upsetting decision to name the last of three periods "fourth" (for continuity with Desnoyer), he preserved the etymological mistake conflating "quartus" with "quaternius." And somehow from this origin we are stuck with the bizarre nomenclature to this day. I think this use of "quaternary" where "quartary" makes more etymological sense spread from geology to other disciplines like biochemistry decades later. By now, "quartary," to the extent that it was ever really a word at all, has essentially disappeared. It was always so rare a word that a single significant application (an anglicized Fench geologic period) swamped all prior uses, and only "quaternary" survived.
In 1911, a Frenchman writing only as J.W.G. submitted a review of Haug's work to nature. He wrote it in English of course, and in several parts. At one point he brings up this very incongruity. I can't find Haug's text, but in the review, J.W.G. states,
"The post-neogene deposits Prof. Haug groups together as the Quaternary, the term proposed by Desnoyers in 1829. He rejects Lyell's term Pleistocene on the grounds that it is not euphonious, and 'tout à fait', incorrect. But is Quaternary any better in these respects? Quaternarius means 'consisting of four,' 'containing four,' as it is defined, for example, in Lewis and Short's 'Latin Dictionary.' The term is properly employed in quaternion and in quaternary compounds, but not for the name of a fourth division of geological time. Should it not be Quartary?"
Etymologically-speaking, he is correct. "Quartary" would have made more sense as a coinage. Later that year, in the same journal Nature, A. Irving Bishop of Strattford made a brief response titled A Point in Geological Nomenclature. I find it difficult to evaluate in terms of sincerity. He assures J.W.G. that an etymologically correct "das Quartar" was used in the German textbook Elemente der Geologie by Hermann Credner of Leipzig in 1908. This may have been a typo, as in fact the German name for the period is "das Quartär," and Bishop does not forget the umlaut in "das Tertiär."
At any rate, there have been complaints over this strange word for well over a century, and likely longer, but by now it is here to stay. Blame Desnoyer, I guess, and maybe Haug for ensuring that confused term would stick. The other posters are right that "quartary," while more logical, practically does not exist at all, except occasionally in explanations about what went wrong with the word "quaternary." Evidently, as late as the 18th century, it saw very occasional use, but this never approached the eventual significance of the geological term "Quarternary Period." For what it's worth, the association with "fourth" is a far more confusing feature in geology than the etymological confusion between distributives and ordinals. In what sense is the Quaternary Period the "fourth" anything? In no sense, that's what.
