We are speaking English, not Latin.
Just because a word comes from another language does not mean that the system of declension (or changes in form to determine syntactic function) follows it.
First, most English users are not familiar with the origin languages of their vocabulary. Old English, French, Latin, Greek, Norse - English users know words from these languages without necessarily knowing anything about these languages' rules for declension, conjugation, or syntax. If the word goes into wider use, users will tend to apply standard English rules to those words: to add an -s for the plural. English is not an obstinate holdout in this regard. Latin does the same thing with words from other languages: compare the Greek-derived abacus (plural abaci) and the Greek root abax (plural abakes). Latin users could have used the third declension structure (which is closer but still not identical to Greek), but they instead created a new word abacus and went with it. Words entering a new language are usually subject to the new language's structures. It would take larger scale structural borrowing to have English use the same declension structure as Latin. That hasn't happened.
Second, even if we wanted to equate a Latin plural form with the English plural form, the question would be, "Which one?" The nominative and accusative plural of the neuter second declension noun virus is vira, but there's also a genitive plural (virorum), a dative plural (viris), and an ablative plural (viris). English lacks all of these forms, but if you're working on the assumption that Latin declension matters, then these forms ought to be accounted for. (That's not to mention the four other declensions, the notion of grammatical gender, or the other distinctions in Latin we're neglecting.)
Using one of these forms is an arbitrary choice, and one that is fundamentally ungrammatical within the structure of that original language. Thus exceptions that preserve the Latin-formed plural like alumni (the nominative plural of alumnus, except in English the end is usually pronounced like "eye" and not like "knee") aren't generative of new plurals, just as woman -> women is not generative; the -i affix cannot be used to form new plurals in English except by exceptional prescriptivism.
Finally, words derived from other languages are frequently subject to more radical changes in spelling or form. Culture comes (via French) from the Latin word cultura, but even the singular form has shifted. When words come into English, its users tend to see them as new words rather than as avatars from their prior language. Forms develop accordingly.
For these reasons, the list of Latin-derived words that have Latin-derived plural forms is small, tends to be academic in usage, and doesn't follow a single rule. Most Latin-derived English words have English or Anglicized affixes.
