Are dictionaries using 'big data'?

Question

Traditionally (and roughly) dictionaries are written by teams of editors who gather large corpora, identify new words or new word usages, examine how those words are used, and then assign parts of speech and write definitions based on their examinations.

With no ill will meant toward dictionary editors, this seems a task well suited for computers. (As an interesting aside, see this anti-capitalist critique of automation.) Google and Apple, especially, have troves of written material, both old and current. It seems to me (and I confess I have no computer background) this could easily be used to largely automate dictionary writing.

Computers can identify which words are used, in books, emails, texts, etc, which are yet undefined. By comparison with other words used in similar contexts (its location in sentence structure, the meanings and parts of speech of other words which appear in the same place, etc.) the computer could create a list of likely synonyms and assign a part of speech. A human editor would probably then still need to write the definition, but the bulk of the work is done.

Are dictionaries doing this? In some ways this is simply glorified text-prediction. I have looked online but found nothing. The problem with searching for information regarding dictionaries is that every result is a definition.

Answer 1

The short answer is yes, dictionaries do use corpora and electronic searches on those corpora to identify words and phrases, as well as their grammatical categories and semantic relationships.

Here is a quote from Macmillan:

"Using intelligent software... we can find every example in the corpus of a particular word, phrase, grammatical pattern, or collocation. It is this information which forms the basis for everything we say about words in the dictionary."

Macmillan goes on to describe how their software not only finds every occurrence of a word and it's variations, it also outputs one page summaries of the important grammatical and semantic relationships of the word. They give an example:

"The program first collects all the examples of the word being investigated.... Then it applies a second stage of analysis. This time, the software looks at particular grammatical relationships. In the case of evidence, it finds all the sentences where evidence is the object of a verb, then identifies the most frequent verbs used in this pattern.... [P]eople often talk (or write) about giving evidence, finding evidence, presenting evidence, or gathering evidence. Similarly, the... [software outputs] a list of the adjectives that most frequently modify this noun: we may say there is little evidence for something, or talk about clear evidence, strong evidence, or scientific evidence."

They also use the corpora to generate the labels for certain words, for example archaic, informal, American, journalism, etc. They can output these labels based on which documents in the corpora (for example, old texts or audio recordings) the words primarily appear in. Here is their example:

"When we look at all the examples of eatery in the corpus we find that a majority come from newspapers and magazines, and most of these newspapers and magazines are from the U.S. So in the dictionary, the word eatery has two ‘labels’: mainly american and mainly journalism."

One thing Macmillan does not mention, however, is finding synonyms for novel words. They suggests that the primary data for new words is citations. The examples they give are (1) using green as a transitive verb to mean "to make something more environmentally friendly" and (2) using handbags as an adjective. But still, if these uses occur in the corpus, they are obviously subject to the same kind of analysis as other words.

These are just example quotes from Macmillan, but certainly other big dictionaries like Oxford are doing the same thing. If you search around, you are likely to find information about their processes.