Using Statistics in Lexical Analysis

This chapter discusses a number of statistical tools and shows some examples of how these tools can be used to enhance productivity of a human trying to solve one of these problems. It begins by introducing two statistics, mutual information and the t-score. British lexicographers, especially those working on dictionaries for foreign learners, are beginning to depend heavily on machine-readable corpora. They use these corpora and the basic concordancing tool mentioned above to fill in detailed syntactic descriptions. The notion that a language consists of a fixed set of rules and a finite set of lexical items, and that they can be learned by students, who will then “know” the language, is too attractive to be easily dislodged, but it has been responsible for some misconceptions inter alia about the status of infelicities typically produced by foreigners, which are sufficient to identify the speaker as foreign, although not actually wrong.