Approach 1: Collexeme analysis

So what Stefanowitsch & Gries (2003) wanted to investigate was the [N waiting to happen] construction, e.g. an accident waiting to happen, a disaster waiting to happen etc. In this collexeme analysis you are quantifying to what degree the words found in a corpus occur in that construction: are they attracted or repulsed, and by how much?

In this scenario, element 1 is e.g. an accident, and element 2 the construction waiting to happen.

Approach 2: Distinctive collexeme analysis

A year later, Gries & Stefanowitsch (2004) extended the methodology to quantify to what degree the words prefer to appear in one of two constructions. For example, the ditransitive give him a call vs. the prepositional give a call to him. Here the table changes into:

Approach 3: Co-varying collexeme analysis

The third seminal paper was published in the same year (Gries & Stefanowitsch 2004; ISBN: 9781575864648) sought to quantify the attraction / repulsion between two different slots in a construction, e.g. trick … into buying, force … into accepting etc. For this, the table is adapted to:

Okay, I get that table stuff, what next?

Let’s say you were able to get the frequencies from a corpus – I should probably mention that, *cough* *cough* not everybody agrees with this kind of contingency tables, the overview paper by Gries (2019) has plenty of interesting references and exciting rebuttals – and you have this table, possibly for lots of elements and/or constructions.

We’ll keep an example here that I calculated with Gries’s (2016) book mentioned above. One of the case studies looks at verbs that co-occur with the modal verb must, e.g. must accept, must agree, must confess, vs. how much these verbs occur with other modal verbs, e.g should accept, should agree etc. So in this case, element 1 was a verb (confess) and element 2 was must; not-element-1 were all the other verbs, and not-element-2 were all the other modal verbs.

The respective table from an abstract level looks like this:

I got these frequencies from the BNC (as Gries shows in his book, but I used my tidyverse skills^TM to get them, so if they are slightly off, then it was because of not following his script completely):

Or in a Base R table:

a <- 11
b <- 1
c <- 1990
d <- 62114

confess <- c(a, b)
notconfess <- c(c, d)

must.table <- rbind(confess, notconfess) 
colnames(must.table) <- c("must", "notmust")
must.table

##            must notmust
## confess      11       1
## notconfess 1990   62114

What do you do now?

Now it’s time for letter math!

The association measure (read: statistical test) that Gries & Stefanowitsch, as well as others, have used most is the Fisher Yates Exact test, and more precisely the negative \(log10\) of its \(p\)-value. Underlying this (see the link in this paragraph) are calculations using those letter cells (a,b,c,d). Luckily we don’t need to do that manually because R has a function for this – fischer.test():

fisher.test(must.table)

## 
##  Fisher's Exact Test for Count Data
## 
## data:  must.table
## p-value = 3.106e-16
## alternative hypothesis: true odds ratio is not equal to 1
## 95 percent confidence interval:
##     49.90447 13241.49567
## sample estimates:
## odds ratio 
##   344.6593

So as you can see the pvalue is 3.105600710^{-16}. If we take the negative \(log10\) of this, it becomes 15.5078544, which gives us the result we expected.

An easier way of computing the Fisher Yates Exact test of this table is by using the letter math and the pv.Fischer.collostr function provided by Levshina’s Rling package:

Rling::pv.Fisher.collostr(a, b, c, d)

## [1] 3.105601e-16

Other tests that have been proposed are so-called Reliance and Attraction (cf. Schmid & Küchenhoff 2013 a.o.). Reliance is the relative frequency of a verb (confess) with must with regard to all uses of the given verb; Attraction is the relative frequency of a verb (confess) with must based on all usages with must.

\[ Reliance = \frac{a}{a+b}\]

\[ Attraction = \frac{a}{a+c} \]

attraction <- a / (a+c) * 100
reliance <- a / (a+b) * 100

The Attraction of confess and must is 0.5497251 and the Reliance is 91.6666667. This high Reliance means that whenever confess occurs in the corpus after a modal it relies on must to occur. Its Attraction, however, has a much lower value: must does not necessarily occur with confess, in fact, it occurs with lots of other verbs as well!

The third group of tests is actually correlated to Attraction and Reliance: respectively \(\Delta P_{word \to construction}\) and \(\Delta P_{construction \to word}\) (cf. Ellis & Ferreira-Junior a.o.). They are also known as tests of cue validity and are calculated as follows:

\[\Delta P_{word \to construction} = cue_{construction} = \frac{a}{a + c} - \frac{b}{b + d}\]

\[\Delta P_{construction \to word} = cue_{verb} = \frac{a}{a + b} - \frac{c}{c + d}\]

dP.cueCx <- a/(a + c) - b/(b + d)
dP.cueVerb <- a/(a + b) - c/(c + d)

So the \(\Delta P_{word \to construction}\) of must confess is 0.0054812 and the \(\Delta P_{construction \to word}\) of must confess is 0.8856234, so these numbers look a lot like those of Attraction and Reliance.

Anyway, I think you get the drift: if you have the contingency table, you choose an association measure (there are more than these three sets) and analyze the results.

Hoop 1: make it really tidy

must.tibble %>%
  pivot_longer(cols = c(must, notmust),
               names_to = "modal",
               values_to = "n") %>% # now it is really tidy
  kable()

Hoop 2: uncount

must.tibble %>%
  pivot_longer(cols = c(must, notmust),
               names_to = "modal",
               values_to = "n") %>% # now it is really tidy
  uncount(n)  # uncount

## # A tibble: 64,116 x 2
##    verb    modal
##    <chr>   <chr>
##  1 confess must 
##  2 confess must 
##  3 confess must 
##  4 confess must 
##  5 confess must 
##  6 confess must 
##  7 confess must 
##  8 confess must 
##  9 confess must 
## 10 confess must 
## # … with 64,106 more rows

Hoop 3: convert to table

must.tibble %>%
  pivot_longer(cols = c(must, notmust),
               names_to = "modal",
               values_to = "n") %>% # now it is really tidy
  uncount(n)  %>% # uncount
  table() # turn to Base R table

##             modal
## verb          must notmust
##   confess       11       1
##   notconfess  1990   62114

Hoop 4: chisquare

must.tibble %>%
  pivot_longer(cols = c(must, notmust),
               names_to = "modal",
               values_to = "n") %>% # now it is really tidy
  uncount(n)  %>% # uncount
  table() %>% # turn to Base R table
  chisq.test() 

## Warning in chisq.test(.): Chi-squared approximation may be incorrect

## 
##  Pearson's Chi-squared test with Yates' continuity correction
## 
## data:  .
## X-squared = 282.63, df = 1, p-value < 2.2e-16

Hoop 5: tidy with broom

must.tibble %>%
  pivot_longer(cols = c(must, notmust),
               names_to = "modal",
               values_to = "n") %>% # now it is really tidy
  uncount(n)  %>% # uncount
  table() %>% # turn to Base R table
  chisq.test()%>%
  broom::tidy() %>%
  kable()

## Warning in chisq.test(.): Chi-squared approximation may be incorrect

I know there are some tidyverse-friendly functions like infer::chisq_test, but it seems to lack arguments like expected values (chisq.test()$exp). So this awkward hoop jumping is annoying but gets the job done (for now?).

From tidy to contingency

Anyway, while for most summary statistics a tidy format (see hoop 1) is the easiest to work with, I don’t think it’s very intuitive for the association measures paradigm.

So in this working example of must + V_inf_ construction what I did was get all the occurrences of all modal verbs + verbs in the infinitive from the BNC corpus. The second step I did was add a column with dplyr::case_when to identify if a modal verb was must or another modal verb.

Let’s look at this data shall we:

df.must

## # A tibble: 64,116 x 3
##    modal  verb      mod.type
##    <chr>  <chr>     <chr>   
##  1 can    find      OTHER   
##  2 should stop      OTHER   
##  3 should recognise OTHER   
##  4 will   cost      OTHER   
##  5 might  think     OTHER   
##  6 can    sink      OTHER   
##  7 can    change    OTHER   
##  8 must   help      MUST    
##  9 ll     help      OTHER   
## 10 ll     take      OTHER   
## # … with 64,106 more rows

As you can see there are 64116 rows, and the table is just a count away form being tidy.

tidy.df.must <- df.must %>%
  count(mod.type, verb, sort = TRUE)
tidy.df.must

## # A tibble: 2,522 x 3
##    mod.type verb      n
##    <chr>    <chr> <int>
##  1 OTHER    get    3750
##  2 OTHER    go     3421
##  3 OTHER    see    2409
##  4 OTHER    take   2056
##  5 OTHER    like   1778
##  6 OTHER    say    1647
##  7 OTHER    come   1460
##  8 OTHER    make   1383
##  9 OTHER    give   1245
## 10 OTHER    put    1162
## # … with 2,512 more rows

Much ‘prettier’, but what next? This is actually the mental step I struggled the most with, this dissonance between tidy and contingency. However, since we coded mod.type with a binary value: “OTHER” or “MUST”, it is actually trivial to dplyr::spread or dplyr::pivot_wider them to a “tidy contingency table”:

spread.tidy.df.must <- tidy.df.must %>%
  pivot_wider(values_from = n,
              names_from = mod.type) 
spread.tidy.df.must

## # A tibble: 2,113 x 3
##    verb  OTHER  MUST
##    <chr> <int> <int>
##  1 get    3750   102
##  2 go     3421    94
##  3 see    2409    14
##  4 take   2056    63
##  5 like   1778     3
##  6 say    1647   105
##  7 come   1460    41
##  8 make   1383    62
##  9 give   1245    22
## 10 put    1162    18
## # … with 2,103 more rows

Hey, this looks a lot like those schematic tables we had at the beginning! But now the real challenge is to turn these numbers into the letters a, b, c, and d, so we can perform our letter math.

After changing all numeric values we have to doubles (instead of integers) and changing all NAs to 0, we can get the letters by following simple arithmetic from our original schematic table, e.g. if we know a and a+c, then c = a+c - a etc.

spread.tidy.df.must %>%
  mutate_if(is.numeric, ~ as.double(.x)) %>% # we'll want doubles instead of integers
  mutate_if(is.numeric, ~ replace_na(.x, 0)) %>% # NAs should be 0
  
  mutate(a = MUST,
       ac = sum(MUST),
       c = ac - a,
       ab = MUST + OTHER,
       b = ab - a,
       abcd = sum(MUST, OTHER),
       d = abcd - a - b -c,
       aExp = (a + b)*(a + c)/(a + b + c + d)) -> abcd.must
abcd.must

## # A tibble: 2,113 x 11
##    verb  OTHER  MUST     a    ac     c    ab     b  abcd     d  aExp
##    <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1 get    3750   102   102  2001  1899  3852  3750 64116 58365 120. 
##  2 go     3421    94    94  2001  1907  3515  3421 64116 58694 110. 
##  3 see    2409    14    14  2001  1987  2423  2409 64116 59706  75.6
##  4 take   2056    63    63  2001  1938  2119  2056 64116 60059  66.1
##  5 like   1778     3     3  2001  1998  1781  1778 64116 60337  55.6
##  6 say    1647   105   105  2001  1896  1752  1647 64116 60468  54.7
##  7 come   1460    41    41  2001  1960  1501  1460 64116 60655  46.8
##  8 make   1383    62    62  2001  1939  1445  1383 64116 60732  45.1
##  9 give   1245    22    22  2001  1979  1267  1245 64116 60870  39.5
## 10 put    1162    18    18  2001  1983  1180  1162 64116 60953  36.8
## # … with 2,103 more rows

So now we can easily perform all of the association measures we want, and rank them accordingly.

Fischer Yates Exact

fye <- abcd.must %>%
  mutate(fye = Rling::pv.Fisher.collostr(a, b, c, d)) %>%
  #filter(verb == "confess") %>%
  mutate(negfye = case_when(a < aExp ~ format(round(log10(fye)), nsmall = 2),
                            TRUE ~ format(round(- log10(fye)), nsmall = 2)),
         negfye = as.double(negfye)) %>% #I did some rounding
  arrange(desc(negfye)) %>%
  select(verb, OTHER, MUST, fye, negfye)

fye %>% top_n(20) %>% kable()

fye %>% arrange(negfye) %>% top_n(-20) %>% kable()

Attraction, Reliance

abcd.must %>%
  mutate(attraction = a / (a+c) * 100,
         reliance = a / (a+b) * 100) %>%
  arrange(desc(attraction)) %>%
  top_n(20) %>%
  select(verb, OTHER, MUST, attraction, reliance) %>%
  kable()

## Selecting by reliance

Delta P

abcd.must %>%
  mutate(dP.cueCx = a/(a + c) - b/(b + d),
         dP.cueVerb = a/(a + b) - c/(c + d)) %>%
  arrange(desc(dP.cueCx)) %>%
  select(verb,  OTHER, MUST, dP.cueCx, dP.cueVerb) %>%
  top_n(20) %>%
  kable()