When Grammaticality Judgments Differ (version 2)

What do we do, as syntacticians, when there are differing grammaticality judgments between two speakers for a particular sentence? How do we interpret those differences theoretically? In this blog post, I outline various alternatives.

The implicit assumption I am making is that accounting for differences in grammaticality judgments is important. If you find a person whose judgments differ from yours (or more generally two people whose judgments differ), it is not an occasion to throw up your arms in frustration. Rather, it should be viewed as an interesting problem to solve that could shed light on syntactic theory. 

Also, I am implicitly assuming that variation in grammaticality judgments is not random. Rather, any such difference between two people can ultimately be explained in some way or the other. Therefore, when presenting research results it is necessary to report variation, even if you do not have a good explanation for it.

In this post, I give the reader advice on how to think of differences in grammaticality judgments.

For convenience, I will call the two protagonists A and B, and the relevant sentence S.

1.

Semantic Anomaly

Either A or B is confused about the task. They are mixing up grammaticality judgements with semantic or pragmatic anomaly. For example, A judges S to be ungrammatical, because it is semantically anomalous (as in “Colorless green ideas sleep furiously.”). On the other hand, B judges S to be grammatical.

For the most part, these kinds of differences fade with experience. Beginning students often make this kind of mistake, but after practice, people can usually distinguish the two kinds of issues (semantic anomaly and ungrammaticality) fairly easily.

However, we must keep an open mind, and admit that there will be difficult cases, where it is difficult to know whether the issue is syntactic grammaticality or semantic/pragmatic anomaly.

2.

Processing Difficulty

Once again, either A or B is confused about the task. They are mixing up grammaticality judgements with processing difficulty. For example, A judges S to be ungrammatical, because it is difficult to process. B on the other hand judges S to be grammatical.

To avoid problems like this, the syntactician should construct sentences that are as simple and as natural as possible. Follow these general rules: Avoid using overly long sentences. Avoid using words that the speakers are unfamiliar with. Avoid using complex constructions that are not directly relevant to what you are investigating. Avoid any kind of construction that gives rise to garden pathing ("The horse raced past the barn fell.").

If the task inherently involves complex sentences, do everything possible to reduce the cognitive load on the consultant. For example, a simple context might help orient the speaker. You may also want to read the sentences out loud, instead of (or in addition to) presenting them on paper or on a computer screen.

By making the sentences as simple and natural as possible, and by reducing the cognitive load on the consultants, you reduce the possibility that there will be variation between the consultants due to processing difficulty.

And as with semantic anomaly, we must keep an open mind, and admit that there will be difficult cases, where it is difficult to know whether the issue is syntactic grammaticality or processing.

3.

Difference in I-Language

Sometimes it is possible to explain differences in grammaticality judgments in terms of differences in I-Language. A and B differ in their grammaticality judgments for S, because A and B have different I-Languages (even though both speak “English”). For example, some speakers accept sentences like “I sent over the boys the pizza” and some do not (I do not). Perhaps there is a micro-parameter operating in this case governing the distribution of particles, where Speaker A has one setting and speaker B has another setting.

If A and B have different I-languages, one might expect other correlated differences in their judgments of different sentences. Also, another way to confirm such an I-language difference is to see whether it correlates with non-linguistic factors such as geography. If all Scottish people judge S in one way, and all North Americans in another, then it is evidence for a difference in I-language. But that is not to say that all differences in I-language must be correlated with geographical differences.

4.

Difference in Structure

It is often the case that A and B differ with respect to their structural analysis of the data. Perhaps A parses the input as S, and B parses the input as S’ (where S and S’ are sentences that have the same words in the same order, but a different structure). In other words, there is a syntactic ambiguity, and A and B resolve the ambiguity in different ways.

Furthermore, A judges S as ungrammatical, but if A had been able to recover the other structure S’, A would have judged it as grammatical.

If there is really a difference in structure, one might expect to be able to disambiguate the structures through prosody and context. A particular prosody might favor S over S’, and similarly a particular context might favor S over S’. Once the prosody and context are fixed, it might be that A and B agree on judgments.

5.

Difference in Tolerance Levels

A and B might have very similar I-languages, and might both be trained syntacticians (not making beginner mistakes), but still differ in grammaticality judgments. Anybody who has been to graduate school in linguistics knows that there are always some students who accept more sentences than others.

I propose that a different kind of variation between individuals has to do with tolerance levels. A tolerates more ungrammaticality than B. For example, A might judge S as *, and B might judge it as ??.

If there is a difference of tolerance, one might expect a global shift (over lots of different judgments) such as the following:

A * à B ??

A ?? à B ?

A ? à B OK

As far as I know, tolerance has not been discussed or investigated before. If people know of relevant literature, please let me know.

6.

Satiation

It has been proposed (see Snyder 2000) that certain kinds of grammaticality judgments shift with repeated exposure to similar ungrammatical sentence types. For example, if a person gives grammaticality judgments for many instances of a whether-island violation, then the violation becomes less severe for them.

In the context of our discussion, A and B might have different grammaticality judgments for S because A might have judged sentences very similar to S many times in the past. For example, this kind of situation occurs when A is a linguist who has been studying examples like S, and B is either a non-linguist, or B is a linguist who has never thought about S like sentences before.

7.

Difference in Meta-Linguistic Awareness

In my blog post Meta-Linguistic Awareness and Syntactic Fieldwork, I discuss the fact that many people, especially elderly people who have no formal education, have difficulty with grammaticality judgment tasks.

It may be that A and B give different grammaticality judgments for S because A or B has a very different level of meta-linguistic awareness. In this case, there might not be any difference in I-language or tolerance level between A and B, but they might still give different judgments.

8.

Absolute and Relative Grammaticality Judgments

Syntacticians often say that the absolute grammaticality judgments of two sentences are not relevant, but rather the difference between them is. For example, people may have widely varying judgments when an argument is extracted from an island, but they all agree it is worse to extract an adjunct from an island.

From this perspective, it is not so much the absolute grammaticality judgments that are relevant, but rather how two sentences in a minimal pair (or a paradigm) differ with respect to their grammaticality judgments. So two people A and B might differ in their judgment of S, but they might agree that S' is worse than S. 

Focussing on relative grammaticality judgments helps to address the issues of tolerance and also to some extent processing difficulty. If two speakers A and B have different tolerance levels, they may still perceive a difference between S and S'. Similarly, even if A and B differ in sensitivity to processing difficulty, they may still perceive a difference between S and S'.

While looking at relative grammaticality judgments is a very powerful tool in the syntactician's toolkit, I do not think it that it absolves us from looking at absolutely grammaticality judgments.