On a dreary Saturday afternoon, I decided to ask ChatGPT a bunch of standard acceptability judgments. My prompt was this:
For each sentence, tell me if it is acceptable in spoken English or not. Do not give any explanations.
After that, I just typed in the sentences one after the other. And it spit out either 'Acceptable' or 'Not Acceptable.'
Out of a total of 32 questions it answered correctly 29 times.
One of the mistakes was a prescriptive issue with case. It said 'Me and John are leaving tomorrow.' is unacceptable. I take this sentence to acceptable in spoken English. Another one of the mistakes was a garden path sentence. It said 'The horse raced past the barn fell.' was unacceptable. I take this sentence to be acceptable. The other mistake was 'John's mother loves herself.' The first time it said the sentence was unacceptable, but then two later times it said the sentence was acceptable.
In my opinion, the results were remarkably good, and warrant further detailed study. More detailed sets of materials should be assembled and tested. These materials should include both acceptable and unacceptable sentences, and both well studied and not so well studied constructions. It should also vary the examples for prescriptively correct versus prescriptively incorrect. Such studies should be done on different domains. All of these studies might detect if ChatGPT has any weaknesses.
For each sentence, tell me if it is acceptable in spoken English or not. Do not give any explanations.
My brother seems sleeping.
Not acceptable.
My brother seems to be asleep.
Acceptable.
John’s mother loves himself.
Not acceptable.
John’s mother loves herself.
Not acceptable.
Second time (later): Acceptable.
Third time (later): Acceptable.
John’s mother loves themselves.
Not acceptable.
Who did you see John with?
Acceptable.
Who did you see John and?
Not acceptable.
John is very happy.
Acceptable.
John is not very happy.
Acceptable.
John does not be very happy.
Not acceptable.
Lee is likely to be elected.
Acceptable.
Lee is probable to be elected.
Not acceptable.
John and I are leaving tomorrow.
Acceptable.
Me and John are leaving tomorrow.
Not acceptable.
Colorless green ideas sleep furiously.
Acceptable in form, but nonsensical in meaning.
Furiously sleep ideas green colorless.
Not acceptable.
I wrote up the paper.
Acceptable.
I wrote the paper up.
Acceptable.
I wrote it up.
Acceptable.
I wrote up it.
Not acceptable.
Mary jogged up the hill.
Acceptable.
Mary jogged the hill up.
Not acceptable.
The horse raced past the barn fell.
Not acceptable.
John is easy to talk.
Not acceptable.
John is easy to talk to.
Acceptable.
Which paper did you throw away without reading?
Acceptable.
Who did you say that John saw?
Acceptable.
Who did you wonder whether John saw?
Not acceptable.
Who did you say was present?
Acceptable.
Who did you say that was present?
Not acceptable.
Who did you say I talked to?
Acceptable.
Who did you say that I talked to?
Acceptable.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.