Published on December 28, 2007
Sentence comprehension in a wider discourse: ERP studies on syntax, sense & referenceJos J.A. van Berkum 1,2,3, Pienie Zwitserlood 4, Valesca Kooijman 2,3, Colin M. Brown 2 & Peter Hagoort 2,3 1 University of Amsterdam, The Netherlands; 2 Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands; 3 F.C. Donders Centre for Cognitive Neuroimaging, Nijmegen, The Netherlands; 4 Westfälische Wilhelms-Universität Münster, Germany: Sentence comprehension in a wider discourse: ERP studies on syntax, sense & reference Jos J.A. van Berkum 1,2,3, Pienie Zwitserlood 4, Valesca Kooijman 2,3, Colin M. Brown 2 & Peter Hagoort 2,3 1 University of Amsterdam, The Netherlands; 2 Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands; 3 F.C. Donders Centre for Cognitive Neuroimaging, Nijmegen, The Netherlands; 4 Westfälische Wilhelms-Universität Münster, Germany Context It has been known for a long time that event-related brain potentials (ERPs) can provide valuable information about the nature and time course of sentence comprehension. However, in part because using brain measures is difficult enough as it is, psycholinguists have until recently limited their use of ERPs to the comprehension of single sentences presented in isolation. Over the past five years, we have begun to use ERPs to study the comprehension of sentences in the context of a wider discourse. On this poster, we summarize the main findings and their implications (see ref 12 for a more extensive review). (1) Semantic analysis is rapidly sensitive to the wider discourse First of all, our work has shown that when people read or listen to a sentence embedded in a wider discourse, they immediately relate the meaning of every incoming word to their knowledge of this wider discourse [1,2, see also 3]. Relative to globally coherent control words, words that are coherent in their local carrier sentence but anomalous with respect to the wider discourse elicit an N400 effect, emerging at about 200 ms from word onset during reading (Fig. 1C) and listening (Fig. 1F). Discourse does not just come into play at sentence ends, but also right in the middle of an unfolding clause (Fig. 2A). In addition, many spoken words are related to the discourse well before their acoustic offset (Fig. 2B), possibly even before listeners know which exact word it is going to be. (2) Functional equivalence of discourse- and sentence-semantic context The scalp distribution, shape, and timing of the differential effect elicited by discourse-dependent semantic anomalies was identical to that of the standard N400 effect elicited by a ‘local’ sentence-dependent semantic anomaly ([1,2] see Fig. 3 for listening results). Because qualitatively identical ERP effects suggest an identical underlying (set of) neural generator(s), this suggests that from the perspective of the semantic comprehension process indexed by the N400, there is no fundamental distinction between the interpretive context provided by a discourse and one provided by the first few words of a single unfolding sentence. As such, our findings support ‘single-step’ models of comprehension in which words are immediately related to the widest interpretive domain available, bypassing ‘local/literal’ semantic analysis. (3) Referential analysis is rapidly sensitive to the wider discourse People also rapidly discover whether a singular definite NP uniquely refers to a single referent in the earlier discourse or ambiguously refers to two equally eligible candidate referents in that discourse [4,5]. Relative to their unambiguous counterparts, referentially ambiguous nouns elicited a frontally dominant and sustained negative shift in ERPs, starting at about 300 ms from word onset during reading (Fig. 1A), and about 300-400 ms from acoustic word onset during listening (Fig. 1D). This frontal negativity can also be observed with referentially ambiguous pronouns, as in Max shot at Onno as he jumped over the fence . This equivalence of ‘sentence-internal’ and discourse-dependent referential ERP effects suggests that the referent identification process at hand does not care about where the candidate antecedents came from. (4) Selective ERP markers for sense and reference The fact that semantic and referential problems in language interpretation elicit qualitatively different ERP effects – centroparietal N400 effect versus frontal negative shift – also reveals (a) that the processing implications associated with difficulties in making sense and establishing reference are at least partially distinct, (b) that the N400 effect thus does not exhaustively reflect all aspects of conceptual integration, and (c) that in a way, the brain honors the distinction between sense and reference. (5) Syntactic analysis is rapidly sensitive to the wider discourse Discourse-induced referential ambiguity also had an immediate impact on how the parser analysed a subsequent local syntactic ambiguity [4,7,8; see Fig. 1 for an example item]. For instance, if the NP “the friend” in “The hippie warned the friend that…” was referentially ambiguous, the parser was more inclined to take the word “that” as initiating a relative clause (RC, e.g., “…that had had quite a few drinks”) than as initiating a complement clause (CC, e.g., “…that there would be some problems soon”). This context-induced RC preference after referentially ambiguous “friend” was revealed by the fact that later evidence for an actual CC continuation (at “there”) elicited a P600/SPS effect, indicating that the parser had been ‘led down a garden path’ by pursuing the RC analysis in this 2-referent context (see Fig. 1B for reading, Fig. 1E for listening). These indications of immediately discourse-sensitive parsing are incompatible with syntax-first models of syntactic ambiguity resolution. (6) A discourse cue can influence the syntactic parser before a gender cue Other P600/SPS evidence (not shown here, but see [4,7,8]) revealed that discourse-dependent referential information can in fact lure the parser into pursuing a relative-clause analysis of “the friend [that…]RC/CC” even if such an analysis is formally ruled out by a local syntactic gender cue. Thus, even in determining syntactic structure, a ‘soft’ discourse cue can sometimes temporarily overrule a hard syntactic constraint. (7) Incremental processing ‘all the way up’ Prior research had already shown that the comprehension of a single isolated sentence is to a large extent incremental, with readers and listeners trying to relate every incoming word to a (partial) syntactic and semantic analysis of the local sentence as it unfolds. Our ERP research on the processing of sentences in a wider discourse context now suggests that this incrementality holds ‘all the way up’, in that (as illustrated in Fig. 1) readers and listeners also immediately relate the syntactic, semantic and referential implications of incoming words to their knowledge of the wider discourse. (8) Not just incremental, but also predictive processing We obtained ERP evidence that listeners can actually also use their knowledge of the discourse to predict specific upcoming words in an unfolding sentence [9,10, see also 11]. We created (Dutch) mini-stories such as “The burglar had no trouble locating the secret family safe. Of course, it was situated behind a […]”, which in a paper-and-pencil cloze test were predominantly completed with one particular critical noun [e.g., painting]. To test whether such discourse-based lexical prediction would also occur ‘on-line’ as part of real-time spoken-language comprehension, we presented the complete stories in spoken form in an ERP experiment, with one modification: the critical noun was now preceded by a gender-inflected adjective whose syntactic gender either agreed with that of the anticipated noun (behind a bigneu paintingneu) or did not agree (behind a bigcom paintingneu). Relative to the gender-congruent prenominal adjective, the gender-incongruent adjective elicited a small but reliable ERP effect right at the inflection (Fig. 4, LPE). Also, in a control study without the wider discourse, the ERP effect disappeared. Because this prediction effect hinges on the arbitrary syntactic gender of an expected but not yet presented noun, it suggests that discourse-level information can indeed lead people to anticipate specific upcoming words ‘on-line’, as a local sentence unfolds. In addition, it suggests that the syntactic properties of those anticipated ‘ghost’ words can immediately begin to interact with locally unfolding syntactic constraints. (9) Equivalent findings for spoken and written language All of the results discussed under (1) to (7) were obtained both with written language input presented at a fixed rate of 600 ms/word (SVP, Serial Visual Presentation) and with fully continuous spoken language unfolding with normal rate and intonation. This suggests that (a) important aspects of comprehension are modality-independent (as one might expect), and that (b) the use of a relatively slow SVP procedure is not necessarily problematic (in contrast to a frequently voiced concern). (10) ERPs help to keep discourse-level language comprehension tractable Like many other brain measures, ERPs impose tight constraints on experimental design. However, it is not the case that this complexity in method should be compensated for by studying only the very simplest of linguistic units. The work summarized here has demonstrated the feasibility and utility of using ERPs to explore how people comprehend the most complex of linguistic stimuli, a (short) piece of discourse. It is the high temporal resolution and, above all, the selective sensitivity of brain potentials that help to keep track of the various processes involved in discourse-level language comprehension [cf. 12].  Van Berkum, J.J.A., Hagoort, P., & Brown, C.M. (1999). Semantic integration in sentences and discourse: Evidence from the N400. Journal of Cognitive Neuroscience, 11(6), 657-671.  Van Berkum, J.J.A., Zwitserlood, P., Hagoort, P., & Brown, C.M., (2003). When and how do listeners relate a sentence to the wider discourse? Evidence from the N400 effect. Cognitive Brain Research, 17, 701-718.  Nieuwland, M., & Van Berkum, J.J.A., (2004). Discourse context can completely overrule lexical-semantic violations: Evidence from the N400. Annual meeting of the Cognitive Neuroscience Society (CNS-2004), San Francisco, April 18-20, 2004.  Van Berkum, J.J.A., Brown, C.M., & Hagoort, P. (1999). Early referential context effects in sentence processing: Evidence from event-related brain potentials. Journal of Memory and Language, 41, 147-182.  Van Berkum, J.J.A., Brown, C.M., Hagoort, P., & Zwitserlood, P. (2003). Event-related brain potentials reflect referential ambiguity in spoken-language comprehension. Psychophysiology, 40, 235-248.  Van Berkum, J.J.A., Zwitserlood, P., Bastiaansen, M.C.M., Brown, C.M. & Hagoort, P. (2004). So who's "he" anyway? Differential ERP and ERSP effects of referential success, ambiguity and failure during spoken language comprehension. Annual meeting of the Cognitive Neuroscience Society (CNS-2004), San Francisco, April 18-20, 2004.  Van Berkum, J.J.A., Brown, C.M., & Hagoort, P. (1999). When does gender constrain parsing? Evidence from ERPs. Journal of Psycholinguistic Research, 28(5), 555-571.  Van Berkum, J.J.A., Hagoort, P. & Brown, C.M. (2000). The use of referential context and grammatical gender in parsing: A reply to Brysbaert and Mitchell (2000). Journal of Psycholinguistic Research, 29(5), 467-481.  Van Berkum, J.J.A., Brown, C.M., Hagoort, P., Zwitserlood, P., & Kooijman, V. (2002). Can people use discourse-level information to predict upcoming words in an unfolding sentence? Evidence from ERPs and self-paced reading. ICON-8, Porquerolles, France, September 9-15, 2002.  Van Berkum, J.J.A., Brown, C.M., Zwitserlood, P., Kooijman, V., & Hagoort, P., (2004). Anticipating upcoming words in discourse: Evidence from ERPs and self-paced reading. Accepted pending revisions, JEP:LMC.  Otten, M., & Van Berkum, J.J.A., (2004). Discourse-based lexical anticipation during language processing: Prediction or priming? Annual meeting of the Cognitive Neuroscience Society (CNS-2004), San Francisco, April 18-20, 2004.  Van Berkum, J.J.A. (2004). Sentence comprehension in a wider discourse: Can we use ERPs to keep track of things? In M. Carreiras & C. Clifton Jr. (Eds.), The on-line study of sentence comprehension: Eyetracking, ERP and beyond. Psychology Press. Notes. Partly funded by an NWO Vidi grant to JvB, by NWO grant 400-56-384 to CB & PH, and by a DFG grant to PZ & PH. Poster uses color coding. Example stimuli translated from Dutch. Poster and above papers can be obtained from www.josvanberkum.nl. Jos J.A. van Berkum, University of Amsterdam, Department of Psychology, Roetersstraat 15, 1018 WB Amsterdam, The Netherlands ([email protected] or [email protected]). Presented at the 14th Annual Meeting of the Society for Text and Discourse (STD-2004), Chicago, August 1-4, 2004.