Welcome to the site of the shared task FACT: Factuality Analysis and Classification Task, a task to classify events in Spanish texts, according to their factuality status. This task is part of IberLEF 2020.

The FACT shared task is organized by Grupo PLN-InCo (UdelaR - Uruguay) and GRIAL (UB-UAB-UDL, Spain).

News

Introduction

In order to analyze event references in texts, it is crucial to determine whether they are presented as having taken place or as potential or not accomplished events. This information can be used for different applications like Question Answering, Information Extraction, or Incremental Timeline Construction.

Despite its centrality for Natural Language Understanding, this task has been underresearched, with the work by Saurí and Pustejovsky (2009) as a reference for English and Wonsever et al. (2009) for Spanish. The bottleneck to advance on this task has usually been the lack of annotated resources, together with its inherent difficulty. Currently PLN-InCo and GRIAL both have ongoing research projects on this topic, which are producing and will produce such annotated resources. This makes the proposal of this task even more interesting.

Task description

For this year's edition of the task, we propose the following two subtasks:

Sub task 1: Factuality Determination

Factuality is understood, following Sauri (2008), as the category that determines the factual status of events, that is, whether events are presented or not as certain. The goal of this task is the determination of the status of verb events with respect to factuality in Spanish texts.

In this task facts are not verified in regard to the real world, just assessed with respect to how they are presented by the source (in this case the writer), that is, the commitment of the source to the truth-value of the event. In this sense, the task could be conceived as a core procedure for other tasks such as fact-checking and fake-news, making it possible, in future tasks, to compare what is narrated in the text (fact tagging) to what is happening in the world (fact-checking and fake-news).

We establish three possible categories:

  • Fact (F): current and past situations in the world that are presented as real.
  • Counterfact (CF): current and past situations that the writer presents as not having happened.
  • Undefined (U): Possibilities, future situations, predictions, hypothesis and other options. Situations presented as uncertain since the writer does not commit openly to the truth-value either because they have not happened yet or because the author does not know.

    • The systems will have to automatically propose a factual tag for each event in the text. for this sub task, the events are already annotated in the texts. The structure of the tags used in the annotation is the following:

    <event factuality=”F”>word</event>

    For example, for the input text:

    De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego <event>ha</event> <event>vuelto</event> a la normalidad, aunque <event>mantiene</event> <event>explosiones</event> moderadas, por lo que no <event>descarta</event> una nueva <event>erupción</event>.

    The systems outcome should be:

    De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego <event factuality="F">ha</event> <event factuality="F">vuelto</event> a la normalidad, aunque <event factuality="F">mantiene</event> <event factuality="F">explosiones</event> moderadas, por lo que no <event factuality="CF">descarta</event> una nueva <event factuality="U">erupción</event>.

    The performance of this subtask will be measured against the evaluation corpus using these metrics:

  • Precision, Recall and F1 score for each category.
  • Macro-F1.
  • Global accuracy.

    • The main score for evaluating the submissions will be Macro-F1.

    Sub task 2: Event Identification

    The recognition of noun events presents different challenges (Saurí et al., 2005; Wonserver et al., 2012), on the one hand, identifying the nouns that transmit eventive information, such as war or construction, and, on the other hand, disambiguating those nouns that are eventive in certain contexts (conversaremos durante la cena) and not eventive in others (la cena está servida).

    In this task, the participants will receive text with no annotations:

    De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego ha vuelto a la normalidad, aunque mantiene explosiones moderadas, por lo que no descarta una nueva erupción.

    and have to identify verbal and noun events:

    De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego <event>ha</event> <event>vuelto</event> a la normalidad, aunque <event>mantiene</event> <event>explosiones</event> moderadas, por lo que no <event>descarta</event> una nueva <event>erupción</event>.

    The performance of this subtask will be measured against the evaluation corpus using these metrics:

  • Precision, Recall and F1 score for event detection.

    • The main score for evaluating the submissions will be Macro-F1.

    Data

    Available in the FACT@IberLEF 2020 CodaLab competition page.

    Important Dates

  • March 11th, 2020: team registration page.
  • March 18th, 2020: training data.
  • May 20th, 2020: test data.
  • May 27th June 17th, 2020: publication of results.
  • June 5th July 15th, 2020: paper submission.
  • June 12th August 5th, 2020: notification of acceptance.
  • June 19th August 26th, 2020: camera ready paper submission.
  • September, 2020: IberLEF 2020 workshop.

    • Results

    The following are the results for Subtask 1:
    Participant Macro-F1 Macro-Precision Macro-Recall Accuracy
    t.romani60.761.260.484.8
    guster59.362.157.483.1
    accg1455.055.654.579.8
    trinidadg53.655.852.080.6
    premjithb39.345.537.671.6
    garain36.635.739.459.9
    FACT_baseline24.625.425.152.4
    The following are the results for Subtask 2:
    Participant F1 Precision Recall
    trinidadg86.595.179.3
    FACT_baseline59.760.359.1

    Contact

    If you want to participate in this task or have any question, please join the Google Group factiberlef2020. We will be sharing news and important information about the task in that group.

    FACT shared task is organized by:

  • PLN-InCo (FIng, UdelaR, Uruguay)
  • GRIAL (UB-UAB-UDL, Spain)

    • Bibliography

    (Rosá et al., 2019) Rosá, A., Castellón, I., Chiruzzo, L., Curell, H., Etcheverry, M., Fernández, A., Vázquez, G., Wonsever, D. (2019). Overview of FACT at IberLEF 2019.

    (Alonso et al., 2018) Alonso, L., I. Castellón, H, Curell, A. Fernández-Montraveta, S. Oliver, G. Vázquez (2018). "Proyecto TAGFACT: Del texto al conocimiento. Factualidad y grados de certeza en español", Procesamiento del Lenguaje Natural, 61, p. 151-154. ISSN: 1135-5948

    (Saurí 2008) Saurí, Roser. 2008. A Factuality Profiler for Eventualities in Text. Ph.D. Thesis. Brandeis University.

    (Saurí and Pustejovsky 2009) Saurí, Roser and James Pustejovsky. 2009. FactBank: A Corpus Annotated with Event Factuality. In: Language Resources and Evaluation.

    (Wonsever et al., 2009) Wonsever, D., Malcuori, M., & Rosá Furman, A. (2009). Factividad de los eventos referidos en textos. Reportes Técnicos 09-12, Pedeciba.

    (Wonsever et al., 2016) Wonsever, D., Rosá, A., & Malcuori, M. (2016). Factuality Annotation and Learning in Spanish Texts. In LREC.