Welcome to the site of the shared task FACT: Factuality Analysis and Classification Task, a task to classify events in Spanish texts, according to their factuality status. This task is part of IberLEF 2020.
The FACT shared task is organized by Grupo PLN-InCo (UdelaR - Uruguay) and GRIAL (UB-UAB-UDL, Spain).
News
- June 17, 2020. Final results published in the results section.
The guidelines for the working notes are described at the end of the overview page. - June 3, 2020. A description of the baseline for each task was published at "Learn the Details > Evaluation".
The guidelines for the working notes are described at the end of the overview page. - May 25, 2020. Results submission deadline has been extended to June 17.
- May 20, 2020. Test data for tasks 1 and 2 is ready to download. You can get it by downloading the Public Data in the "Participate > Files" section. You can now upload your results for the tasks in the "Participate > Submit / View Results" section. Please use the data format specified in "Learn the Details > Evaluation".
Due to issues with Codalab we are not able to show the results for task 2 in the leaderboard, so during this phase the leaderboard will only show the results for task 1. We will publish the leaderboard table for task 2 when the competition ends. In the meantime, you will be able to see your own results for task 2 using the "View scoring output log" for your submission. - March 19, 2020. The training data is available at the FACT@IberLEF 2020 CodaLab competition page.
- March 11, 2020. We will be using the platform CodaLab for handling the submissions, please register in the FACT@IberLEF 2020 CodaLab competition page. Also, please remember to join the Google Group factiberlef2020 for the latest updates.
- February 5, 2020. Welcome to FACT at IberLEF 2020. The site for the previous edition of the competition can be checked here: FACT at IberLEF 2019.
Introduction
In order to analyze event references in texts, it is crucial to determine whether they are presented as having taken place or as potential or not accomplished events. This information can be used for different applications like Question Answering, Information Extraction, or Incremental Timeline Construction.
Despite its centrality for Natural Language Understanding, this task has been underresearched, with the work by Saurí and Pustejovsky (2009) as a reference for English and Wonsever et al. (2009) for Spanish. The bottleneck to advance on this task has usually been the lack of annotated resources, together with its inherent difficulty. Currently PLN-InCo and GRIAL both have ongoing research projects on this topic, which are producing and will produce such annotated resources. This makes the proposal of this task even more interesting.
Task description
For this year's edition of the task, we propose the following two subtasks:
Sub task 1: Factuality Determination
Factuality is understood, following Sauri (2008), as the category that determines the factual status of events, that is, whether events are presented or not as certain. The goal of this task is the determination of the status of verb events with respect to factuality in Spanish texts.
In this task facts are not verified in regard to the real world, just assessed with respect to how they are presented by the source (in this case the writer), that is, the commitment of the source to the truth-value of the event. In this sense, the task could be conceived as a core procedure for other tasks such as fact-checking and fake-news, making it possible, in future tasks, to compare what is narrated in the text (fact tagging) to what is happening in the world (fact-checking and fake-news).
We establish three possible categories:
The systems will have to automatically propose a factual tag for each event in the text. for this sub task, the events are already annotated in the texts. The structure of the tags used in the annotation is the following:
<event factuality=”F”>word</event>
For example, for the input text:
De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego <event>ha</event> <event>vuelto</event> a la normalidad, aunque <event>mantiene</event> <event>explosiones</event> moderadas, por lo que no <event>descarta</event> una nueva <event>erupción</event>.
The systems outcome should be:
De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego <event factuality="F">ha</event> <event factuality="F">vuelto</event> a la normalidad, aunque <event factuality="F">mantiene</event> <event factuality="F">explosiones</event> moderadas, por lo que no <event factuality="CF">descarta</event> una nueva <event factuality="U">erupción</event>.
The performance of this subtask will be measured against the evaluation corpus using these metrics:
The main score for evaluating the submissions will be Macro-F1.
Sub task 2: Event Identification
The recognition of noun events presents different challenges (Saurí et al., 2005; Wonserver et al., 2012), on the one hand, identifying the nouns that transmit eventive information, such as war or construction, and, on the other hand, disambiguating those nouns that are eventive in certain contexts (conversaremos durante la cena) and not eventive in others (la cena está servida).
In this task, the participants will receive text with no annotations:
De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego ha vuelto a la normalidad, aunque mantiene explosiones moderadas, por lo que no descarta una nueva erupción.
and have to identify verbal and noun events:
De acuerdo con el Instituto Nacional de Sismología, Vulcanología, Meteorología e Hidrología (Insivumeh), el volcán de Fuego <event>ha</event> <event>vuelto</event> a la normalidad, aunque <event>mantiene</event> <event>explosiones</event> moderadas, por lo que no <event>descarta</event> una nueva <event>erupción</event>.
The performance of this subtask will be measured against the evaluation corpus using these metrics:
The main score for evaluating the submissions will be Macro-F1.
Data
Available in the FACT@IberLEF 2020 CodaLab competition page.
Important Dates
Results
The following are the results for Subtask 1:Participant | Macro-F1 | Macro-Precision | Macro-Recall | Accuracy |
---|---|---|---|---|
t.romani | 60.7 | 61.2 | 60.4 | 84.8 |
guster | 59.3 | 62.1 | 57.4 | 83.1 |
accg14 | 55.0 | 55.6 | 54.5 | 79.8 |
trinidadg | 53.6 | 55.8 | 52.0 | 80.6 |
premjithb | 39.3 | 45.5 | 37.6 | 71.6 |
garain | 36.6 | 35.7 | 39.4 | 59.9 |
FACT_baseline | 24.6 | 25.4 | 25.1 | 52.4 |
Participant | F1 | Precision | Recall |
---|---|---|---|
trinidadg | 86.5 | 95.1 | 79.3 |
FACT_baseline | 59.7 | 60.3 | 59.1 |
Contact
If you want to participate in this task or have any question, please join the Google Group factiberlef2020. We will be sharing news and important information about the task in that group.FACT shared task is organized by:
Bibliography
(Rosá et al., 2019) Rosá, A., Castellón, I., Chiruzzo, L., Curell, H., Etcheverry, M., Fernández, A., Vázquez, G., Wonsever, D. (2019). Overview of FACT at IberLEF 2019.
(Alonso et al., 2018) Alonso, L., I. Castellón, H, Curell, A. Fernández-Montraveta, S. Oliver, G. Vázquez (2018). "Proyecto TAGFACT: Del texto al conocimiento. Factualidad y grados de certeza en español", Procesamiento del Lenguaje Natural, 61, p. 151-154. ISSN: 1135-5948
(Saurí 2008) Saurí, Roser. 2008. A Factuality Profiler for Eventualities in Text. Ph.D. Thesis. Brandeis University.
(Saurí and Pustejovsky 2009) Saurí, Roser and James Pustejovsky. 2009. FactBank: A Corpus Annotated with Event Factuality. In: Language Resources and Evaluation.
(Wonsever et al., 2009) Wonsever, D., Malcuori, M., & Rosá Furman, A. (2009). Factividad de los eventos referidos en textos. Reportes Técnicos 09-12, Pedeciba.
(Wonsever et al., 2016) Wonsever, D., Rosá, A., & Malcuori, M. (2016). Factuality Annotation and Learning in Spanish Texts. In LREC.