Overview of IberAuTexTification at IberLEF 2024: Detection and Attribution of Machine-Generated Text on Languages of the Iberian Peninsula

• Autores: Areg Mikael Sarvazyan, José Ángel González, Francisco Manuel Rangel Pardo , Paolo Rosso , Marc Franco Salvador
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 73, 2024, págs. 421-434
• Idioma: inglés
• Títulos paralelos:
  • Resumen de IberAuTexTification en IberLEF 2024: Detección y Atribución de Texto Generado por Máquina en Idiomas de la Península Ibérica
• Enlaces
  • Texto completo
• Resumen
  • español

    Este artículo presenta un resumen de la tarea IberAuTexTification como parte del workshop IberLEF 2023 en el Iberian Languages Evaluation Forum, dentro del marco de la conferencia SEPLN 2024. IberAuTexTification extiende nuestra tarea previa, AuTexTification, en tres dimensiones: (i) más dominios, (ii) más idiomas de la Península Ibérica y (iii) LLMs más destacados. Esta tarea propone un escenario multilingüe, multi-dominio y multi-modelo consistente en dos subtareas. En la Subtarea 1, los participantes deben determinar si el autor de un texto es un humano o una máquina. En la Subtarea 2, los participantes deben atribuir un texto generado al modelo de lenguaje que lo generó. Nuestro dataset de IberAuTexTification contiene alrededor de 168.000 textos en seis idiomas (Inglés, Español, Portugués, Catalán, Vasco y Gallego) y siete dominios (chat, noticias, literatura, reseñas, tweets, wikipedia y artículos instructivos). Un total de 21 equipos participaron en la tarea, enviando 68 resultados, 54 para la Subtarea 1 y 14 para la Subtarea 2. En este artículo, presentamos la tarea IberAuTexTification, los sistemas enviados por los participantes y sus resultados.

  • English

    This paper presents the overview of the IberAuTexTification shared task as part of the IberLEF 2024 Workshop in Iberian Languages Evaluation Forum, within the framework of the SEPLN 2024 conference. IberAuTexTification extends our previous AuTexTification shared task in three dimensions: (i) more domains, (ii) more languages from the Iberian Peninsula, and (iii) more prominent LLMs. This shared task frames a multilingual, multi-domain, and multi-model setting consisting of two subtasks. For Subtask 1, participants have to determine whether a text’s author is a human or machine. For Subtask 2, participants have to attribute a machine-generated text to a large language model. Our IberAuTexTification dataset contains about 168,000 texts across six languages (English, Spanish, Portuguese, Catalan, Basque, and Galician) and seven domains (chat, news, literary, reviews, tweets, wikipedia, and how-to articles). A total of 21 teams participated in the task with 68 runs, 54 for Subtask 1 and 14 for Subtask 2. In this overview, we present the IberAuTexTification task, the submitted participating systems, and the results.

• Referencias bibliográficas
  • Achiam, J., S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. 2023. Gpt-4...
  • Antoun, W., B. Sagot, and D. Seddah. 2024. From text to source: Results in detecting large language model-generated content. In N. Calzolari,...
  • Bakhtin, A., S. Gross, M. Ott, Y. Deng, M. Ranzato, and A. Szlam. 2019. Real or fake? learning to discriminate machine from human generated...
  • Chiruzzo, L., S. M. Jiménez-Zafra, and F. Rangel. 2024. Overview of IberLEF 2024: Natural Language Processing Challenges for Spanish and other...
  • Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2020. Unsupervised...
  • de Gibert, O., K. Kharitonova, B. C. Figueras, J. Armengol-Estapé, and M. Melero. 2022. Sequence-to-sequence resources for catalan. arXiv...
  • Eloundou, T., S. Manning, P. Mishkin, and D. Rock. 2024. Gpts are gpts: Labor market impact potential of llms. Science, 384(6702):1306–1308.
  • Fernández García, J. and I. Segura-Bedmar. 2024. Human after all: Using transformers based models to identify automatically generated text....
  • Fivez, P., W. Daelemans, T. Van de Cruys, Y. Kashnitsky, S. Chamezopoulos, H. Mohammadi, A. Giachanou, A. Bagheri, W. Poelman, J. Vladika,...
  • Gonzalez-Agirre, A., M. Marimon, C. Rodriguez-Penagos, J. Aula-Blasco, I. Baucells, C. Armentano-Oller, J. Palomar-Giner, B. Kulebi, and M....
  • Guo, M., Z. Han, X. Wang, and J. Peng. 2024. Multidimensional text feature analysis: Unveiling the veil of automatically generated text. In...
  • Hans, A., A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, and T. Goldstein. 2024. Spotting LLMs with binoculars:...
  • Hasan, T., A. Bhattacharjee, M. S. Islam, K. Mubasshir, Y.-F. Li, Y.-B. Kang, M. S. Rahman, and R. Shahriyar. 2021. XLsum: Large-scale multilingual...
  • He, P., J. Gao, and W. Chen. 2022. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing....
  • Henderson, P., X. Li, D. Jurafsky, T. Hashimoto, M. A. Lemley, and P. Liang. 2023. Foundation models and fair use. Journal of Machine Learning...
  • Ippolito, D., D. Duckworth, C. Callison-Burch, and D. Eck. 2020. Automatic detection of generated text is easiest when humans are fooled....
  • Jiang, A. Q., A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al....
  • Joulin, A., E. Grave, P. Bojanowski, and T. Mikolov. 2017. Bag of tricks for efficient text classification. In M. Lapata, P. Blunsom, and...
  • Kasneci, E., K. Seßler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, et al. 2023. Chatgpt for good? on opportunities and challenges...
  • Köpf, A., Y. Kilcher, D. von Rütte, S. Anagnostidis, Z. R. Tam, K. Stevens, A. Barhoum, D. Nguyen, O. Stanley, R. Nagyfi, et al. 2024. Openassistant...
  • Kryscinski, W., N. Rajani, D. Agarwal, C. Xiong, and D. Radev. 2022. BOOKSUM: A collection of datasets for longform narrative summarization....
  • Ladhak, F., E. Durmus, C. Cardie, and K. McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In...
  • Lai, V., C. Nguyen, N. Ngo, T. Nguyen, F. Dernoncourt, R. Rossi, and T. Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple...
  • Liu, Y., T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu, et al. 2023. Summary of chatgpt/gpt-4 research and perspective...
  • Mitchell, E., Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability...
  • Molla, D., H. Zhan, X. He, and Q. Xu. 2023. Overview of the 2023 alta shared task: Discriminate between human-written and machine-generated...
  • Narayan, S., S. B. Cohen, and M. Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for...
  • Nasr, M., N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. 2023. Scalable...
  • Ortiz-Fuentes, J. 2022. Crawled Spanish books.
  • Ortiz Suárez, P. J., B. Sagot, and L. Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures....
  • Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton,...
  • Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas,...
  • Przybyla, P., N. Duran-Silva, and S. Egea-Gómez. 2023. I’ve seen things you machines wouldn’t believe: Measuring content predictability to...
  • Sanh, V. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of Thirty-third Conference on...
  • Sarvazyan, A. M., J. Á. González, and M. Franco-Salvador. 2024a. Genaios at SemEval-2024 task 8: Detecting machine-generated text by mixing...
  • Sarvazyan, A. M., J. Á. González, and M. Franco-Salvador. 2024b. Textmachina: Seamless generation of machine-generated text datasets. arXiv...
  • Sarvazyan, A. M., J. Á. González, M. Franco-Salvador, F. Rangel, B. Chulvi, and P. Rosso. 2023a. Overview of autextification at iberlef 2023:...
  • Sarvazyan, A. M., J. A. González, M. Franco- Salvador, and P. Rosso. 2023b. Supervised machine-generated text detectors: Family and scale...
  • Team, G., R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. 2023. Gemini: a family...
  • Touvron, H., L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. 2023. Llama...
  • Uchendu, A., T. Le, K. Shu, and D. Lee. 2020. Authorship attribution for neural text generation. In B. Webber, T. Cohn, Y. He, and Y. Liu,...
  • Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. Advances...
  • Wang, L., N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei. 2024a. Multilingual e5 text embeddings: A technical report.
  • Wang, Y., J. Mansurov, P. Ivanov, J. Su, A. Shelmanov, A. Tsvigun, O. Mohammed Afzal, T. Mahmoud, G. Puccetti, and T. Arnold. 2024b. SemEval-2024...
  • Wolf, T., L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von...
  • Xue, L., N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. 2021. mT5: A massively multilingual pre-trained...
  • Zellers, R., A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi. 2019. Defending against neural fake news. Advances in...
  • Zhang, Q., C. Gao, D. Chen, Y. Huang, Y. Huang, Z. Sun, S. Zhang, W. Li, Z. Fu, Y. Wan, and L. Sun. 2024. LLMas-a-coauthor: Can mixed human-written...
  • Zhang, X., J. Zhao, and Y. LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing...
  • Zubiaga, A., I. San Vicente, P. Gamallo, N. Aranberri, and A. Ezeiza. 2014. Overview of tweetlid: Tweet language identification at sepln 2014....