Resumen de AuTexTification en IberLEF 2023: Detección y Atribución de Texto Generado Automáticamente en Múltiples Dominios

• Autores: Berta Chulvi, Paolo Rosso , Areg Mikael Sarvazyan, José Ángel González, Marc Franco Salvador, Francisco Manuel Rangel Pardo
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 71, 2023, págs. 275-288
• Idioma: español
• Títulos paralelos:
  • Overview of AuTexTification at IberLEF 2023: Detection and Attribution of Machine-Generated Text in Multiple Domains
• Enlaces
  • Texto completo
• Dialnet Métricas: 2 Citas
• Resumen
  • Multiple

    Este artículo presenta un resumen de la tarea AuTexTification como parte del workshop IberLEF 2023 sobre el Iberian Languages Evaluation Forum, en el marco de la conferencia SEPLN 2023. AuTexTification consta de dos subtareas: en la Subtarea 1, los participantes tuvieron que determinar si un texto fue escrito por un humano o generado por un modelo de lenguaje masivo. Para la Subtarea 2, los participantes debían atribuir un texto generado automáticamente a uno de seis modelos de generación de texto diferentes. El conjunto de datos AuTexTification contiene más de 160.000 textos en dos idiomas (inglés y español) y cinco dominios (tweets, reseñas, noticias, legislación y artículos instructivos). Un total de 114 equipos se inscribieron para participar, de los cuales 36 enviaron 175 resultados y 20 de ellos enviaron artículos. En este artículo, presentamos el conjunto de datos y la tarea AuTexTification, los sistemas enviados por los participantes y sus resultados.

  • English

    This paper presents the overview of the AuTexTification shared task as part of the IberLEF 2023 Workshop in Iberian Languages Evaluation Forum, within the framework of the SEPLN 2023 conference. AuTexTification consists of two subtasks: for Subtask 1, participants had to determine whether a text is human-authored or has been generated by a large language model. For Subtask 2, participants had to attribute a machine-generated text to one of six different text generation models. Our AuTexTification 2023 dataset contains more than 160.000 texts across two languages (English and Spanish) and five domains (tweets, reviews, news, legal, and how-to articles). A total of 114 teams signed up to participate, of which 36 sent 175 runs, and 20 of them sent their working notes. In this overview, we present the AuTexTification dataset and task, the submitted participating systems, and the results.

• Referencias bibliográficas
  • Abburi, H., M. Suesserman, N. Pudota, B. Veeramani, E. Bowen, and S. Bhattacharya. 2023. Generative ai text classification using ensemble...
  • Alonso, L., J. A. Gonzalo, A. M. Fernández- Pampillón, M. Fernández, and M. V. Escadell-Vidal. 2023. Using linguistic knowledge for automated...
  • Bakhtin, A., S. Gross, M. Ott, Y. Deng, M. Ranzato, and A. Szlam. 2019. Real or fake? learning to discriminate machine from human generated...
  • Barbieri, F., L. Espinosa Anke, and J. Camacho-Collados. 2022. Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond....
  • Benjamin, M., P. Gagnon, N. Rostamzadeh, C. Pal, Y. Bengio, and A. Shee. 2019. Towards standardization of data licenses: The montreal data...
  • Blei, D. M., A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993– 1022.
  • Brown, T., B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. 2020. Language...
  • Chalkidis, I., M. Fergadiotis, and I. Androutsopoulos. 2021. Multieurlex – a multi-lingual and multi-label legal document classification dataset...
  • Chung, H. W., T. Fevry, H. Tsai, M. Johnson, and S. Ruder. 2020. Rethinking embedding coupling in pre-trained language models. In International...
  • Conneau, A., K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, é. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2020. Unsupervised...
  • Contractor, D., D. McDuff, J. K. Haines, J. Lee, C. Hines, B. Hecht, N. Vincent, and H. Li. 2022. Behavioral use licensing for responsible...
  • Cortes, C. and V. Vapnik. 1995. Supportvector networks. Machine learning, 20:273–297.
  • Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding....
  • Dietterich, T. G. and G. Bakiri. 1994. Solving multiclass learning problems via errorcorrecting output codes. Journal of artificial intelligence...
  • Eloundou, T., S. Manning, P. Mishkin, and D. Rock. 2023. Gpts are gpts: An early look at the labor market impact potential of large language...
  • Fandiño, A. G., J. A. Estapé, M. Pàmies, J. L. Palao, J. S. Ocampo, C. P. Carrino, C. A. Oller, C. R. Penagos, A. G. Agirre, and M. Villegas....
  • González, M. D., E. Martínez-Cámara, M. Martín-Valdivia, and L. López. 2014. Cross-domain sentiment analysis using spanish opinionated words....
  • Hasan, T., A. Bhattacharjee, M. S. Islam, K. Mubasshir, Y.-F. Li, Y.-B. Kang, M. S. Rahman, and R. Shahriyar. 2021. XLsum: Large-scale multilingual...
  • He, P., J. Gao, and W. Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradientdisentangled embedding sharing....
  • Hochreiter, S. and J. Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780.
  • Holtzman, A., J. Buys, L. Du, M. Forbes, and Y. Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning...
  • Joulin, A., E. Grave, P. Bojanowski, and T. Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference...
  • Kashnitsky, Y., D. Herrmannova, A. de Waard, G. Tsatsaronis, C. C. Fennell, and C. Labbe. 2022. Overview of the dagpap22 shared task on detecting automatically...
  • Ladhak, F., E. Durmus, C. Cardie, and K. McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In...
  • LeCun, Y., B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. 1989. Handwritten digit recognition with a back-propagation...
  • Leis, A., F. Ronzano, M. A. Mayer, L. I. Furlong, and F. Sanz. 2019. Detecting signs of depression in tweets in spanish: Behavioral and linguistic...
  • Liu, Y., T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu, et al. 2023. Summary of chatgpt/gpt-4 research and perspective...
  • Liu, Y., M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized...
  • Maronikolakis, A., H. Schutze, and M. Stevenson. 2021. Identifying automatically generated headlines using transformers. In Proceedings of...
  • McAuley, J. and J. Leskovec. 2013. Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of the...
  • Mitchell, E., Y. Lee, A. Khazatsky, C. D. Manning, and C. Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability...
  • Mueller, T., G. Perez-Torro, and M. Franco- Salvador. 2022. Few-shot learning with siamese networks and label tuning. In Proceedings of the...
  • Naji, I. 2012. TSATC: Twitter Sentiment Analysis Training Corpus. In thinknook. Narayan, S., S. B. Cohen, and M. Lapata. 2018. Don’t give...
  • Ouyang, L., J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Gray, J. Schulman, J. Hilton, F. Kelton,...
  • Pedregosa, F., G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas,...
  • Pizarro, J. 2019. Using n-grams to detect bots on twitter. In Conference and Labs of the Evaluation Forum.
  • Przybyla, P., N. Duran-Silva, and S. Egea- Gomez. 2023. I’ve seen things you machines wouldn’t believe: Measuring content predictability to...
  • Radford, A., J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI.
  • Rangel, F., M. Franco-Salvador, and P. Rosso. 2018. A low dimensionality representation for language variety identification. In Computational...
  • Sarvazyan, A. M., J. Gonzalez, M. Franco- Salvador, and P. Rosso. 2023. Supervised machine-generated text detectors: Family and scale matters....
  • Scao, T. L., A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagne, A. S. Luccioni, F. Yvon, M. Galle, et al. 2022. Bloom: A 176b-parameter...
  • Schuster, M. and K. K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673–2681.
  • Scialom, T., P.-A. Dray, S. Lamprier, B. Piwowarski, and J. Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of...
  • Seger, E., A. Ovadya, B. Garfinkel, D. Siddarth, and A. Dafoe. 2023. Democratising ai: Multiple meanings, goals, and methods. arXiv preprint arXiv:2303.12642.
  • Shamardina, T., V. Mikhailov, D. Chernianskii, A. Fenogenova, M. Saidov, A. Valeeva, T. Shavrina, I. Smurov, E. Tutubalina, and E. Artemova....
  • Spitale, G., N. Biller-Andorno, and F. Germani. 2023. Ai model gpt-3 (dis)informs us better than humans. Science Advances, 9(26):eadh1850.
  • Touvron, H., T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Roziere, N. Goyal, E. Hambro, F. Azhar, et al. 2023. Llama:...
  • Uchendu, A., T. Le, K. Shu, and D. Lee. 2020. Authorship attribution for neural text generation. In Proceedings of the 2020 Conference on...
  • Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. Advances...
  • Widder, D. G., D. Nafus, L. Dabbish, and J. Herbsleb. 2022. Limits and possibilities for “ethical ai” in open source: A study of deepfakes....
  • Wolf, T., L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. 2020. Transformers: State-of-the-art...
  • Zellers, R., A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi. 2019a. Defending against neural fake news. Advances in...
  • Zellers, R., A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi. 2019b. Defending against neural fake news. Advances in...
  • Zhang, S., S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. 2022a. Opt: Open pre-trained transformer...
  • Zhang, X., Y. Malkov, O. Florez, S. Park, B. McWilliams, J. Han, and A. El-Kishky. 2022b. Twhin-bert: A socially-enriched pre-trained language...