Merging Multiple Features to Evaluate the Content of Text Summary

• Autores: Samira Ellouze, Lamia Hadrich Belguith, Maher Jaoua
• Localización: Procesamiento del lenguaje natural, ISSN 1135-5948, Nº. 58, 2017, págs. 69-76
• Idioma: inglés
• Títulos paralelos:
  • Combinación varias Características para evaluar el contenido del resumen de texto
• Enlaces
  • Texto completo
• Resumen
  • español

    En este artículo proponemos un método que evalúa el contenido de un resumen de texto utilizando un enfoque de aprendizaje automático. Este método funciona combinando múltiples Características para construir modelos que predicen las puntuaciones PYRAMID para nuevos resúmenes. Hemos probado varios clasificadores individuales y "Ensemble Learning" para construir el mejor modelo. La evaluación del sistema de resumen se realiza utilizando el promedio de las puntuaciones de los resúmenes que se construyen a partir de cada sistema. Los resultados muestran que nuestro método ha logrado un buen rendimiento en la predicción de la puntuación de contenido para un resumen, así como para un sistema de resumen.

  • English

    In this paper, we propose a method that evaluates the content of a text summary using a machine learning approach. This method operates by combining multiple features to build models that predict the PYRAMID scores for new summaries. We have tested several single and "Ensemble Learning" classifiers to build the best model. The evaluation of summarization system is made using the average of the scores of summaries that are built from each system. The results show that our method has achieved good performance in predicting the content score for a summary as well as for a summarization system.

• Referencias bibliográficas
  • Breiman, L. 1996. Bagging predictors. Machine learning, 24:123–140.
  • Cohan, A, G. N. 2016. Revisiting summarization evaluation for scientific articles. In Proceedings of LREC conference, pages 806–813.
  • Ellouze, S., M. Jaoua, and L. Hadrich Belguith. 2013. An evaluation summary method based on a combination of content and linguistic metrics....
  • Ellouze, S., M. Jaoua, and L. Hadrich Belguith. 2016. Automatic evaluation of a summary’s linguistic quality. In Proceedings of NLDB 2016...
  • Finkel, J. R., T. Grenager, and C. Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling....
  • Giannakopoulos, G., V. Karkaletsis, G. Vouros, and P. Stamatopoulos. 2008. Summarization system evaluation revisited: N-gram graphs. ACM Trans....
  • Hovy, E., C. Lin, L. Zhou, and J. Fukumoto. 2006. Automated summarization evaluation with basic elements. In Proceedings of the LREC conference.
  • Kendall, M. G. 1938. A new measure of rank correlation. Biometrika, 30:81–89.
  • Klein, D. and C. D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the Annual Meeting on ACL, pages 423–430.
  • Kohavi, R. and G. H. John. 1997. Wrappers for feature subset selection. Artif. Intell., 97(12):273–324.
  • Kullback, S. and R. A. Leibler. 1951. On information and sufficiency. Ann. Math. Statist., 22(1):79–86.
  • Kuncheva, L. I. 2004. Combining pattern classifiers: Methods and Algorithms. WileyInterscience.
  • Lee, H., Y. Peirsman, A. Chang, N. Chambers, M. Surdeanu, and D. Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system...
  • Lin, C. 2004. Rouge: a package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL Workshop,...
  • Lin, J. 1991. Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theory, 37:145–151.
  • Louis, A. and A. Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267–300.
  • Nenkova, A. and R. J. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In HLT-NAACL, pages 145–152.
  • Pearson, K. 1895. Mathematical contributions to the theory of evolution, ii: Skew variation in homogeneous material. Philosophical Transactions...
  • Pitler, E., A. Louis, and A. Nenkova. 2010. Automatic evaluation of linguistic quality in multidocument summarization. In Proceedings of the...
  • Spearman, C. E. 1910. Correlation calculated from faulty data. British Journal of Psychology, 3:271–295.
  • Stolcke, A. 2002. Srilm-an extensible language modeling toolkit. In Proceedings of ICSLP, pages 257–286.
  • Toutanova, K., D. Klein, C. D. Manning, and Y. Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings...
  • Witten, I. H., E. Frank, and M. A. Hall. 2011. Data mining: Practical machine learning tools and techniques. Morgan Kaufmann Publishers Inc.
  • Wolpert, D. H. 1992. Stacked generalization. Neural Networks, 5:241–259.