DOCUMENT CATEGORIZATION AND SENTIMENT ANALYSIS USING A MULTILINGUAL TRANSFORMER MODEL WITH MULTIPLE OUTPUTS
DOI:
https://doi.org/10.24867/22BE18MilunovicKeywords:
Natural language processing, text classification, sentiment analysis, multilingual transformer models, multi-output modelsAbstract
This paper presents a model for categorization and sentiment analysis of texts in one hundred languages using transformer neural networks. A way to optimize that model using model distillation is also described.
References
[1] GCP taksonomija https://cloud.google.com/natural-language/docs/categories (pristupljeno u septembru 2022.)
[2] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., ... & Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
[3] Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
[4] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
[5] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
[6] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
[7] GLUE Benchmark https://gluebenchmark.com/
[8] The Stanford Analysis Treebank https://nlp.stanford.edu/sentiment/
[9] Mediawiki https://github.com/barrust/mediawiki
[10] Wikipedia https://www.wikipedia.org/
[11] Huggingface https://huggingface.co/
[12] Tensorflow https://www.tensorflow.org/
[13] Pytorch https://pytorch.org/
[14] CommonCrawl https://commoncrawl.org/
[15] Agarwal, A., Dahleh, M., Shah, D., Sleeper, D., Tsai, A., & Wong, M. (2019). Zorro: A Model Agnostic System to Price Consumer Data. arXiv preprint arXiv:1906.02420.
[2] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., ... & Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
[3] Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
[4] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
[5] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
[6] Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
[7] GLUE Benchmark https://gluebenchmark.com/
[8] The Stanford Analysis Treebank https://nlp.stanford.edu/sentiment/
[9] Mediawiki https://github.com/barrust/mediawiki
[10] Wikipedia https://www.wikipedia.org/
[11] Huggingface https://huggingface.co/
[12] Tensorflow https://www.tensorflow.org/
[13] Pytorch https://pytorch.org/
[14] CommonCrawl https://commoncrawl.org/
[15] Agarwal, A., Dahleh, M., Shah, D., Sleeper, D., Tsai, A., & Wong, M. (2019). Zorro: A Model Agnostic System to Price Consumer Data. arXiv preprint arXiv:1906.02420.
Downloads
Published
2023-03-06
Issue
Section
Electrotechnical and Computer Engineering
