Анализа перформанси Whisper модела у превођењу српског говора на енглески језик

Milana Vitković

doi:10.24867/

Energetika, elektronika i telekomunikacije

God. 41 Br. 08 (2026): Зборник радова Факултета техничких наука

Анализа перформанси Whisper модела у превођењу српског говора на енглески језик

Milana Vitković

PDF

DOI:: https://doi.org/10.24867/
Predato: September 7, 2026
Objavljeno: 2026-09-09

Apstrakt

Овај рад анализира методе аутоматског превођења српског говора на енглески језик, фокусирајући се на Whisper моделе и велике језичке моделе. Поређени су директни end-to-end приступ и каскадни систем који раздваја препознавање говора и превођење. Евалуација је спроведена на више тест скупова користећи BLEU, METEOR и WER метрике. Резултати показују да каскадни приступ са Whisper транскрипцијом и преводом помоћу великих језичких модела даје боље перформансе од директног превођења Whisper модела. Дообука Whisper модела за транскрипцију на српском значајно побољшава квалитет транскрипције и превода. Велики језички модели показују робусност чак и уз грешке у транскрипцији, што указује на њихов потенцијал у овој области. Као будући кораци препоручује се даља дообука и интеграција језичких модела ради унапређења
квалитета превођења.

Reference

[1] A. B´erard, O. Pietquin, C. Servan, and L. Besacier,
“Listen and translate: A proof of concept for end-to
end speech-to-text translation,” in NIPS Workshop on
End-to-End Learning for Speech and Audio
Processing, 2016.
[2] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z.
Chen, “Sequence-to-sequence models can directly
translate foreign speech,” in Interspeech. ISCA, 2017,
pp. 2625–2629.
[3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.
Jones, A. N. Gomez, Kaiser, and I. Polosukhin,
“Attention is all you need,” in Advances in Neural
Information Processing Systems (NeurIPS), 2017, pp.
5998–6008.
[4] A. Radford, J. W. Kim, T. Xu, G. Brockman, C.
McLeavey, and I. Sutskever, “Robust speech
recognition via large-scale weak supervision,” arXiv
preprint arXiv:2212.04356, 2022.
[5] Deng, L., & Yu, D. (2014). Deep learning: methods
and applications. Foundations and trends® in signal
processing, 7(3–4), 197-387.
[6] I. Goodfellow, Y. Bengio, and A. Courville, Deep
Learning. Cambridge, MA: MIT Press, 2016, ch. 9:
“Convolutional Networks”.
[7] Hendrycks, D. and Gimpel, K. Gaussian error linear
units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[8] Press, O. and Wolf, L. Using the output embedding to
improve language models. In Proceedings of the 15th
Conference of the European Chapter of the Associa
tion for Computational Linguistics: Volume 2, Short
Papers, pp. 157–163, Valencia, Spain, April 2017. As
sociation for Computational Linguistics. URL https:
//aclanthology.org/E17-2025.
[9] Child, R., Gray, S., Radford, A., and Sutskever, I. Gen
erating long sequences with sparse transformers.
arXiv preprint arXiv:1904.10509, 2019.
[10] Sennrich, R., Haddow, B., and Birch, A. Neural
machine translation of rare words with subword units.
arXiv preprint arXiv:1508.07909, 2015.
[11] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D.,
and Sutskever, I. Language models are unsupervised
multitask learners. 2019.
[12] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,
“Bleu: a method for automatic evaluation of machine
translation,” in Proceedings of the 40th Annual
Meeting of the Association for Computational
Linguistics, 2002, pp. 311–318.
[13] S. Banerjee and A. Lavie, “Meteor: An automatic
metric for mt evalua-tion with improved correlation
with human judgments,” in Proceedings of the ACL
Workshop on Intrinsic and Extrinsic Evaluation
Measures for Machine Translation and/or
Summarization, 2005, pp. 65–72.
[14] A. Ali and S. Renalds, “Word Error Rate Estimation
for Speech Recognition: e-WER,” in Proceedings of
the 56th Annual Meeting of the Association for
Computational Linguistics (Short Papers), pp. 20–24
Melbourne, Australia, July 15 - 20, 2018.
[15] V. Timmel, C. Paonessa, M. Vogel, D. Perruchoud
and R. Kakooe, “Fine-tuning Whisper on
Low-Resource Languages for Real-World
Applications,” arXiv preprint arXiv:2412.15726,
Apr. 2025.
[16] R. Ma, M. Qian, Y. Fathullah, S. Tang, M. Gales and
K. Knill, “Cross-Lingual Transfer Learning for Speech
Translation,” arXiv preprint, arXiv:2407.01130,
Feb. 2025.
[17] P. Peng, B. Yan, S. Watanabe and D. Harwath,
“Prompting the hidden talent of web-scale speech
models for zero-shot task generalization,” arXiv
preprint, arXiv:2305.11095v3, Aug. 2023.

[1] [1] A. B´erard, O. Pietquin, C. Servan, and L. Besacier,

[2] “Listen and translate: A proof of concept for end-to

[3] end speech-to-text translation,” in NIPS Workshop on

[4] End-to-End Learning for Speech and Audio

[5] Processing, 2016.

[6] [2] R. J. Weiss, J. Chorowski, N. Jaitly, Y. Wu, and Z.

[7] Chen, “Sequence-to-sequence models can directly

[8] translate foreign speech,” in Interspeech. ISCA, 2017,

[9] pp. 2625–2629.

[10] [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.

[11] Jones, A. N. Gomez, Kaiser, and I. Polosukhin,

[12] “Attention is all you need,” in Advances in Neural

[13] Information Processing Systems (NeurIPS), 2017, pp.

[14] 5998–6008.

[15] [4] A. Radford, J. W. Kim, T. Xu, G. Brockman, C.

[16] McLeavey, and I. Sutskever, “Robust speech

[17] recognition via large-scale weak supervision,” arXiv

[18] preprint arXiv:2212.04356, 2022.

[19] [5] Deng, L., & Yu, D. (2014). Deep learning: methods

[20] and applications. Foundations and trends® in signal

[21] processing, 7(3–4), 197-387.

[22] [6] I. Goodfellow, Y. Bengio, and A. Courville, Deep

[23] Learning. Cambridge, MA: MIT Press, 2016, ch. 9:

[24] “Convolutional Networks”.

[25] [7] Hendrycks, D. and Gimpel, K. Gaussian error linear

[26] units (gelus). arXiv preprint arXiv:1606.08415, 2016.

[27] [8] Press, O. and Wolf, L. Using the output embedding to

[28] improve language models. In Proceedings of the 15th

[29] Conference of the European Chapter of the Associa

[30] tion for Computational Linguistics: Volume 2, Short

[31] Papers, pp. 157–163, Valencia, Spain, April 2017. As

[32] sociation for Computational Linguistics. URL https:

[33] //aclanthology.org/E17-2025.

[34] [9] Child, R., Gray, S., Radford, A., and Sutskever, I. Gen

[35] erating long sequences with sparse transformers.

[36] arXiv preprint arXiv:1904.10509, 2019.

[37] [10] Sennrich, R., Haddow, B., and Birch, A. Neural

[38] machine translation of rare words with subword units.

[39] arXiv preprint arXiv:1508.07909, 2015.

[40] [11] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D.,

[41] and Sutskever, I. Language models are unsupervised

[42] multitask learners. 2019.

[43] [12] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu,

[44] “Bleu: a method for automatic evaluation of machine

[45] translation,” in Proceedings of the 40th Annual

[46] Meeting of the Association for Computational

[47] Linguistics, 2002, pp. 311–318.

[48] [13] S. Banerjee and A. Lavie, “Meteor: An automatic

[49] metric for mt evalua-tion with improved correlation

[50] with human judgments,” in Proceedings of the ACL

[51] Workshop on Intrinsic and Extrinsic Evaluation

[52] Measures for Machine Translation and/or

[53] Summarization, 2005, pp. 65–72.

[54] [14] A. Ali and S. Renalds, “Word Error Rate Estimation

[55] for Speech Recognition: e-WER,” in Proceedings of

[56] the 56th Annual Meeting of the Association for

[57] Computational Linguistics (Short Papers), pp. 20–24

[58] Melbourne, Australia, July 15 - 20, 2018.

[59] [15] V. Timmel, C. Paonessa, M. Vogel, D. Perruchoud

[60] and R. Kakooe, “Fine-tuning Whisper on

[61] Low-Resource Languages for Real-World

[62] Applications,” arXiv preprint arXiv:2412.15726,

[63] Apr. 2025.

[64] [16] R. Ma, M. Qian, Y. Fathullah, S. Tang, M. Gales and

[65] K. Knill, “Cross-Lingual Transfer Learning for Speech

[66] Translation,” arXiv preprint, arXiv:2407.01130,

[67] Feb. 2025.

[68] [17] P. Peng, B. Yan, S. Watanabe and D. Harwath,

[69] “Prompting the hidden talent of web-scale speech

[70] models for zero-shot task generalization,” arXiv

[71] preprint, arXiv:2305.11095v3, Aug. 2023.