Approach to fine-tuning large language models on Ukrainian text corpora
DOI:
https://doi.org/10.34121/1028-9763-2026-2-46-57Keywords:
adaptive data selection, chatbot, natural language processing, ChatGPT, objective function, sample sizeAbstract
The paper proposes an approach to fine-tuning large language models on Ukrainian text corpora, which is based on the formalization of the training data selection process and its structuring using a set of parameters. A model for constructing an optimal training subset has been developed, taking into account data quality, domain relevance, structural diversity, and the level of annotation. The results show that the use of a multi-criteria objective function allows controlled formation of the training dataset under computational constraints. An adaptive mechanism for selecting text fragments is proposed, which combines statistical analysis of language patterns, consideration of dialogue scenarios, and data quality evaluation. The paper also justifies the need to use specialized Ukrainian-language corpora to improve the performance of language models in practical tasks. A method for integrating the fine-tuned model into a chatbot system has been developed, including dialogue context management, access to external knowledge, and response quality control. The proposed architecture ensures coordinated operation of system components and improves model adaptability to real user interaction. A computational experiment confirmed the effectiveness of the proposed approach: perplexity decreased by 18–22 %, answer accuracy increased by 15 %, and semantic coherence improved by 17 %. In addition, a reduction in response generation time has been observed, which positively affects the quality of user interaction. The obtained results demonstrate the feasibility of using the proposed approach for building intelligent chatbots focused on the Ukrainian language environment and confirm the prospects of further research in adapting language models to national language resources. Таbl.: 6. Fig.: 1. Refs.: 14 titles.
References
1. Lloret A. Can natural language processing technologies help the digital transformation of local public administrations? CEUR Workshop Proceedings. 2024. Vol. 3797. URL: https://ceur-ws.org/Vol-3797/paper13.pdf.
2. Brown T., Mann B., Ryder N. et al. Language models are few-shot learners / H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (eds.). Advances in Neural Information Proc. Systems. 2020. Vol. 33. P. 1877–1901. Curran Associates, Inc. URL: https://dl.acm.org/doi/abs/10.5555/3495724.3495883.
3. Virtanen A., Kanerva J., Ilo R. et al. Multilingual is not enough: BERT for Finnish, 2019. arXiv. DOI: https://doi.org/10.48550/arXiv.1912.07076.
4. Howard J., Ruder S. Universal language model fine-tuning for text classification. 2018. arXiv. DOI: https://doi.org/10.48550/arXiv.1801.06146.
5. Chronopoulou A., Peters M., Dodge J. Efficient hierarchical domain adaptation for pretrained language models. 2021. arXiv. URL: https://arxiv.org/abs/2112.08786.
6. Wang S., Fu Y., Kim J. Toward construction-specialized, small language models: The interplay of domain adaptation, model scale and data volume. Advanced Engineering Informatics. 2026. Vol. 69. P. 104035. DOI: https://doi.org/10.1016/j.aei.2025.104035.
7. Syvokon O., Romanyshyn M., Kyslyi R. The UNLP 2024 shared task on fine-tuning large language models for Ukrainian. Proc. of the Third Ukrainian Natural Language Processing Workshop (UNLP) @ LREC-COLING. 2024. P. 67-74. ELRA. URL: https://aclanthology.org/2024.unlp-1.9/.
8. Kiulian A., Polishko A., Khandoga M. et al. From bytes to borsch: Fine-tuning Gemma and Mistral for the Ukrainian language representation. 2024. arXiv. DOI: https://doi.org/10.48550/arXiv.2404.09138.
9. Brown T., Mann B., Ryder N. et al. / H. Larochelle et al. (eds.). Advances in Neural Information Processing Systems. 2020. Vol. 33. P. 1877–1901. Curran Associates, Inc. DOI: https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
10. Adiwardana D., Luong M.-T., So D.R. et al. Towards a human-like open-domain chatbot. 2020. arXiv. DOI: https://doi.org/10.48550/arXiv.2001.09977.
11. Kryazhych O., Ivanov I., Isak L., Babak O. Development of an approach to chat-bot personalization with generative artificial intelligence when realize an online assistant. Technology Audit and Production Reserves. 2025. Vol. 3 (2 (83)). P. 12–19. DOI: https://doi.org/10.15587/2706-5448.2025.326914.
12. Dodge J., Sap M., Marasović A. et al. / M.-F. Moens, X. Huang, L. Specia, S.W.-t. Yih (eds.). Proc. of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021). 2021. P. 1286–1305. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/2021.emnlp-main.98.
13. Gao L., Biderman S., Black S. et al. The Pile: An 800GB dataset of diverse text for language modeling. 2020. arXiv. DOI: https://doi.org/10.48550/arXiv.2101.00027.
14. Roller S., Dinan E., Goyal N. et al. Recipes for building an open-domain chatbot / P. Merlo, J. Tiedemann, R. Tsarfaty (eds.). Proc. of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021). 2021. P. 300–325. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/2021.eacl-main.24.
135
64
