Paper #2

Investigating the Accuracy of the GPT2 Algorithm in Classifying Identified Targets for an Intelligent Virtual Assistant

Shangying Guo and Jing Zhao

Abstract: Natural Language Understanding (NLU) is a branch of Natural Language Processing (NLP) that focuses on enabling computers to interpret human language with a level of understanding comparable to humans. NLU encompasses several tasks, including parsing sentences to understand grammatical structure, identifying word and phrase meanings, and determining user intent from natural language inputs. Many AI systems today—such as chatbots and virtual assistants—rely on NLU to accurately interpret and respond to user inputs in real time. This study addresses the challenge of accurately classifying user intents in multilingual intelligent virtual assistants a task critical for enhancing real-time human-computer interaction, by exploring the application of seven GPT-2 based models, leveraging their embedding matrices and tokenizers to design a robust intent-classification framework. The variation in the GPT-2 models in this study lies in the number of final layers and dimensional configurations used for classification. Through a large-scale case study with over one million utterances in 51 languages, the models were evaluated based on key metrics such as Accuracy, Precision, Recall, and F1-Score. Findings indicate that the GPT-256 model consistently achieved the highest values across these metrics, establishing it as the most accurate among the models tested. The GPT-256256 and GPT-128128 models followed closely, both of which showed competitive performance but with slightly lower accuracy than GPT-256. These results underscore the effectiveness of specific model configurations in improving NLU for virtual assistants, particularly in multilingual applications. The findings provide insight into optimizing AI systems for accurate goal classification, enhancing the ability of virtual assistants to understand and respond to diverse user inputs more precisely across languages, making them highly adaptable for global applications.

Keywords: Classification; GPT2; Natural Language Processing; Natural Language Understanding; Transformer; Virtual Assistant.