Purpose: This research compared the accuracy level of six algorithms based on the ROC method and the Confusion Matrix evaluation on data regarding public sentiments towards electric cars.
Methods: Data collection was conducted for data sourced from TikTok. Next, the data underwent text preprocessing (data cleaning and case folding) and text processing (stemming, tokenizing, stopword removal, word frequency, word relation, TF-IDF, scoring, and labeling). Modeling was then conducted using supervised (labeled) algorithms consisting of the Support Vector Machine (SVM), Decision Tree, Naive Bayes, Random Forest, K-Neighbor, and Logistic Regression. Finally, an evaluation was conducted (confusion matrix and ROC).
Result: The results revealed that the Decision Tree algorithm with the Confusion Matrix and ROC evaluation obtained the highest result of 87%. The algorithm with the lowest result is KNN, which has an accuracy of 56%. The classification result for the neutral sentiment has a percentage of 57.1%, followed by negative sentiment at 26.8% and positive sentiment at 16.1%. The KNN algorithm is suitable for large and low-dimensional data, SVM is suitable for data with many features and clear separation between classes, and Naive Bayes is efficient for large datasets with many low-quality features. Additionally, the Random Forest algorithm could overcome overfitting and unbalanced data. Logistic regression is also suitable for linear data without assuming a certain distribution. The Decision Tree algorithm is good for complex data as it provides a visual explanation of predictions. In this study, the Decision Tree algorithm obtained high results because it has the best characteristics and is a linear technique.
Novelty: This study found that based on the ROC method and the Confusion Matrix evaluation conducted, the Decision Tree algorithm is more accurate than the other algorithms studied.