Fraud in digital transactions has become a serious issue threatening the security and integrity of the fintech and e-commerce sectors. To address this problem, machine learning technology has emerged as an effective solution for automatically detecting anomalies and fraudulent transactions. This study aims to analyze the application of machine learning algorithms, specifically Support Vector Machine (SVM), Random Forest, and Ensemble Learning, in detecting fraud in digital transactions. The research adopts a quantitative approach with experimentation, testing the effectiveness of the three algorithms using a digital transaction dataset consisting of both fraudulent and non-fraudulent transactions. The results show that the Random Forest algorithm performs the best in terms of accuracy and recall, followed by Ensemble Learning, which enhances fraud detection performance by combining multiple prediction models. Meanwhile, SVM demonstrates satisfactory performance but is prone to overfitting issues when handling large and complex datasets. The study also finds that the problem of imbalanced data can affect model accuracy, and data balancing techniques such as oversampling are required to improve fraud detection performance. Overall, the findings suggest that machine learning, particularly Random Forest and Ensemble Learning algorithms, can be relied upon to improve fraud detection in digital transactions. However, challenges such as model interpretability and the need for periodic algorithm updates still need to be addressed to enhance the effectiveness of fraud prevention systems in countering the ever-evolving nature of fraud.