AN ADAPTIVE MACHINE LEARNING AND DECEPTION-DRIVEN FRAMEWORK FOR ADVANCED PERSISTENT THREAT DETECTION IN REMOTE DESKTOP PROTOCOL SESSIONS

Abstract

Advanced Persistent Threats (APTs) pose a significant challenge to modern cybersecurity infrastructures due to their stealthy behavior, prolonged presence, and multi-stage attack strategies. Among the various attack phases, lateral movement remains particularly difficult to detect, as attackers often exploit legitimate services such as Remote Desktop Protocol (RDP) to propagate within enterprise networks. This paper proposes an integrated cyber defence framework, termed CMCOADL-TDC, which combines machine learning-based detection with an adaptive deception-driven response mechanism to identify and mitigate malicious RDP activities associated with APT attacks.

The proposed methodology utilizes Windows event logs related to RDP sessions, capturing authentication behavior, session characteristics, and access anomalies. Multiple datasets containing benign and malicious traces are consolidated to enhance data diversity and improve generalization. After preprocessing, feature extraction and selection are applied to represent lateral movement patterns effectively. Several supervised learning classifiers Logistic Regression, Random Forest, Gaussian Naive Bayes, Feedforward Neural Network, Decision Tree, and AdaBoost are evaluated using a 10-fold cross-validation strategy. An ensemble learning approach with weighted voting is employed to further improve detection reliability.

Experimental results demonstrate that the proposed framework achieves consistently high performance across multiple attack classes. The AdaBoost classifier outperforms other models, attaining an accuracy of up to 99.9%, precision of 99.9%, recall of 98%, and an F1 score of 0.99. Furthermore, the deception-driven belief update mechanism enhances defensive adaptability by dynamically responding to attacker behavior, reducing false positives and limiting lateral movement. The results confirm that the proposed CMCOADL-TDC framework is effective, robust, and suitable for real-world APT detection scenarios.