INTELLIGENT ORCHESTRATION AND ENERGY-AWARE MICROSERVICES FOR SCALABLE BIG DATA PROCESSING IN HYBRID CLOUD ENVIRONMENTS

Abstract

The rapid growth of data-intensive applications has accelerated the adoption of distributed processing frameworks such as Apache Hadoop and Apache Spark. While these frameworks pro- vide scalable performance, their efficiency and manageability depend heavily on the underlying deployment strategy. This study introduces an intelligent orchestration framework that integrates machine learning-based resource prediction and optimization with containerized microservices de- ployed across federated Kubernetes clusters. The proposed system enables dynamic scheduling of Spark and Hadoop workloads based on execution time, energy consumption, and monetary cost, while also supporting fault-tolerant and modular deployments. A hybrid benchmarking methodol- ogy was employed to simulate job executions across varying node counts and cloud zones. Predic- tive models, including Random Forest and Gradient Boosting, were trained on synthetic datasets to guide scheduling decisions. Experimental results demonstrate that the system achieves notable improvements in energy efficiency (up to 18%) and cost savings while maintaining low execution latency. Additionally, the platform exhibits strong resilience under simulated zone failures and scales effectively to large, heterogeneous environments. This work contributes a comprehensive, quality-aware orchestration framework that advances sustainable and intelligent Big Data deploy- ment.