TOWARDS AUTONOMOUS CODE OPTIMIZATION: A REINFORCEMENT LEARNING FRAMEWORK FOR COMPILER DESIGN

Abstract

Modern compiler optimization remains one of the most challenging problems in computer systems research. Conventional compiler pipelines rely on static, manually crafted heuristics to determine optimization passes, instruction scheduling, and register allocation. However, as software complexity and hardware heterogeneity increase, these heuristics struggle to generalize across workloads, architectures, and programming paradigms. This paper proposes an autonomous reinforcement learning (RL) framework for compiler design, in which optimization pass selection and parameter tuning are treated as sequential decision-making tasks.

The proposed system formulates compiler optimization as a Markov Decision Process (MDP), where the state represents the intermediate representation (IR) of code, the actions correspond to possible optimization passes, and the reward is derived from performance improvements such as reduced execution time or binary size. A Graph Neural Network (GNN) encoder captures structural information from IR graphs, while a deep reinforcement learning agent (e.g., PPO or DQN) learns optimization policies that generalize across programs and architectures. The framework integrates with the LLVM and MLIR compiler infrastructures and is evaluated on benchmark suites including SPEC CPU2017 and PolyBench. Experimental results indicate up to 35% performance improvement over standard -O3 optimization levels and 20% reduction in code size without compromising compilation time. Ablation studies confirm that GNN-based state encoding and multi-objective reward shaping are essential to policy stability and cross-architecture generalization. This study contributes a modular, scalable approach to autonomous code optimization, bridging the gap between classical compiler theory and data-driven decision systems. The paper concludes with open challenges in interpretability, real-time adaptation, and integration with differentiable compiler toolchains.