Reimagining Exploration: Theoretical Insights and Practical Advancements in Policy Gradient Methods