In today’s world, the expansion of cyberspace and growing societal dependence on information systems have turned cyber threats into a serious challenge for national security. This paper presents a mathematical model based on game theory to analyze and simulate optimal strategies in cyber attacks and defenses. First, game theory is introduced as an analytical tool in cyberspace, and its significance in determining the best strategies for both attackers and defenders is examined. Next, a mathematical model is defined in which both parties aim to maximize their own payoff while minimizing damages caused by attacks through the selection of various strategies. Subsequently, several cyber scenarios—including DDoS attacks, phishing, and sophisticated malware—are investigated via MATLAB simulations, and the impact of varying levels of attack and defense is analyzed. The results demonstrate that optimal defensive strategies can significantly reduce the success rate of cyber attacks and enhance cybersecurity. Finally, the proposed model serves as an effective tool to assist cybersecurity policymakers in developing efficient strategies to counter cyber threats.