Heap overflow occurs when a program writes more data to a dynamically allocated memory area on the heap than it was allocated, potentially leading to unpredictable behavior or security vulnerabilities. This can be exploited by attackers to execute arbitrary code or crash the system, making it a critical issue in software security.