Heap fragmentation occurs when free memory is split into small, non-contiguous blocks, making it difficult to allocate large contiguous memory segments despite having enough total free memory. This can lead to inefficient memory usage and performance degradation in applications that require dynamic memory allocation.