Ordinal notation is a system used to uniquely represent ordinals, which are a generalization of natural numbers used to describe order types of well-ordered sets. It extends beyond finite numbers to describe infinite sequences, providing a framework for understanding the hierarchy of infinity in set theory.