An unstable equilibrium occurs when a system is in a state where any small disturbance or deviation will lead to a departure from the equilibrium state, often resulting in a new equilibrium or dynamic behavior. This concept is critical in understanding systems that are sensitive to initial conditions, such as in physics, economics, and biology, where small changes can lead to significant and sometimes unpredictable outcomes.