A unitary transformation is a linear operation on a complex vector space that preserves the inner product, meaning it maintains the length and angle between vectors. This property makes unitary transformations fundamental in quantum mechanics, where they describe the evolution of quantum states in a way that conserves probability amplitudes.