Unitary transformations are linear transformations in quantum mechanics that preserve the inner product, ensuring that probabilities remain consistent. They are represented by unitary matrices, which are crucial for describing the evolution of quantum states in a reversible manner.