Unitary representations are homomorphisms from a group to the group of unitary operators on a Hilbert space, preserving the inner product structure and ensuring the representation is continuous and norm-preserving. They play a crucial role in quantum mechanics and representation theory, as they allow for the study of symmetries in a mathematically rigorous way.