Transition relation is a mathematical relation that describes how a system moves from one state to another within a state space, often used in the analysis and verification of dynamic systems. It is fundamental in model checking, where it helps in verifying that a system behaves as intended by exploring all possible state transitions.