The Nyquist criterion is a fundamental principle in signal processing and control theory that determines the stability of a system by analyzing the frequency response. It states that for a system to be stable, the Nyquist plot of its open-loop transfer function must not encircle the critical point (-1,0) in the complex plane when the plot is traced as the frequency varies from zero to infinity.