Gaussian processes are a powerful non-parametric Bayesian approach used for regression and probabilistic classification, characterized by defining a distribution over functions. They provide a principled way of incorporating prior knowledge and handling uncertainty in predictions, making them particularly useful in machine learning tasks where data is scarce or noisy.