Cost functions are mathematical representations used to measure the error or 'cost' of a model's predictions compared to actual outcomes, guiding the optimization process in machine learning and statistical models. They are essential for training algorithms, as they provide a quantitative basis to adjust model parameters and improve accuracy.