A cost function is a mathematical formula used in optimization problems to quantify the error or cost associated with a particular solution, often guiding the learning process in machine learning models. It evaluates how well a model's predictions match the actual data, and the goal is to minimize this cost to improve model accuracy.