An augmented matrix is a compact representation of a system of linear equations, combining the coefficient matrix and the constants into a single matrix. It is used in methods like Gaussian elimination to find solutions to the system by performing row operations that simplify the matrix into a form where the solutions become evident.