In formal grammar, a non-terminal is a symbol used to represent a set of strings and can be replaced by other symbols according to production rules. They are essential for defining the structure of a language, as they help in generating strings of a language by recursively applying these rules until only terminal symbols remain.