Non-terminals are symbols used in formal grammars to define the structure of a language, serving as placeholders for patterns of strings that can be further expanded into terminal symbols. They are essential for generating the syntax of programming languages and natural languages, enabling recursive definitions and hierarchical organization of language constructs.