A composite state in state machine design is a hierarchical state that contains substates, allowing for more organized and modular representation of complex behaviors. It simplifies the state machine by encapsulating related states and transitions, facilitating easier maintenance and scalability.