Internal set theory (IST) is an extension of standard set theory that incorporates a rigorous framework for handling infinitesimals and infinite numbers, developed by Edward Nelson. It introduces a new predicate 'standard' to distinguish between standard and non-standard elements, allowing for a more intuitive treatment of calculus and analysis within a non-standard framework.