LEXICAL ANALYSIS

(n.) The process by which a stream of characters (often from a source program) is rearranged into elementary words and symbols, called token s. The tokens can include the reserved words of a programming language, its identifiers and constants, and special symbols such as =, :=, and ;.