When the lexical analyzer reads through a document, it attempts to group characters into tokens. A Token is associated with a TerminalSymbol and the text which was read to create it. Each TerminalSymbol indicates the possible text that can be used to create tokens for it via its Value and Comparison properties. The Comparison indicates how the Value should be interpreted: as literal text, case-insensitive literal text, or a regular expression.
The lexical analyzer is always associated with a specified LexerState when it is reading through a document. Initially, that is the DefaultLexerState of the Grammar.LexerStates collection. Each LexerState owns a collection of LexerState.Symbols, which are the TerminalSymbol instances that can possibly have tokens created for them in that state. The lexical analyzer will read characters until it can create the longest possible token. If two or more TerminalSymbol instances can represent the longest token, the TerminalSymbol which is at the lowest index in the Symbols collection will be used. Therefore, TerminalSymbol instances representing reserved keywords should always be declared before a TerminalSymbol instance representing any identifier, so the reserved keyword is always used when possible.
Tokens created by the lexer are marked as either significant or insignificant. This indicates whether the tokens should be analyzed by the syntax analyzer or ignored by it. The significance of each token is determined by the value of the IsSignificant property of the associated TerminalSymbol if it is set. If not, the significance of the token is determined by the LanguageElement property. If the language element is Whitespace, Newline, or Comment, the token will be insignificant. Otherwise, it will be significant.
When the syntax analyzer processes the tokens created by the lexical analyzer, the significant tokens will be associated with SyntaxNode instances in the syntax tree and their SyntaxNode.IsTokenNode value will be True. Insignificant tokens will be represented in the leading ignored content of the next token node created.
Target Platforms: Windows 10, Windows 8.1, Windows 8, Windows 7, Windows Server 2012, Windows 7, Windows Vista SP1 or later, Windows XP SP3, Windows Server 2008 (Server Core not supported), Windows Server 2008 R2 (Server Core supported with SP1 or later), Windows Server 2003 SP2