Version

Lexical Errors (Syntax Parsing Engine)

Topic Overview

Purpose

This topic explains the lexical errors found by the Syntax Parsing Engine.

Required background

The following topics are prerequisites to understanding this topic:

Topic Purpose

This topic provides an overview of the Syntax Parsing Engine.

This topic provides an overview of the Syntax Parsing Engine’s Grammar.

The topics in this group explain the lexical analysis performed by the Syntax Parsing Engine.

In this topic

This topic contains the following sections:

Lexical Errors

Overview

Lexical errors are detected relatively easily and the lexical analyzer recovers from them easily as well. There is really only one type of lexical error: none of the terminal symbols in the current lexer state can represent the text at the current location.

Example

Consider a grammar with a default lexer state defined as having the following terminal symbols:

  • NewLineSymbol – matches one new line, which is either “\r”, “\n”, or “\r\n”

  • WhitespaceSymbol – matches one or more space or tab characters.

  • Identifier – matches an underscore or letter followed by zero or more underscores, letters, or digits.

Document content to be parsed: “x += y;”

While the lexer is analyzing this document content it will first create tokens <Identifier, “x”> and <WhitespaceToken, “ ”>. Then it will come to the “+” character and no terminal symbol will be able to recognize it. The lexer will continue reading unrecognized characters until it comes to a piece of text which can be matched by any terminal symbol. In this case, that is the space after the “=”. The lexer will then gather up all contiguous unrecognized characters and make them a match of the special symbol exposed by Grammar.UnrecognizedSymbol. So the full token stream for this document content will be the following:

<Identifier, “x”> <WhitespaceToken, “ ”> <UnrecognizedToken, “+=”> <WhitespaceToken, “ ”> <Identifier, “y”> <UnrecognizedToken, “;”> <EndOfStreamToken>

Related Content

Topics

The following topics provide additional information related to this topic.

Topic Purpose

This topic explains the syntax errors found by the Syntax Parsing Engine.

The topics in this group explain the strategies used during the syntax analyzing of the document to identify syntax errors.

This topic explains semantic errors, which are not handled by the Syntax Parsing Engine.