Trait menhir_runtime::lexing::Lexer [−] [src]

pub trait Lexer {
    type Location;
    type Token;
    type Error;
    fn input(&mut self) -> Result<(Self::Location, Self::Token), Self::Error>;
}

The lexer interface.

This trait describes the interface that a Menhir parser expects from the lexer. It can be seen as an infinite iterator over tokens and locations, that can also report errors.

Several lexer generators or lexing tools expose a simpler interface based on the Iterator trait. They can be easily converted to this interface using IteratorLexer.

The stream of tokens

Associated Types

A type that describes the position of a token in the input source, for example a line and column number.

The actual value of this type is left purely at the discretion of the user. Menhir does not make any assumption on it and simply carries it through the parsing process to report it when an error occurs. The user can use any type appropriate for their error reporting mechanism, or even () if tracking of locations is not desired.

The type of tokens returned by this lexer.

A type that describes the possible errors that the lexer might encounter while processing the input source.

As with Location, Menhir does not make any assumption on this type and simply returns it unaltered when the lexer returns it. The user can use any type appropriate for their error reporting mechanism, or even () if if this lexer cannot fail.

Required Methods

Reads the next token from the input.

On success, returns a pair of the token and its location in the input source. On failure, returns a value describing the error. Returning an error from the lexer will cause a Menhir parser to stop the parsing process and report it.

Menhir parsers are able to detect successful termination without requiring extra lookahead. Because of this, no special value is required to indicate the end of the stream. If the end of the stream is encountered, input should report an error. This behaviour allows the parser not to consume any token it does not need and leave the lexer intact to be used for another parsing process. If one does want the parser to use exactly the whole input, one should either implement explicitly an “EOF” token in the grammar, or simply manually check that the lexer has no more tokens to offer once parsing succeeded.

Implementors