Module lexer 

Turn text input into a sequence of tokens.

We perform two levels of lexing:

• lower: handles comments, annotations, spaces and newlines. Generates a sequence of lower::Lexeme instances.
• Lexer: pulls text from the lower-level lexer and recognizes tokens. Generates a sequence of Token instances.

Modules

lower 🔒

A “partial” lexer which determines whether we’re inside an RC-block or a comment.

Structs

GlyphRecognizer 🔒

Convert a string into a sequence of Elevated<&Glyph>.

GlyphTokenizer 🔒

Tokenize a small part of the input, unifying the Unicode and @...@ representations.

Lexer 🔒

Tokenize the body of a source code file.

NumericLiteral 🔒

SpannedIter 🔒

Enums

ErrorTokenKind 🔒

Represents an item in the input we didn’t recognise of which we don’t support.

Token 🔒

The parser consumes these tokens.

TokenMergeResult 🔒

Constants

DOT_CHAR 🔒

DOT_STR 🔒

Functions

merge_tokens 🔒

tokenise_single_glyph 🔒

Type Aliases

Span 🔒