Expand description
Turn text input into a sequence of tokens.
We perform two levels of lexing:
lower: handles comments, annotations, spaces and newlines. Generates a sequence oflower::Lexemeinstances.Lexer: pulls text from the lower-level lexer and recognizes tokens. Generates a sequence ofTokeninstances.
Re-exportsΒ§
use std::fmt::Display;πuse std::fmt::Write;πuse std::ops::Range;πuse std::str::CharIndices;πuse base::Unsigned36Bit;πuse base::charset::Script;πuse base::charset::subscript_char;πuse base::charset::superscript_char;πuse base::error::StringConversionFailed;πuse super::glyph::Elevated;πuse super::glyph::Glyph;πuse super::glyph::GlyphShape;πuse super::glyph::Unrecognised;πuse super::glyph::elevate;πuse super::glyph::glyph_from_name;πuse super::glyph::glyph_of_char;πuse super::glyph::is_allowed_in_symex;πuse super::parser::helpers;πuse super::state::NumeralMode;π
ModulesΒ§
- lower π
- A βpartialβ lexer which determines whether weβre inside an RC-block or a comment.
StructsΒ§
- Glyph
Recognizer π - Convert a string into a sequence of
Elevated<&Glyph>. - Glyph
Tokenizer π - Tokenize a small part of the input, unifying the Unicode and
@...@representations. - Lexer π
- Tokenize the body of a source code file.
- Numeric
Literal π - Spanned
Iter π
EnumsΒ§
- Error
Token πKind - Represents an item in the input we didnβt recognise of which we donβt support.
- Token π
- The parser consumes these tokens.
- Token
Merge πResult
ConstantsΒ§
FunctionsΒ§
- merge_
tokens π - tokenise_
single_ πglyph
Type AliasesΒ§
- Span π