Struct GlyphTokenizer 

struct GlyphTokenizer<'a> {
    current: Option<(Token, Range<usize>)>,
    inner: GlyphRecognizer<'a>,
}

Tokenize a small part of the input, unifying the Unicode and @...@ representations.

Fields

current: Option<(Token, Range<usize>)>

inner: GlyphRecognizer<'a>

Implementations

impl<'a> GlyphTokenizer<'a>

fn new(input: &'a str) -> GlyphTokenizer<'a>

fn get_next_spanned_token(&mut self) -> Option<(Token, Range<usize>)>

Trait Implementations

impl<'a> Clone for GlyphTokenizer<'a>

fn clone(&self) -> GlyphTokenizer<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for GlyphTokenizer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.