Skip to main content

tokenize

Function tokenize 

Source
fn tokenize(input: &str, span_offset: usize) -> Vec<(Token, SimpleSpan)>