Struct Scanner 

pub struct Scanner<'input> { /* private fields */ }

Scanner handles the low-level details of reading characters from a raw input stream of bytes. It decodes those bytes into UTF-8 characters, and associates each character with the [miden_debug_types::ByteIndex] at which it occurs.

The Scanner is intended to be consumed by a lexer, which handles converting the stream of characters into a token stream for use by the parser.

Scanner Lifecycle

The following illustrates how content flows from the raw input stream through the scanner.

lexer <- (peek) <- pending <- source
      <- (pop) <- current <- pending <- source

As shown above, the lexer is “pulling” characters from the scanner.

When “peeking” a character, we return the character currently in the pending field, but if pending is empty, we read enough bytes from the source to construct a UTF-8 character, and store it as pending, as well as returning it to the lexer.

When “popping” a character (i.e. we are advancing the scanner in the input), we are returning the character in the current field, and then moving the character in pending into current. Accordingly, if any of those fields is empty, we must pull from the next field in the chain, reading bytes from the input as we go.

Implementations

impl<'input> Scanner<'input>

pub fn new(input: &'input str) -> Self

Construct a new Scanner for the given source.

pub fn start(&self) -> usize

Returns the byte offset representing the start of the source

pub fn advance(&mut self)

Advance scanner pipeline by a single character.

pending becomes current, and bytes are read from the input to repopulate pending.

pub fn pop(&mut self) -> (usize, char)

Return the current character and advance our position in the source

pub fn peek(&self) -> (usize, char)

Return the next character in the input, but do not advance.

pub fn peek_next(&mut self) -> (usize, char)

Return the character after the next character in the input, but do not advance.

pub fn read(&self) -> (usize, char)

Get current character in the input.

pub fn slice(&self, span: impl Into<Range<usize>>) -> &'input str

Get a string slice representing the given range in the underlying source