Enum Token 

pub(crate) enum Token {
    Error(ErrorTokenKind),
    LeftBrace(Script),
    RightBrace(Script),
    Newline,
    Tab,
    LeftParen(Script),
    RightParen(Script),
    Hold,
    NotHold,
    Arrow(Script),
    Hand(Script),
    Hash(Script),
    Equals(Script),
    Asterisk(Script),
    Pipe(Script),
    DoublePipe(Script),
    ProperSuperset(Script),
    SubsetOf(Script),
    IdenticalTo(Script),
    Tilde(Script),
    LessThan(Script),
    GreaterThan(Script),
    Query(Script),
    Intersection(Script),
    Union(Script),
    Solidus(Script),
    Plus(Script),
    Minus(Script),
    Times(Script),
    LogicalOr(Script),
    LogicalAnd(Script),
    Digits(Script, NumericLiteral),
    BitPosition(Script, String, String),
    SymexSyllable(Script, String),
    Dot(Script),
    Comma(Script),
}

The parser consumes these tokens.

Variants

Error(ErrorTokenKind)

LeftBrace(Script)

RightBrace(Script)

Newline

Tab

LeftParen(Script)

The parser currently only handled parenthesised expressions in normal script.

RightParen(Script)

The parser currently only handled parenthesised expressions in normal script.

Hold

Accept either ‘h’ or ‘:’ signalling the hold bit (of the instruction word) should be set. The documentation seems to use both, though perhaps ‘:’ is the older usage.

While h is indeed a letter, it is not one of the letters which can form part of a symex. See the TX-2 Users Handbook, section 6-3.2, “RULES FOR SYMEX FORMATION”.

NotHold

Arrow(Script)

Hand(Script)

Hash(Script)

Equals(Script)

Asterisk(Script)

Asterisk is used quite heavily (indicating deferred addressing) but while the TX-2 supports superscript and subscript asterisks, they don’t seem to be used. They are not valid as part of a symex (see User handbook, section 6-2.3) and are not macro terminators (6-4.5). However, they are valid as part of a superposed character sequence making up a compound-character macro name.

Pipe(Script)

DoublePipe(Script)

ProperSuperset(Script)

SubsetOf(Script)

IdenticalTo(Script)

Tilde(Script)

LessThan(Script)

GreaterThan(Script)

Query(Script)

Intersection(Script)

Union(Script)

Solidus(Script)

Solidus is often called “slash” but people often confuse slash and backslash. So we don’t call it either.

Plus(Script)

Minus(Script)

Times(Script)

LogicalOr(Script)

LogicalAnd(Script)

Digits(Script, NumericLiteral)

BitPosition(Script, String, String)

SymexSyllable(Script, String)

The rules concerning which characters can be part of a symex are given in the TX-2 Users Handbook, section 6-3.2, “RULES FOR SYMEX FORMATION”.

We so not accept dot as part of this token becuase it behaves differently in some circumstances (it is a macro terminator). However it is part of a valid symex also, and so we will need to parse it as such.

Dot(Script)

Comma(Script)

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Token

impl StructuralPartialEq for Token