ZenTokener (ZenScript 4.1.20.711 API)

java.lang.Object
- stanhebben.zenscript.parser.TokenStream
- - stanhebben.zenscript.ZenTokener

All Implemented Interfaces:

java.util.Iterator<Token>
```
public class ZenTokener
extends TokenStream
```
A tokener is capable of splitting a single file into tokens. It's intended for use by LL(*) parsers.

Field Summary

Fields
Modifier and Type	Field and Description
`boolean`	`ignoreBracketErrors`
`static int`	`T_ACLOSE`
`static int`	`T_AND`
`static int`	`T_AND2`
`static int`	`T_ANDASSIGN`
`static int`	`T_ANY`
`static int`	`T_AOPEN`
`static int`	`T_AS`
`static int`	`T_ASSIGN`
`static int`	`T_BOOL`
`static int`	`T_BRCLOSE`
`static int`	`T_BREAK`
`static int`	`T_BROPEN`
`static int`	`T_BYTE`
`static int`	`T_COLON`
`static int`	`T_COMMA`
`static int`	`T_CONTINUE`
`static int`	`T_DIV`
`static int`	`T_DIVASSIGN`
`static int`	`T_DOLLAR`
`static int`	`T_DOT`
`static int`	`T_DOT2`
`static int`	`T_DOUBLE`
`static int`	`T_ELSE`
`static int`	`T_EQ`
`static int`	`T_FALSE`
`static int`	`T_FLOAT`
`static int`	`T_FLOATVALUE`
`static int`	`T_FOR`
`static int`	`T_FUNCTION`
`static int`	`T_GLOBAL`
`static int`	`T_GT`
`static int`	`T_GTEQ`
`static int`	`T_ID`
`static int`	`T_IF`
`static int`	`T_IMPORT`
`static int`	`T_IN`
`static int`	`T_INSTANCEOF`
`static int`	`T_INT`
`static int`	`T_INTVALUE`
`static int`	`T_LONG`
`static int`	`T_LT`
`static int`	`T_LTEQ`
`static int`	`T_MINUS`
`static int`	`T_MINUSASSIGN`
`static int`	`T_MOD`
`static int`	`T_MODASSIGN`
`static int`	`T_MUL`
`static int`	`T_MULASSIGN`
`static int`	`T_NOT`
`static int`	`T_NOTEQ`
`static int`	`T_NULL`
`static int`	`T_OR`
`static int`	`T_OR2`
`static int`	`T_ORASSIGN`
`static int`	`T_PLUS`
`static int`	`T_PLUSASSIGN`
`static int`	`T_QUEST`
`static int`	`T_RETURN`
`static int`	`T_SEMICOLON`
`static int`	`T_SHORT`
`static int`	`T_SQBRCLOSE`
`static int`	`T_SQBROPEN`
`static int`	`T_STATIC`
`static int`	`T_STRING`
`static int`	`T_STRINGVALUE`
`static int`	`T_TILDE`
`static int`	`T_TILDEASSIGN`
`static int`	`T_TRUE`
`static int`	`T_VAL`
`static int`	`T_VAR`
`static int`	`T_VERSION`
`static int`	`T_VOID`
`static int`	`T_WHILE`
`static int`	`T_XOR`
`static int`	`T_XORASSIGN`
`static int`	`T_ZEN_CLASS`
`static int`	`T_ZEN_CONSTRUCTOR`

Constructor Summary

Constructors
Constructor and Description
`ZenTokener(java.io.Reader contents, IZenCompileEnvironment environment, java.lang.String fileNameFallback, boolean ignoreBracketErrors)` Constructs a tokener from the given reader.
`ZenTokener(java.lang.String contents, IZenCompileEnvironment environment, java.lang.String fileNameFallback, boolean ignoreBracketErrors)` Constructs a tokener from the given string.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`IZenCompileEnvironment`	`getEnvironment()` Retrieves the compile environment of this tokener.
`Token`	`process(Token token)` Can be implemented by subclasses to post-process tokens.

Methods inherited from class stanhebben.zenscript.parser.TokenStream
getFile, getLine, getLineOffset, hasNext, isNext, next, optional, optional, peek, popMark, pushMark, remove, required, reset, setFile

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator
forEachRemaining

- Field Detail
  - T_ID
```
public static final int T_ID
```
    See Also:
    
    Constant Field Values
  - T_INTVALUE
```
public static final int T_INTVALUE
```
    See Also:
    
    Constant Field Values
  - T_FLOATVALUE
```
public static final int T_FLOATVALUE
```
    See Also:
    
    Constant Field Values
  - T_STRINGVALUE
```
public static final int T_STRINGVALUE
```
    See Also:
    
    Constant Field Values
  - T_AOPEN
```
public static final int T_AOPEN
```
    See Also:
    
    Constant Field Values
  - T_ACLOSE
```
public static final int T_ACLOSE
```
    See Also:
    
    Constant Field Values
  - T_SQBROPEN
```
public static final int T_SQBROPEN
```
    See Also:
    
    Constant Field Values
  - T_SQBRCLOSE
```
public static final int T_SQBRCLOSE
```
    See Also:
    
    Constant Field Values
  - T_DOT2
```
public static final int T_DOT2
```
    See Also:
    
    Constant Field Values
  - T_DOT
```
public static final int T_DOT
```
    See Also:
    
    Constant Field Values
  - T_COMMA
```
public static final int T_COMMA
```
    See Also:
    
    Constant Field Values
  - T_PLUSASSIGN
```
public static final int T_PLUSASSIGN
```
    See Also:
    
    Constant Field Values
  - T_PLUS
```
public static final int T_PLUS
```
    See Also:
    
    Constant Field Values
  - T_MINUSASSIGN
```
public static final int T_MINUSASSIGN
```
    See Also:
    
    Constant Field Values
  - T_MINUS
```
public static final int T_MINUS
```
    See Also:
    
    Constant Field Values
  - T_MULASSIGN
```
public static final int T_MULASSIGN
```
    See Also:
    
    Constant Field Values
  - T_MUL
```
public static final int T_MUL
```
    See Also:
    
    Constant Field Values
  - T_DIVASSIGN
```
public static final int T_DIVASSIGN
```
    See Also:
    
    Constant Field Values
  - T_DIV
```
public static final int T_DIV
```
    See Also:
    
    Constant Field Values
  - T_MODASSIGN
```
public static final int T_MODASSIGN
```
    See Also:
    
    Constant Field Values
  - T_MOD
```
public static final int T_MOD
```
    See Also:
    
    Constant Field Values
  - T_ORASSIGN
```
public static final int T_ORASSIGN
```
    See Also:
    
    Constant Field Values
  - T_OR
```
public static final int T_OR
```
    See Also:
    
    Constant Field Values
  - T_OR2
```
public static final int T_OR2
```
    See Also:
    
    Constant Field Values
  - T_ANDASSIGN
```
public static final int T_ANDASSIGN
```
    See Also:
    
    Constant Field Values
  - T_AND2
```
public static final int T_AND2
```
    See Also:
    
    Constant Field Values
  - T_AND
```
public static final int T_AND
```
    See Also:
    
    Constant Field Values
  - T_XORASSIGN
```
public static final int T_XORASSIGN
```
    See Also:
    
    Constant Field Values
  - T_XOR
```
public static final int T_XOR
```
    See Also:
    
    Constant Field Values
  - T_QUEST
```
public static final int T_QUEST
```
    See Also:
    
    Constant Field Values
  - T_COLON
```
public static final int T_COLON
```
    See Also:
    
    Constant Field Values
  - T_BROPEN
```
public static final int T_BROPEN
```
    See Also:
    
    Constant Field Values
  - T_BRCLOSE
```
public static final int T_BRCLOSE
```
    See Also:
    
    Constant Field Values
  - T_TILDEASSIGN
```
public static final int T_TILDEASSIGN
```
    See Also:
    
    Constant Field Values
  - T_TILDE
```
public static final int T_TILDE
```
    See Also:
    
    Constant Field Values
  - T_SEMICOLON
```
public static final int T_SEMICOLON
```
    See Also:
    
    Constant Field Values
  - T_LTEQ
```
public static final int T_LTEQ
```
    See Also:
    
    Constant Field Values
  - T_LT
```
public static final int T_LT
```
    See Also:
    
    Constant Field Values
  - T_GTEQ
```
public static final int T_GTEQ
```
    See Also:
    
    Constant Field Values
  - T_GT
```
public static final int T_GT
```
    See Also:
    
    Constant Field Values
  - T_EQ
```
public static final int T_EQ
```
    See Also:
    
    Constant Field Values
  - T_ASSIGN
```
public static final int T_ASSIGN
```
    See Also:
    
    Constant Field Values
  - T_NOTEQ
```
public static final int T_NOTEQ
```
    See Also:
    
    Constant Field Values
  - T_NOT
```
public static final int T_NOT
```
    See Also:
    
    Constant Field Values
  - T_DOLLAR
```
public static final int T_DOLLAR
```
    See Also:
    
    Constant Field Values
  - T_ANY
```
public static final int T_ANY
```
    See Also:
    
    Constant Field Values
  - T_BOOL
```
public static final int T_BOOL
```
    See Also:
    
    Constant Field Values
  - T_BYTE
```
public static final int T_BYTE
```
    See Also:
    
    Constant Field Values
  - T_SHORT
```
public static final int T_SHORT
```
    See Also:
    
    Constant Field Values
  - T_INT
```
public static final int T_INT
```
    See Also:
    
    Constant Field Values
  - T_LONG
```
public static final int T_LONG
```
    See Also:
    
    Constant Field Values
  - T_FLOAT
```
public static final int T_FLOAT
```
    See Also:
    
    Constant Field Values
  - T_DOUBLE
```
public static final int T_DOUBLE
```
    See Also:
    
    Constant Field Values
  - T_STRING
```
public static final int T_STRING
```
    See Also:
    
    Constant Field Values
  - T_FUNCTION
```
public static final int T_FUNCTION
```
    See Also:
    
    Constant Field Values
  - T_IN
```
public static final int T_IN
```
    See Also:
    
    Constant Field Values
  - T_VOID
```
public static final int T_VOID
```
    See Also:
    
    Constant Field Values
  - T_AS
```
public static final int T_AS
```
    See Also:
    
    Constant Field Values
  - T_VERSION
```
public static final int T_VERSION
```
    See Also:
    
    Constant Field Values
  - T_IF
```
public static final int T_IF
```
    See Also:
    
    Constant Field Values
  - T_ELSE
```
public static final int T_ELSE
```
    See Also:
    
    Constant Field Values
  - T_FOR
```
public static final int T_FOR
```
    See Also:
    
    Constant Field Values
  - T_RETURN
```
public static final int T_RETURN
```
    See Also:
    
    Constant Field Values
  - T_VAR
```
public static final int T_VAR
```
    See Also:
    
    Constant Field Values
  - T_VAL
```
public static final int T_VAL
```
    See Also:
    
    Constant Field Values
  - T_WHILE
```
public static final int T_WHILE
```
    See Also:
    
    Constant Field Values
  - T_BREAK
```
public static final int T_BREAK
```
    See Also:
    
    Constant Field Values
  - T_CONTINUE
```
public static final int T_CONTINUE
```
    See Also:
    
    Constant Field Values
  - T_NULL
```
public static final int T_NULL
```
    See Also:
    
    Constant Field Values
  - T_TRUE
```
public static final int T_TRUE
```
    See Also:
    
    Constant Field Values
  - T_FALSE
```
public static final int T_FALSE
```
    See Also:
    
    Constant Field Values
  - T_IMPORT
```
public static final int T_IMPORT
```
    See Also:
    
    Constant Field Values
  - T_GLOBAL
```
public static final int T_GLOBAL
```
    See Also:
    
    Constant Field Values
  - T_STATIC
```
public static final int T_STATIC
```
    See Also:
    
    Constant Field Values
  - T_INSTANCEOF
```
public static final int T_INSTANCEOF
```
    See Also:
    
    Constant Field Values
  - T_ZEN_CLASS
```
public static final int T_ZEN_CLASS
```
    See Also:
    
    Constant Field Values
  - T_ZEN_CONSTRUCTOR
```
public static final int T_ZEN_CONSTRUCTOR
```
    See Also:
    
    Constant Field Values
  - ignoreBracketErrors
```
public final boolean ignoreBracketErrors
```
- Constructor Detail
  - ZenTokener
```
public ZenTokener(java.io.Reader contents,
                  IZenCompileEnvironment environment,
                  java.lang.String fileNameFallback,
                  boolean ignoreBracketErrors)
           throws java.io.IOException
```
    Constructs a tokener from the given reader.
    
    Parameters:
    
    contents - file reader
    
    environment - compile environment
    
    Throws:
    
    java.io.IOException - if the file could not be read properly
  - ZenTokener
```
public ZenTokener(java.lang.String contents,
                  IZenCompileEnvironment environment,
                  java.lang.String fileNameFallback,
                  boolean ignoreBracketErrors)
           throws java.io.IOException
```
    Constructs a tokener from the given string.
    
    Parameters:
    
    contents - content string
    
    environment - compile environment
    
    Throws:
    
    java.io.IOException - shouldn't happen
- Method Detail
  - getEnvironment
```
public IZenCompileEnvironment getEnvironment()
```
    Retrieves the compile environment of this tokener.
    
    Returns:
    
    compile environment
  - process
```
public Token process(Token token)
```
    Description copied from class: TokenStream
    
    Can be implemented by subclasses to post-process tokens.
    
    Overrides:
    
    process in class TokenStream
    
    Returns:
    
    processed token

Class ZenTokener

Field Summary

Constructor Summary

Method Summary

Methods inherited from class stanhebben.zenscript.parser.TokenStream

Methods inherited from class java.lang.Object

Methods inherited from interface java.util.Iterator

Field Detail

T_ID

T_INTVALUE

T_FLOATVALUE

T_STRINGVALUE

T_AOPEN

T_ACLOSE

T_SQBROPEN

T_SQBRCLOSE

T_DOT2

T_DOT

T_COMMA

T_PLUSASSIGN

T_PLUS

T_MINUSASSIGN

T_MINUS

T_MULASSIGN

T_MUL

T_DIVASSIGN

T_DIV

T_MODASSIGN

T_MOD

T_ORASSIGN

T_OR

T_OR2

T_ANDASSIGN

T_AND2

T_AND

T_XORASSIGN

T_XOR

T_QUEST

T_COLON

T_BROPEN

T_BRCLOSE

T_TILDEASSIGN

T_TILDE

T_SEMICOLON

T_LTEQ

T_LT

T_GTEQ

T_GT

T_EQ

T_ASSIGN

T_NOTEQ

T_NOT

T_DOLLAR

T_ANY

T_BOOL

T_BYTE

T_SHORT

T_INT

T_LONG

T_FLOAT

T_DOUBLE

T_STRING

T_FUNCTION

T_IN

T_VOID

T_AS

T_VERSION

T_IF

T_ELSE

T_FOR

T_RETURN

T_VAR

T_VAL

T_WHILE

T_BREAK

T_CONTINUE

T_NULL

T_TRUE

T_FALSE

T_IMPORT