Package org.openzen.zenscript.lexer

Class TokenParser<T extends Token<TT>,TT extends TokenType>

java.lang.Object

org.openzen.zenscript.lexer.TokenParser<T,TT>

Type Parameters:

T - token class

TT - token type class

All Implemented Interfaces:

TokenStream<TT,T>

public class TokenParser<T extends Token<TT>,TT extends TokenType>
extends Object
implements TokenStream<TT,T>

Represents a token stream. A token stream reads characters from a reader and presents it as a series of tokens. Can be used to implement LL(*) parsers.

Token classes with a negative class are considered to be whitespace.

Constructor Summary

Constructors

Constructor	Description
TokenParser(SourceFile file, String data, CompiledDFA<TT> dfa, TT eof, TT invalid, TokenFactory<T,TT> factory)	Creates a token stream which reads data from the specified string.
TokenParser(SourceFile file, CharReader reader, CompiledDFA<TT> dfa, TT eof, TT invalid, TokenFactory<T,TT> factory)	Creates a token stream using the specified reader and DFA.

Method Summary

Modifier and Type	Method	Description
TT	getEOF()
CodePosition	getPosition()
boolean	hasNext()
T	next()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TokenParser

public TokenParser(SourceFile file,
 CharReader reader,
 CompiledDFA<TT> dfa,
 TT eof,
 TT invalid,
 TokenFactory<T,TT> factory)

Creates a token stream using the specified reader and DFA.

Parameters:

file - filename

reader - reader to read characters from

dfa - DFA to tokenize the stream

eof - end of file token type

TokenParser

public TokenParser(SourceFile file,
 String data,
 CompiledDFA<TT> dfa,
 TT eof,
 TT invalid,
 TokenFactory<T,TT> factory)

Creates a token stream which reads data from the specified string.

Parameters:

file - filename

data - data to read

dfa - DFA to tokenize the stream

eof - end of file token type

Method Details

getPosition

public CodePosition getPosition()

Specified by:

getPosition in interface TokenStream<T extends Token<TT>,TT extends TokenType>

hasNext

public boolean hasNext()

getEOF

public TT getEOF()

Specified by:

getEOF in interface TokenStream<T extends Token<TT>,TT extends TokenType>

next

public T next()
       throws ParseException

Specified by:

next in interface TokenStream<T extends Token<TT>,TT extends TokenType>

Throws:

ParseException