public class Tokenizer
extends java.lang.Object
Linus
1 character at a time,
building parse tokens as they are defined in a DFA
. Beginning in
currState
= 0, it consumes 1 character of input and consults the
DFA table.
For each non-empty cell in the row for currState, it reads the
character_class being sought and asks JavaLex
if the current
character belongs to that class. The first 'true' ends the search. The next
state is read from that cell -> currState
. If the state =
-99, then a completed token is accepted (type is read from the state's
(row's) tokenAccepted
field). Otherwise, go to the next state
and repeat. If the search of all cells in a row fails to get a 'true' -- that
signals an ERROR.
NOTE: Error messages intended for debugging are not translated.
Constructor and Description |
---|
Tokenizer(DFA myDFA,
Linus lineSvr)
Construct a Tokenizer.
|
Modifier and Type | Method and Description |
---|---|
TokenScanned |
lookAhead()
Returns the current token without consuming it.
|
TokenScanned |
readToken()
Returns the current token and reads a new one to take its place.
|
void |
spinTo(java.lang.String kinTerm)
Spin thru lines until this symbol is found at the head of a clause.
|
void |
spinTo(java.lang.String kinTerm,
int cbSeqNmbr)
Spin thru lines until this symbol is found at the head of a clause.
|
public Tokenizer(DFA myDFA, Linus lineSvr) throws KSParsingErrorException
myDFA
- a DFA
containing a parse table suitable for the file served by lineSvr
.lineSvr
- serves contents of the file to be parsed one line at a time.KSParsingErrorException
public void spinTo(java.lang.String kinTerm, int cbSeqNmbr) throws KSParsingErrorException
KSParsingErrorException
public void spinTo(java.lang.String kinTerm) throws KSParsingErrorException
KSParsingErrorException
public TokenScanned lookAhead()
public TokenScanned readToken() throws KSParsingErrorException
KSParsingErrorException