aud.example.expr.Tokenizer Class Reference

Breaks input string into pieces ("tokens"). More...

Inheritance diagram for aud.example.expr.Tokenizer:

Collaboration diagram for aud.example.expr.Tokenizer:

Public Member Functions
	Tokenizer (String input)
	create new tokenizer for input More...
Public Member Functions inherited from aud.util.LexicalScanner
	LexicalScanner (Rule[] rules, String input)
	create new scanner processing input @endiliteral More...
void	setInput (String input)
	set input (resets scanner state) More...
String	matchedText ()
	get text of last match or call to next More...
int	matchedTokenId ()
	get result of last call to next() More...
String	remainder ()
	get remaining text More...
boolean	endOfInput ()
	reached end of input? More...
int	next ()
	match remainder to rules provided to constructor More...

Static Public Member Functions
static void	main (String[] args)
	testing and example for usage More...
static void	main (String[] args)
	testing and example for usage More...

Static Public Attributes
static final int	PLUS ='+'
static final int	MINUS ='-'
static final int	TIMES ='*'
static final int	DIVIDE ='/'
static final int	POWER ='^'
static final int	LEFT_PAREN ='('
static final int	RIGHT_PAREN =')'
static final int	IDENTIFIER =0x100
static final int	NUMBER =0x101
Static Public Attributes inherited from aud.util.LexicalScanner
static final int	END_OF_INPUT = -1
	no more input More...
static final int	NO_MATCH = -2
	no match (usually implies a syntax error) More...
static final Pattern	P_WHITESPACE = Pattern.compile("\\s+")
	white space More...
static final Pattern	P_IDENTIFIER
	identifiers More...
static final Pattern	P_FLOAT
	floating point number More...

Static Protected Attributes
static final Pattern	P_PLUS = Pattern.compile("\\+")
static final Pattern	P_MINUS = Pattern.compile("-")
static final Pattern	P_TIMES = Pattern.compile("\\*")
static final Pattern	P_DIVIDE = Pattern.compile("/")
static final Pattern	P_POWER = Pattern.compile("(\\*\\*)|\\^")
static final Pattern	P_LEFTPAREN = Pattern.compile("\\‍(")
static final Pattern	P_RIGHTPAREN = Pattern.compile("\\‍)")
static final LexicalScanner.Rule[]	RULES

Additional Inherited Members
Protected Member Functions inherited from aud.util.LexicalScanner
void	eatWhiteSpace ()
	ignore white space (called by match More...
boolean	match (Pattern p)
	Match remainder against pattern p. More...
int	next (Rule[] rules)
	match remainder to table of rules @endiliteral More...

Detailed Description

Breaks input string into pieces ("tokens").

Definition at line 9 of file Tokenizer.java.

Constructor & Destructor Documentation

Tokenizer()

aud.example.expr.Tokenizer.Tokenizer

(

String

input

)

create new tokenizer for input

Definition at line 44 of file Tokenizer.java.

                                 {
    super(RULES,input);
  }

References aud.example.expr.Tokenizer.RULES.

Member Function Documentation

main()

static void aud.example.expr.Tokenizer.main ( String[]  args )

static

testing and example for usage

Reimplemented from aud.util.LexicalScanner.

Definition at line 49 of file Tokenizer.java.

                                         {
 
    Tokenizer t=new Tokenizer
      (args.length==0 ? "2*(3+4)/x-3.14+2^3-2**3" : args[0]);
 
    System.out.println("input = '"+t.remainder()+"'");
 
    while (t.next()!=END_OF_INPUT) {
      if (t.matchedTokenId()==NO_MATCH) {
        System.out.println("syntax error near '"+t.remainder()+"'");
        break;
      }
      else if (t.matchedTokenId()==NUMBER) {
        double d=Double.parseDouble(t.matchedText());
        System.out.println("read a number: "+d);
      }
      System.out.println("next token id = "+t.matchedTokenId());
      System.out.println("matched text = '"+t.matchedText()+"'");
      System.out.println("remaining input = '"+t.remainder()+"'");
    }
  }

References aud.util.LexicalScanner.END_OF_INPUT, aud.util.LexicalScanner.matchedText(), aud.util.LexicalScanner.matchedTokenId(), aud.util.LexicalScanner.next(), aud.util.LexicalScanner.NO_MATCH, aud.example.expr.Tokenizer.NUMBER, and aud.util.LexicalScanner.remainder().

Here is the call graph for this function:

Member Data Documentation

DIVIDE

final int aud.example.expr.Tokenizer.DIVIDE ='/'

static

Definition at line 22 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.product().

IDENTIFIER

final int aud.example.expr.Tokenizer.IDENTIFIER =0x100

static

Definition at line 27 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.factor().

LEFT_PAREN

final int aud.example.expr.Tokenizer.LEFT_PAREN ='('

static

Definition at line 24 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.factor().

MINUS

final int aud.example.expr.Tokenizer.MINUS ='-'

static

Definition at line 20 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.factor(), and aud.example.expr.ExpressionParser.sum().

NUMBER

final int aud.example.expr.Tokenizer.NUMBER =0x101

static

Definition at line 28 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.factor(), and aud.example.expr.Tokenizer.main().

P_DIVIDE

final Pattern aud.example.expr.Tokenizer.P_DIVIDE = Pattern.compile("/")

staticprotected

Definition at line 14 of file Tokenizer.java.

P_LEFTPAREN

final Pattern aud.example.expr.Tokenizer.P_LEFTPAREN = Pattern.compile("\\‍(")

staticprotected

Definition at line 16 of file Tokenizer.java.

P_MINUS

final Pattern aud.example.expr.Tokenizer.P_MINUS = Pattern.compile("-")

staticprotected

Definition at line 12 of file Tokenizer.java.

P_PLUS

final Pattern aud.example.expr.Tokenizer.P_PLUS = Pattern.compile("\\+")

staticprotected

Definition at line 11 of file Tokenizer.java.

P_POWER

final Pattern aud.example.expr.Tokenizer.P_POWER = Pattern.compile("(\\*\\*)|\\^")

staticprotected

Definition at line 15 of file Tokenizer.java.

P_RIGHTPAREN

final Pattern aud.example.expr.Tokenizer.P_RIGHTPAREN = Pattern.compile("\\‍)")

staticprotected

Definition at line 17 of file Tokenizer.java.

P_TIMES

final Pattern aud.example.expr.Tokenizer.P_TIMES = Pattern.compile("\\*")

staticprotected

Definition at line 13 of file Tokenizer.java.

PLUS

final int aud.example.expr.Tokenizer.PLUS ='+'

static

Definition at line 19 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.factor(), and aud.example.expr.ExpressionParser.sum().

POWER

final int aud.example.expr.Tokenizer.POWER ='^'

static

Definition at line 23 of file Tokenizer.java.

RIGHT_PAREN

final int aud.example.expr.Tokenizer.RIGHT_PAREN =')'

static

Definition at line 25 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.factor().

RULES

final LexicalScanner.Rule [] aud.example.expr.Tokenizer.RULES

staticprotected

Initial value:

=
  {
    new LexicalScanner.Rule(PLUS,  P_PLUS),
    new LexicalScanner.Rule(MINUS, P_MINUS),
    new LexicalScanner.Rule(POWER, P_POWER), 
    new LexicalScanner.Rule(TIMES, P_TIMES),
    new LexicalScanner.Rule(DIVIDE,P_DIVIDE),
    new LexicalScanner.Rule(LEFT_PAREN,P_LEFTPAREN),
    new LexicalScanner.Rule(RIGHT_PAREN,P_RIGHTPAREN),
    new LexicalScanner.Rule(NUMBER,LexicalScanner.P_FLOAT),
    new LexicalScanner.Rule(IDENTIFIER,LexicalScanner.P_IDENTIFIER)
  }

Definition at line 30 of file Tokenizer.java.

Referenced by aud.example.expr.Tokenizer.Tokenizer().

TIMES

final int aud.example.expr.Tokenizer.TIMES ='*'

static

Definition at line 21 of file Tokenizer.java.

Referenced by aud.example.expr.ExpressionParser.product().

---

The documentation for this class was generated from the following file:

• Tokenizer.java