Examples of de.susebox.jtopas.Token

de.susebox.jtopas.Token

Instances of this class are created by the classes implementing the {@link Tokenizer} interface. Token describes a portion of text according to the settings given to the producing {@link Tokenizer} in form of a {@link TokenizerProperties} object. Beside the token type the token image itself, its position in the input stream, line and column position and associated informations can be obtained from the Token (provided, the nessecary parse flags are set in the tokenizer).

This class replaces the older {@link de.susebox.java.util.Token} which isdeprecated.

@author Heiko Blau @see Tokenizer @see TokenizerProperties

    long    start       = System.currentTimeMillis();
    long    localStart  = start;
    
    // tokenizer loop
    while (tokenizer.hasMoreToken()) {
      Token token = tokenizer.nextToken();
      int   type  = token.getType();
      
      switch (type) {
      case Token.WHITESPACE:
      case Token.EOF:
        break;
      default:
        // use if instead of assertTrue alone for performance reasons (no string
        // concatenation nessecary)
        // System.out.println(tokenizer.currentImage());
        if (expected[index] != type) {
          assertTrue("Line/Column " + token.getStartLine() + "/" + token.getStartColumn() + ": Expected " + Token.getTypeName(expected[index]) + ", got " + Token.getTypeName(type) + ".\n" + tokenizer.currentImage(), false);
        }
        if (++index >= expected.length) {
          long localEnd = System.currentTimeMillis();
          
          diffTime   = localEnd - localStart;

Examples of de.susebox.jtopas.Token

Related Classes of de.susebox.jtopas.Token