Examples of fri.patterns.interpreter.parsergenerator.Token

Package fri.patterns.interpreter.parsergenerator

Examples of fri.patterns.interpreter.parsergenerator.Token

fri.patterns.interpreter.parsergenerator.Token
Lexer-Parser communication struct. Utility methods defining EPSILON and terminals. Definition of all global constants for parsergenerator packages.
This class defines special token identifiers: "token" (Token.TOKEN) and "ignored" (Token.IGNORED), which are needed when specifiying a syntax with mixed parser and lexer rules.
This class defines special symbols needed to define character sets with Token.UPTO (e.g. A..Z) and intersections with Token.BUTNOT (e.g. `char` - `newline`). @author (c) 2000, Fritz Ritzberger

        if (listeners != null && listeners.size() > 0)  // creating a token takes time, do it only when listeners are present
          fireTokenReceived(createToken(item.getTokenIdentifier(), item.getResultTree(), lexerSemantic), true);
        return getNextToken(expectedTokenSymbols);
      }
      else  {
        Token token = createToken(item.getTokenIdentifier(), item.getResultTree(), lexerSemantic);
        fireTokenReceived(token, false);
        return token;
      }
    }

View Full Code Here

    return createToken(tokenIdentifier, result.toString(), result.getRange());  // toString() takes time as it builds the token text
  }
  
  /** Token factory method. Can be overridden to convert token.text to some Java object. */
  protected Token createToken(String tokenIdentifier, String text, Token.Range range)  {
    return new Token(tokenIdentifier, text, range);
  }

View Full Code Here

      InputStream in = ExampleLexer.class.getResourceAsStream("ExampleLexer.java");  //Reader in = new FileReader(args[0]);
      lexer.setInput(in);


      System.err.println("time to build input was: "+ts.getInterval());
      
      Token t;
      do  {
        t = lexer.getNextToken(null);
        //System.err.println(ts.getInterval()+" was time to read token "+t.symbol+" -> "+t.text);
        
        if (t.symbol == null)  {

View Full Code Here

    throws LexerException, IOException
  {
    try  {
      lexer.setInput(in);


      Token t;
      do  {
        t = lexer.getNextToken(null);
        
        if (t.symbol == null)
          lexer.dump(System.err);

View Full Code Here

TOP

Related Classes of fri.patterns.interpreter.parsergenerator.Token

fri.patterns.interpreter.parsergenerator.lexer.examples.CStyleCommentStrip

fri.patterns.interpreter.parsergenerator.lexer.examples.ExampleLexer

fri.patterns.interpreter.parsergenerator.lexer.LexerImpl

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.