Examples of CommonTokenStream

com.dotcms.repackage.org.antlr.runtime.CommonTokenStream

org.antlr.runtime.CommonTokenStream

The most common stream of tokens is one where every token is buffered up and tokens are prefiltered for a certain channel (the parser will only see these tokens and cannot change the filter channel number during the parse). TODO: how to access the full token stream? How to track all tokens matched per rule?

org.antlr.v4.runtime.CommonTokenStream

This class extends {@link BufferedTokenStream} with functionality to filtertoken streams to tokens on a particular channel (tokens where {@link Token#getChannel} returns a particular value).

This token stream provides access to all tokens by index or when calling methods like {@link #getText}. The channel filtering is only used for code accessing tokens via the lookahead methods {@link #LA}, {@link #LT}, and {@link #LB}.

By default, tokens are placed on the default channel ( {@link Token#DEFAULT_CHANNEL}), but may be reassigned by using the {@code ->channel(HIDDEN)} lexer command, or by using an embedded action tocall {@link Lexer#setChannel}.

Note: lexer rules which use the {@code ->skip} lexer command or call{@link Lexer#skip} do not produce tokens at all, so input text matched bysuch a rule will not be available as part of the token stream, regardless of channel.

    setup(cu);
    registry.clearForType(JavaCompletionTypes.CUSTOM_TYPE);
    registry.clearForType(JavaCompletionTypes.FIELD);
    registry.clearForType(JavaCompletionTypes.NAME);
    Lexer lexer = new JavaLexer(new ANTLRInputStream(txt));
    CommonTokenStream tokens = new CommonTokenStream(lexer);


    // Create a parser that reads from the scanner
    JavaParser parser = new JavaParser(tokens);
    parser.removeErrorListeners();

        .openStream();
    ANTLRInputStream input = new ANTLRInputStream(is);
    MsonLexer lexer = new MsonLexer(input);
//    SymbolASTFactory factory = new SymbolASTFactory();
//    lexer.setTokenFactory(factory);
    CommonTokenStream tokens = new CommonTokenStream(lexer);
    
    MsonParser parser = new MsonParser(tokens);
//    parser.setTokenFactory(factory);
    parser.addErrorListener(new DiagnosticErrorListener());
    parser.getInterpreter().setPredictionMode(

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of CommonTokenStream

Examples of org.antlr.v4.runtime.CommonTokenStream

Examples of org.antlr.v4.runtime.CommonTokenStream