Examples of Tokenizer

  • org.apache.jena.riot.tokens.Tokenizer
  • org.apache.lucene.analysis.Tokenizer
    A Tokenizer is a TokenStream whose input is a Reader.

    This is an abstract class.

    NOTE: subclasses must override {@link #incrementToken()} if the new TokenStream API is usedand {@link #next(Token)} or {@link #next()} if the oldTokenStream API is used.

    NOTE: Subclasses overriding {@link #incrementToken()} mustcall {@link AttributeSource#clearAttributes()} beforesetting attributes. Subclasses overriding {@link #next(Token)} must call{@link Token#clear()} before setting Token attributes.

  • org.apache.myfaces.trinidadinternal.el.Tokenizer
    converts a EL expression into tokens. @author The Oracle ADF Faces Team
  • org.apache.uima.lucas.indexer.Tokenizer
  • org.crsh.cli.impl.tokenizer.Tokenizer
  • org.eclipse.orion.server.cf.manifest.v2.Tokenizer
  • org.eclipse.osgi.framework.internal.core.Tokenizer
    Simple tokenizer class. Used to parse data.
  • org.exist.storage.analysis.Tokenizer
  • org.geoserver.ows.util.KvpUtils.Tokenizer
  • org.hsqldb.Tokenizer
    Provides the ability to tokenize SQL character sequences. Extensively rewritten and extended in successive versions of HSQLDB. @author Thomas Mueller (Hypersonic SQL Group) @version 1.8.0 @since Hypersonic SQL
  • org.jboss.dna.common.text.TokenStream.Tokenizer
  • org.jboss.forge.shell.command.parser.Tokenizer
    @author Lincoln Baxter, III
  • org.jstripe.tokenizer.Tokenizer
  • org.languagetool.tokenizers.Tokenizer
    Interface for classes that tokenize text into smaller units. @author Daniel Naber
  • org.modeshape.common.text.TokenStream.Tokenizer
  • org.openjena.riot.tokens.Tokenizer
  • org.radargun.utils.Tokenizer
    Tokenizer that allows string delims instead of char delims @author Radim Vansa <rvansa@redhat.com>
  • org.sonatype.maven.polyglot.atom.parsing.Tokenizer
    Taken from the Loop programming language compiler pipeline. @author dhanji@gmail.com (Dhanji R. Prasanna)
  • org.spoofax.jsglr.client.imploder.Tokenizer
  • org.supercsv_voltpatches.tokenizer.Tokenizer
    Reads the CSV file, line by line. If you want the line-reading functionality of this class, but want to define your own implementation of {@link #readColumns(List)}, then consider writing your own Tokenizer by extending AbstractTokenizer. @author Kasper B. Graversen @author James Bassett
  • org.zkoss.selector.lang.Tokenizer
    @author simonpai
  • weka.core.tokenizers.Tokenizer
    A superclass for all tokenizer algorithms. @author FracPete (fracpete at waikato dot ac dot nz) @version $Revision: 1.3 $

    • Examples of edu.stanford.nlp.process.Tokenizer

    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33

      public static void main(String[] args) throws IOException {

        Reader in = new FileReader(args[0]);
        Tokenizer st = new NegraPennTokenizer(in);

        while (st.hasNext()) {
          String s = (String) st.next();
          System.out.println(s);
        }
      }
    View Full Code Here

    Examples of edu.umd.cs.findbugs.Tokenizer

    407
    408
    409
    410
    411
    412
    413
    414
    415
    416
    417
    418
    419
    420
    421
    422
    423
    424
    425
                int offset = sourceFile.getLineOffset(scanStartLine - 1);
                if (offset < 0)
                {
                    return false; // Source file has changed?
                }
                Tokenizer tokenizer = new Tokenizer(UTF8.reader(sourceFile.getInputStreamFromOffset(offset)));

                // Read the tokens into an ArrayList,
                // keeping track of where the catch block is reported
                // to start
                ArrayList<Token> tokenList = new ArrayList<Token>(40);
                int eolOfCatchBlockStart = -1;
                for (int line = scanStartLine; line < scanStartLine + MAX_LINES;) {
                    Token token = tokenizer.next();
                    int kind = token.getKind();
                    if (kind == Token.EOF) {
                        break;
                    }
    View Full Code Here

    Examples of es.upv.simulator.Tokenizer

    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
            nMuestras = 0;
            posMuestra = 0.0F;
            velozMuestreo = 6000F;
            try
            {
                Tokenizer tok = new Tokenizer(fMuestra);
                tok.whitespaceChars(58, 255);
                tok.wordChars(48, 57);
                valorMax = tok.readFloat();
                valorMin = tok.readFloat();
                nMuestras = tok.readInt();
                velozMuestreo = tok.readInt();
                int longitud = nMuestras * 2;
                byte buf[] = new byte[longitud];
                fMuestra.skip(1L);
                int num = fMuestra.read(buf, 0, longitud);
                if(num >= 0)
    View Full Code Here

    Examples of ivory.core.tokenize.Tokenizer

    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
        // set average doc length
        mModel.setAvgDocLength(avgLen);

        List<HMapSFW> transDocs = new ArrayList<HMapSFW>();
        Tokenizer tokenizer = TokenizerFactory.createTokenizer(eLang,
            eTokenizerModelFile, true, eStopwordsFile, eStopwordsFile + ".stemmed", null);

        // translate doc texts here
        for (HMapSIW deDoc : docs) {
          HMapIFW tfS = new HMapIFW();
    View Full Code Here

    Examples of net.percederberg.grammatica.parser.Tokenizer

    212
    213
    214
    215
    216
    217
    218
    219
    220
    221
    222
    223
    224
    225
    226
    227
         *             or initialized correctly
         */
        public Tokenizer createTokenizer(Reader in)
            throws GrammarException {

            Tokenizer  tokenizer;

            try {
                tokenizer = new Tokenizer(in, !getCaseSensitive());
                for (int i = 0; i < tokens.size(); i++) {
                    tokenizer.addPattern((TokenPattern) tokens.get(i));
                }
            } catch (ParserCreationException e) {
                if (e.getName() == null) {
                    throw new GrammarException(fileName, e.getMessage());
                } else {
    View Full Code Here

    Examples of net.sf.collabreview.transform.tokened.Tokenizer

    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
      public void setExternalFilter(TrivialTokenFilter externalFilter) {
        this.externalFilter = externalFilter;
      }

      public void addTokensFromArtifact(Artifact artifact) {
        Tokenizer tokenizer = owningRepository.getTokenizerFactory().create(artifact.getContent());
        List<Token> tokens = tokenizer.listTokens();
        for (Token token : tokens) {
          IndexEntry entry = getIndexEntryForToken(token, true);
          if (!isTrivialToken(token)) {
            entry.addOrigin(artifact.getId());
            testNonTriviality(entry, token);
    View Full Code Here

    Examples of net.sf.mpxj.utility.Tokenizer

    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
             // Read the file creation record. At this point we are reading
             // directly from an input stream so no character set decoding is
             // taking place. We assume that any text in this record will not
             // require decoding.
             //
             Tokenizer tk = new InputStreamTokenizer(bis);
             tk.setDelimiter(m_delimiter);

             Record record;
             String number;

             //
             // Add the header record
             //
             parseRecord(Integer.toString(MPXConstants.FILE_CREATION_RECORD_NUMBER), new Record(m_locale, tk, m_formats));
             ++line;

             //
             // Now process the remainder of the file in full. As we have read the
             // file creation record we have access to the field which specifies the
             // codepage used to encode the character set in this file. We set up
             // an input stream reader using the appropriate character set, and
             // create a new tokenizer to read from this Reader instance.
             //
             InputStreamReader reader = new InputStreamReader(bis, m_projectFile.getFileCreationRecord().getCodePage().getCharset());
             tk = new ReaderTokenizer(reader);
             tk.setDelimiter(m_delimiter);

             //
             // Read the remainder of the records
             //
             while (tk.getType() != Tokenizer.TT_EOF)
             {
                record = new Record(m_locale, tk, m_formats);
                number = record.getRecordNumber();

                if (number != null)
    View Full Code Here

    Examples of net.sf.saxon.expr.Tokenizer

    1562
    1563
    1564
    1565
    1566
    1567
    1568
    1569
    1570
    1571
    1572
    1573
    1574
    1575
    1576
    1577
    1578
    1579
    1580
    1581
    1582
    1583
    1584
    1585
    1586
        return xml;
      }

      public static String replaceNameInPathOrQuery( String pathOrQuery, String oldName, String newName ) throws Exception
      {
        Tokenizer t = new Tokenizer();
        t.tokenize( pathOrQuery, 0, -1, 1 );
        StringBuffer result = new StringBuffer();
        int lastIx = 0;

        while( t.currentToken != Token.EOF )
        {
          if( t.currentToken == Token.NAME && t.currentTokenValue.equals( oldName ) )
          {
            result.append( pathOrQuery.substring( lastIx, t.currentTokenStartOffset ) );
            result.append( newName );
            lastIx = t.currentTokenStartOffset + t.currentTokenValue.length();
          }

          t.next();
        }

        if( lastIx < pathOrQuery.length() )
          result.append( pathOrQuery.substring( lastIx ) );
        //
    View Full Code Here

    Examples of net.sourceforge.jdbclogger.core.util.Tokenizer

    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
        public PreparedStatementWrapper(
                PreparedStatement statement, String sql, List formatters)
        {
            super(statement);

            Tokenizer tokenizer = new Tokenizer(sql,new String[]{"?"},true,
                    true,'\"','\"');

            int delimiterCount = 1;

            while(tokenizer.hasMoreTokens())
            {
                String token = tokenizer.nextToken();

                if(token.equals("?"))
                {
                    _paramIndexToListIndex.put(new Integer(delimiterCount),
                            new Integer(getSqlAsList().size()));
    View Full Code Here

    Examples of net.sourceforge.pmd.cpd.Tokenizer

    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
      private TokenizerBridge bridge;

      @Before
      public void setUp() {
        Tokenizer tokenizer = new Tokenizer() {
          public void tokenize(SourceCode tokens, Tokens tokenEntries) throws IOException {
            tokenEntries.add(new TokenEntry("t1", "src", 1));
            tokenEntries.add(new TokenEntry("t2", "src", 1));
            tokenEntries.add(new TokenEntry("t3", "src", 2));
            tokenEntries.add(new TokenEntry("t1", "src", 4));
    View Full Code Here
    TOP
    Copyright © 2018 www.massapi.com. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.