Examples of net.sf.collabreview.transform.tokened.Tokenizer

net.sf.collabreview.transform.tokened.Tokenizer

The Tokenizer takes a string and splits it into tokens. However, the Tokenizer is a bit different from normal Tokenizers because of the special nature of the Tokens that the Tokenizer creates from the tokenized string. A Token consists of two parts: a core and a rim. The core is the actual normalized content of a Token while the rim is irrelevant noise surrounding and permeating the core.

The Tokens owned by the Tokenizer completely cover the entire original string. Thus for CollabReview a Token is e.g. a complete line of source code with indentation being its rim.

All (except the last) Tokens must also include the Token terminator character/string. As the last Token is usually not terminated the Tokenizer must supply a virtual terminating string with the respective method. @author Christian Prause (chris) @date 2009-05-25 16:57:16

  public void setExternalFilter(TrivialTokenFilter externalFilter) {
    this.externalFilter = externalFilter;
  }


  public void addTokensFromArtifact(Artifact artifact) {
    Tokenizer tokenizer = owningRepository.getTokenizerFactory().create(artifact.getContent());
    List<Token> tokens = tokenizer.listTokens();
    for (Token token : tokens) {
      IndexEntry entry = getIndexEntryForToken(token, true);
      if (!isTrivialToken(token)) {
        entry.addOrigin(artifact.getId());
        testNonTriviality(entry, token);

Examples of net.sf.collabreview.transform.tokened.Tokenizer

Related Classes of net.sf.collabreview.transform.tokened.Tokenizer