Examples of WordDelimiterFilter

The combinations parameter affects how subwords are combined: One use for {@link WordDelimiterFilter} is to help match words with differentsubword delimiters. For example, if the source text contained "wi-fi" one may want "wifi" "WiFi" "wi-fi" "wi+fi" queries to all match. One way of doing so is to specify combinations="1" in the analyzer used for indexing, and combinations="0" (the default) in the analyzer used for querying. Given that the current {@link StandardTokenizer} immediately removes many intra-worddelimiters, it is recommended that this filter be used after a tokenizer that does not do this (such as {@link WhitespaceTokenizer}).
  • org.apache.solr.analysis.WordDelimiterFilter
    ---there, 'dude'" -> "hello", "there", "dude" - trailing "'s" are removed for each subword - "O'Neil's" -> "O", "Neil" - Note: this step isn't performed in a separate filter because of possible subword combinations. The combinations parameter affects how subwords are combined: - combinations="0" causes no subword combinations. - "PowerShot" -> 0:"Power", 1:"Shot" (0 and 1 are the token positions) - combinations="1" means that in addition to the subwords, maximum runs of non-numeric subwords are catenated and produced at the same position of the last subword in the run. - "PowerShot" -> 0:"Power", 1:"Shot" 1:"PowerShot" - "A's+B's&C's" -> 0:"A", 1:"B", 2:"C", 2:"ABC" - "Super-Duper-XL500-42-AutoCoder!" -> 0:"Super", 1:"Duper", 2:"XL", 2:"SuperDuperXL", 3:"500" 4:"42", 5:"Auto", 6:"Coder", 6:"AutoCoder" One use for WordDelimiterFilter is to help match words with different subword delimiters. For example, if the source text contained "wi-fi" one may want "wifi" "WiFi" "wi-fi" "wi+fi" queries to all match. One way of doing so is to specify combinations="1" in the analyzer used for indexing, and combinations="0" (the default) in the analyzer used for querying. Given that the current StandardTokenizer immediately removes many intra-word delimiters, it is recommended that this filter be used after a tokenizer that does not do this (such as WhitespaceTokenizer). @version $Id: WordDelimiterFilter.java 1166766 2011-09-08 15:52:10Z rmuir $

    • Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
      }

      @Override
      public TokenFilter create(TokenStream input) {
        if (luceneMatchVersion.onOrAfter(Version.LUCENE_48)) {
          return new WordDelimiterFilter(luceneMatchVersion, input, typeTable == null ? WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE : typeTable,
                                       flags, protectedWords);
        } else {
          return new Lucene47WordDelimiterFilter(input, typeTable == null ? WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE : typeTable,
                                      flags, protectedWords);
        }
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    275
    276
    277
    278
    279
    280
    281
    282
    283
    284
    285
        Analyzer a = new Analyzer() {
          @Override
          protected TokenStreamComponents createComponents(String fieldName, Reader reader) {
            Tokenizer tokenizer = new WikipediaTokenizer(reader);
            TokenStream stream = new SopTokenFilter(tokenizer);
            stream = new WordDelimiterFilter(TEST_VERSION_CURRENT, stream, table, -50, protWords);
            stream = new SopTokenFilter(stream);
            return new TokenStreamComponents(tokenizer, stream);
          } 
        };
        checkAnalysisConsistency(random(), a, false, "B\u28c3\ue0f8[ \ud800\udfc2 </p> jb");
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
      @Test
      public void testOffsets() throws IOException {
        int flags = GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | CATENATE_ALL | SPLIT_ON_CASE_CHANGE | SPLIT_ON_NUMERICS | STEM_ENGLISH_POSSESSIVE;
        // test that subwords and catenated subwords have
        // the correct offsets.
        WordDelimiterFilter wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new SingleTokenTokenStream(new Token("foo-bar", 5, 12)), DEFAULT_WORD_DELIM_TABLE, flags, null);

        assertTokenStreamContents(wdf,
            new String[] { "foo", "foobar", "bar" },
            new int[] { 5, 5, 9 },
            new int[] { 8, 12, 12 });

        wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new SingleTokenTokenStream(new Token("foo-bar", 5, 6)), DEFAULT_WORD_DELIM_TABLE, flags, null);
       
        assertTokenStreamContents(wdf,
            new String[] { "foo", "bar", "foobar" },
            new int[] { 5, 5, 5 },
            new int[] { 6, 6, 6 });
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
      }
     
      @Test
      public void testOffsetChange() throws Exception {
        int flags = GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | CATENATE_ALL | SPLIT_ON_CASE_CHANGE | SPLIT_ON_NUMERICS | STEM_ENGLISH_POSSESSIVE;
        WordDelimiterFilter wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new SingleTokenTokenStream(new Token("übelkeit)", 7, 16)), DEFAULT_WORD_DELIM_TABLE, flags, null);
       
        assertTokenStreamContents(wdf,
            new String[] { "übelkeit" },
            new int[] { 7 },
            new int[] { 15 });
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
      }
     
      @Test
      public void testOffsetChange2() throws Exception {
        int flags = GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | CATENATE_ALL | SPLIT_ON_CASE_CHANGE | SPLIT_ON_NUMERICS | STEM_ENGLISH_POSSESSIVE;
        WordDelimiterFilter wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new SingleTokenTokenStream(new Token("(übelkeit", 7, 17)), DEFAULT_WORD_DELIM_TABLE, flags, null);
       
        assertTokenStreamContents(wdf,
            new String[] { "übelkeit" },
            new int[] { 8 },
            new int[] { 17 });
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
      }
     
      @Test
      public void testOffsetChange3() throws Exception {
        int flags = GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | CATENATE_ALL | SPLIT_ON_CASE_CHANGE | SPLIT_ON_NUMERICS | STEM_ENGLISH_POSSESSIVE;
        WordDelimiterFilter wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new SingleTokenTokenStream(new Token("(übelkeit", 7, 16)), DEFAULT_WORD_DELIM_TABLE, flags, null);
       
        assertTokenStreamContents(wdf,
            new String[] { "übelkeit" },
            new int[] { 8 },
            new int[] { 16 });
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
      }
     
      @Test
      public void testOffsetChange4() throws Exception {
        int flags = GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | CATENATE_ALL | SPLIT_ON_CASE_CHANGE | SPLIT_ON_NUMERICS | STEM_ENGLISH_POSSESSIVE;
        WordDelimiterFilter wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new SingleTokenTokenStream(new Token("(foo,bar)", 7, 16)), DEFAULT_WORD_DELIM_TABLE, flags, null);
       
        assertTokenStreamContents(wdf,
            new String[] { "foo", "foobar", "bar"},
            new int[] { 8, 8, 12 },
            new int[] { 11, 15, 15 });
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
            new int[] { 11, 15, 15 });
      }

      public void doSplit(final String input, String... output) throws Exception {
        int flags = GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | SPLIT_ON_CASE_CHANGE | SPLIT_ON_NUMERICS | STEM_ENGLISH_POSSESSIVE;
        WordDelimiterFilter wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new MockTokenizer(
                    new StringReader(input), MockTokenizer.KEYWORD, false), WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE, flags, null);
       
        assertTokenStreamContents(wdf, output);
      }
    View Full Code Here

    Examples of org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter

    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
      }
     
      public void doSplitPossessive(int stemPossessive, final String input, final String... output) throws Exception {
        int flags = GENERATE_WORD_PARTS | GENERATE_NUMBER_PARTS | SPLIT_ON_CASE_CHANGE | SPLIT_ON_NUMERICS;
        flags |= (stemPossessive == 1) ? STEM_ENGLISH_POSSESSIVE : 0;
        WordDelimiterFilter wdf = new WordDelimiterFilter(TEST_VERSION_CURRENT, new MockTokenizer(
            new StringReader(input), MockTokenizer.KEYWORD, false), flags, null);

        assertTokenStreamContents(wdf, output);
      }
    View Full Code Here

    Examples of org.apache.solr.analysis.WordDelimiterFilter

    114
    115
    116
    117
    118
    119
    120
    121
          flags |= STEM_ENGLISH_POSSESSIVE;
        }
      }

      public WordDelimiterFilter create(TokenStream input) {
        return new WordDelimiterFilter(input, typeTable == null ? WordDelimiterIterator.DEFAULT_WORD_DELIM_TABLE : typeTable,
                                       flags, protectedWords);
      }
    View Full Code Here
    TOP
    Copyright © 2018 www.massapi.com. All rights reserved.
    All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.