Examples of org.sindice.siren.analysis.JsonTokenizer

org.sindice.siren.analysis.JsonTokenizer

A tokenizer for data following the JSON syntax.

The tokenizer parses JSON data and generates a token for each field name and each value. The tokenizer attaches a node label and a datatype to each token.

Regarding datatype, the convention is the following:

If a field name is parsed, the datatype {@link JSONDatatype#JSON_FIELD} is assigned;
If a value string is parsed, the datatype {@link XSDDatatype#XSD_STRING} is assigned;
If a boolean value is parsed, the datatype {@link XSDDatatype#XSD_BOOLEAN} is assigned;
If a numerical value is parsed, the datatype {@link XSDDatatype#XSD_LONG} is assigned;
If a numerical value with a fraction is parsed, the datatype {@link XSDDatatype#XSD_DOUBLE} is assigned;


      @Override
      protected TokenStreamComponents createComponents(final String fieldName,
                                                       final Reader reader) {
        final Version matchVersion = Version.LUCENE_40;
        final JsonTokenizer src = new JsonTokenizer(reader);
        TokenStream tok = new DatatypeAnalyzerFilter(matchVersion, src,
          new StandardAnalyzer(matchVersion),
          new StandardAnalyzer(matchVersion));
        tok = new LowerCaseFilter(matchVersion, tok);
        // The PositionAttributeFilter and SirenPayloadFilter are mandatory

    super.init(args);
  }


  @Override
  public JsonTokenizer create(final Reader input) {
    return new JsonTokenizer(input);
  }

    args.put(URILocalnameFilterFactory.MAXLENGTH_KEY, "8");
    final URILocalnameFilterFactory factory = new URILocalnameFilterFactory();
    factory.init(args);


    final Reader reader = new StringReader("{ \"uri\" : \"http://test/anotherLocalname\" }");
    final TokenStream stream = factory.create(new JsonTokenizer(reader));
    this.assertTokenStreamContents(stream,
        new String[] { "uri", "anotherLocalname", "http://test/anotherLocalname" });
  }

Examples of org.sindice.siren.analysis.JsonTokenizer

Related Classes of org.sindice.siren.analysis.JsonTokenizer