Examples of org.apache.commons.lang3.text.StrTokenizer

org.apache.commons.lang3.tuple.ImmutablePair
Tokenizes a string based based on delimiters (separators) and supporting quoting and ignored character concepts.
This class can split a String into many smaller strings. It aims to do a similar job to {@link java.util.StringTokenizer StringTokenizer}, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.
The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.
Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.
Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.
At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.
Empty tokens may be removed or returned as null.
```
 "a,b,c"         - Three tokens "a","b","c"   (comma delimiter) " a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace) "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched) 
```
This tokenizer has the following properties and options:

Property Type Default

delim CharSetMatcher { \t\n\r\f}

quote NoneMatcher {}

ignore NoneMatcher {}

emptyTokenAsNull boolean false

ignoreEmptyTokens boolean true

@since 2.2 @version $Id: StrTokenizer.java 1153241 2011-08-02 18:49:52Z ggregory $

Property	Type	Default
delim	CharSetMatcher	{ \t\n\r\f}
quote	NoneMatcher	{}
ignore	NoneMatcher	{}
emptyTokenAsNull	boolean	false
ignoreEmptyTokens	boolean	true

    decomponeQuery();
  }


  // PRIVATE METHODS
  private void decomponeQuery() {
    StrTokenizer tokenizer = new StrTokenizer(originalQuery);
    Pattern pattern = Pattern.compile("\\$\\{.*?\\}");


    String partialStr = "";
    String logicOperator = "";
    String prevOperator = "";
    while (tokenizer.hasNext()) {
      String elem = (String) tokenizer.next();


      if (logicOperators.containsKey(elem)) {
        logicOperator += elem + " ";
      } else {
        int iterations = 1;
        if (!tokenizer.hasNext())
          iterations = 2;


        for (int i = 0; i < iterations; i++) {
          if (!logicOperator.equals("") || !tokenizer.hasNext()) {
            StringBuilder partStr = new StringBuilder(partialStr);


            QueryPart queryPart = new QueryPart();


            Matcher matcher = pattern.matcher(partStr.toString());

View Full Code Here

            final Map<String, String> uriTemplateVariables) {
        if (path.startsWith(pathSeparator) != pattern.startsWith(pathSeparator)) {
            return false;
        }


        final String[] patternDirs = new StrTokenizer(pattern, pathSeparator)
                .setIgnoreEmptyTokens(true).getTokenArray();
        final String[] pathDirs = new StrTokenizer(path, pathSeparator)
                .setIgnoreEmptyTokens(true).getTokenArray();


        int pattIdxStart = 0;
        int pattIdxEnd = patternDirs.length - 1;
        int pathIdxStart = 0;

View Full Code Here

     * <code>pattern</code>' and '<code>path</code>', but does
     * <strong>not</strong> enforce this.
     */
    public String extractPathWithinPattern(final String pattern,
            final String path) {
        final String[] patternParts = new StrTokenizer(pattern, pathSeparator)
                .setIgnoreEmptyTokens(true).getTokenArray();
        final String[] pathParts = new StrTokenizer(path, pathSeparator)
                .setIgnoreEmptyTokens(true).getTokenArray();


        final StringBuilder builder = new StringBuilder();


        // Add any path parts that have a wildcarded pattern part.

View Full Code Here

            return;
        }


        final Set<String> disallowedOperationSet = new HashSet<String>();
        if (!"".equals(disallowedOperations)) {
            final String[] disallowedOperationsTokens = new StrTokenizer(
                    disallowedOperations, ",").getTokenArray();
            for (final String operation : disallowedOperationsTokens) {
                if (!("create".equals(operation) || "update".equals(operation) || "delete"
                        .equals(operation))) {
                    LOGGER.warning("-disallowedOperations options can only contain 'create', 'update', 'delete': -disallowedOperations update,delete");

View Full Code Here

        if (StringUtils.isBlank(filter)) {
            return finders;
        }


        final Set<String> requiredEntries = new HashSet<String>();
        final String[] filterTokens = new StrTokenizer(filter, ",")
                .getTokenArray();
        for (final String requiredString : filterTokens) {
            requiredEntries.add(requiredString.toLowerCase());
        }
        if (requiredEntries.isEmpty()) {

View Full Code Here

     * @see <a href="http://www.w3.org/International/questions/qa-escapes">Using character escapes in markup and CSS</a>
     * @see <a href="https://issues.apache.org/jira/browse/LANG-728">LANG-728</a>
     */
    @Test
    public void testEscapeXmlSupplementaryCharacters() {
        CharSequenceTranslator escapeXml = 
            StringEscapeUtils.ESCAPE_XML.with( NumericEntityEscaper.between(0x7f, Integer.MAX_VALUE) );


        assertEquals("Supplementary character must be represented using a single escape", "&#144308;",
                escapeXml.translate("\uD84C\uDFB4"));
    }

View Full Code Here

     * @see <a href="http://www.w3.org/International/questions/qa-escapes">Using character escapes in markup and CSS</a>
     * @see <a href="https://issues.apache.org/jira/browse/LANG-728">LANG-728</a>
     */
    @Test
    public void testEscapeXmlSupplementaryCharacters() {
        CharSequenceTranslator escapeXml = 
            StringEscapeUtils.ESCAPE_XML.with( NumericEntityEscaper.between(0x7f, Integer.MAX_VALUE) );


        assertEquals("Supplementary character must be represented using a single escape", "&#144308;",
                escapeXml.translate("\uD84C\uDFB4"));
    }

View Full Code Here

     * @see <a href="https://issues.apache.org/jira/browse/LANG-728">LANG-728</a>
     */
    @Test
    @SuppressWarnings( "deprecation" ) // ESCAPE_XML has been replaced by ESCAPE_XML10 and ESCAPE_XML11 in 3.3
    public void testEscapeXmlSupplementaryCharacters() {
        final CharSequenceTranslator escapeXml = 
            StringEscapeUtils.ESCAPE_XML.with( NumericEntityEscaper.between(0x7f, Integer.MAX_VALUE) );


        assertEquals("Supplementary character must be represented using a single escape", "&#144308;",
                escapeXml.translate("\uD84C\uDFB4"));


        assertEquals("Supplementary characters mixed with basic characters should be encoded correctly", "a b c &#144308;",
                        escapeXml.translate("a b c \uD84C\uDFB4"));
    }

View Full Code Here

    @SuppressWarnings( "deprecation" ) // ESCAPE_XML has been replaced by ESCAPE_XML10 and ESCAPE_XML11 in 3.3
    public void testEscapeXmlAllCharacters() {
        // http://www.w3.org/TR/xml/#charsets says:
        // Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] /* any Unicode character,
        // excluding the surrogate blocks, FFFE, and FFFF. */
        final CharSequenceTranslator escapeXml = StringEscapeUtils.ESCAPE_XML
                .with(NumericEntityEscaper.below(9), NumericEntityEscaper.between(0xB, 0xC), NumericEntityEscaper.between(0xE, 0x19),
                        NumericEntityEscaper.between(0xD800, 0xDFFF), NumericEntityEscaper.between(0xFFFE, 0xFFFF), NumericEntityEscaper.above(0x110000));


        assertEquals("&#0;&#1;&#2;&#3;&#4;&#5;&#6;&#7;&#8;", escapeXml.translate("\u0000\u0001\u0002\u0003\u0004\u0005\u0006\u0007\u0008"));
        assertEquals("\t", escapeXml.translate("\t")); // 0x9
        assertEquals("\n", escapeXml.translate("\n")); // 0xA
        assertEquals("&#11;&#12;", escapeXml.translate("\u000B\u000C"));
        assertEquals("\r", escapeXml.translate("\r")); // 0xD
        assertEquals("Hello World! Ain&apos;t this great?", escapeXml.translate("Hello World! Ain't this great?"));
        assertEquals("&#14;&#15;&#24;&#25;", escapeXml.translate("\u000E\u000F\u0018\u0019"));
    }

View Full Code Here

        String sql = "SELECT v.proposal_id as proposalId, avg(v.note) as moy, count(v.proposal_id) as nbVote, t.title as proposalTitle FROM vote v, proposal t where (v.proposal_id=t.id) group by v.proposal_id order by moy desc";
        List<SqlRow> rows = Ebean.createSqlQuery(sql).findList();
        
        Map<Long, Pair<Double, Integer>> moyennes = new HashMap<Long, Pair<Double, Integer>>();
        for (SqlRow row : rows) {
            Pair<Double, Integer> moyProposal = new ImmutablePair(row.getDouble("moy"), row.getInteger("nbVote"));
            moyennes.put(row.getLong("proposalId"), moyProposal);
        }
        
        return moyennes;
    }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.commons.lang3.text.StrTokenizer

au.org.intersect.samifier.domain.PeptideSearchResult

co.jirm.orm.writer.SqlWriterStrategy

com.crawljax.forms.FormInput

com.lassitercg.faces.components.event.SheetUpdate

com.lassitercg.faces.components.sheet.BadUpdate

com.lassitercg.faces.components.sheet.Sheet$RowColIndex

com.softwareloop.tstconfig.Config

hudson.plugins.git.GitSCM

hudson.plugins.git.opt.PreBuildMergeOptions

jp.vmi.selenium.selenese.DriverIndependentTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.