Examples of org.apache.commons.lang.text.StrTokenizer

org.apache.commons.lang.text.StrTokenizer
Tokenizes a string based based on delimiters (separators) and supporting quoting and ignored character concepts.
This class can split a String into many smaller strings. It aims to do a similar job to {@link java.util.StringTokenizer StringTokenizer}, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.
The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.
Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.
Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.
At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.
Empty tokens may be removed or returned as null.
```
 "a,b,c"         - Three tokens "a","b","c"   (comma delimiter) " a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace) "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched) 
```
This tokenizer has the following properties and options:

Property Type Default

delim CharSetMatcher { \t\n\r\f}

quote NoneMatcher {}

ignore NoneMatcher {}

emptyTokenAsNull boolean false

ignoreEmptyTokens boolean true

@author Apache Software Foundation @author Matthew Inger @author Gary D. Gregory @since 2.2 @version $Id: StrTokenizer.java 907631 2010-02-08 12:22:48Z sebb $

Property	Type	Default
delim	CharSetMatcher	{ \t\n\r\f}
quote	NoneMatcher	{}
ignore	NoneMatcher	{}
emptyTokenAsNull	boolean	false
ignoreEmptyTokens	boolean	true


    public List<String> split(String input) {
        List<String> statements = new ArrayList<String>();
        StrBuilder currentSql = new StrBuilder();


        StrTokenizer lineTokenizer = new StrTokenizer(input);
        lineTokenizer.setDelimiterMatcher(StrMatcher.charSetMatcher("\r\n"));


        for (String line : lineTokenizer.getTokenArray()) {
          String strippedLine = StringUtils.stripEnd(line, null);
            if (!currentSql.isEmpty()) {
                currentSql.append(lineEnding.get());
            }

View Full Code Here

        }
      
        if (response.getStatusCode() == HttpStatus.SC_OK)
        {
          // logger.warn(method.getResponseCharSet());
          StrTokenizer tokenize = new StrTokenizer(response.getResponseBodyAsString());
          tokenize.setDelimiterString("\n");
          String [] containers = tokenize.getTokenArray();          
          ArrayList <FilesContainer> containerList = new ArrayList<FilesContainer>();
          for(String container : containers) { 
            containerList.add(new FilesContainer(container, this));
          }
          return containerList;

View Full Code Here

            }
          }


          if (response.getStatusCode() == HttpStatus.SC_OK)
          {
            StrTokenizer tokenize = new StrTokenizer(response.getResponseBodyAsString());
            tokenize.setDelimiterString("\n");
            String [] containers = tokenize.getTokenArray();
            List<String> returnValue = new ArrayList<String>();
            for (String containerName: containers)
            {
              returnValue.add(containerName);
            }

View Full Code Here

    String value = params.get(key);
    if (value != null) {
      if (decode) {
        value = decode(value);
      }
      return new StrTokenizer(value, delimiter).setIgnoreEmptyTokens(false).getTokenArray();
    }
    return EMPTY;
  }

View Full Code Here


    World world = plugin.getSenderWorld(sender);
    AutoRefMatch match = plugin.getMatch(world);


    // reparse the args properly using the string tokenizer from org.apache.commons
    args = new StrTokenizer(StringUtils.join(args, ' '), StrMatcher.splitMatcher(),
      StrMatcher.quoteMatcher()).setTrimmerMatcher(StrMatcher.trimMatcher()).getTokenArray();


    try
    {
      HandlerNode node = cmap.get(command.getName());

View Full Code Here

        this.region = region;
    }


    public List<Container> handleResponse(final HttpResponse response) throws ClientProtocolException, IOException {
        if(response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
            final StrTokenizer tokenize = new StrTokenizer(EntityUtils.toString(response.getEntity()));
            tokenize.setDelimiterString("\n");
            final String[] containers = tokenize.getTokenArray();
            ArrayList<Container> list = new ArrayList<Container>();
            for(String container : containers) {
                list.add(new Container(region, container));
            }
            return list;

View Full Code Here

0 1 2

TOP

Related Classes of org.apache.commons.lang.text.StrTokenizer

ch.iterate.openstack.swift.handler.ContainerResponseHandler

com.dbdeploy.database.QueryStatementSplitter

com.odiago.flumebase.io.CharBufferUtils

com.rackspacecloud.client.cloudfiles.FilesClient

de.sub.goobi.forms.ModuleServerForm

de.sub.goobi.helper.GoobiScript

org.apache.felix.sigil.eclipse.preferences.PrefsUtils

org.apache.sqoop.mapreduce.db.DBConfiguration

org.criticalfailure.torchlight.core.calculators.object.BaseObjectPropertyCalculator

org.criticalfailure.torchlight.core.calculators.object.MapObjectPropertyCalculator

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.