Examples of com.ibm.icu.text.UnicodeSet

com.ibm.icu.text.UnicodeSet

cu-project.org/userguide/unicodeSet.html"> http://www.icu-project.org/userguide/unicodeSet.html. Actual determination of property data is defined by the underlying Unicode database as implemented by UCharacter.

Patterns specify individual characters, ranges of characters, and Unicode property sets. When elements are concatenated, they specify their union. To complement a set, place a '^' immediately after the opening '['. Property patterns are inverted by modifying their delimiters; "[:^foo]" and "\P{foo}". In any other location, '^' has no special meaning.

Ranges are indicated by placing two a '-' between two characters, as in "a-z". This specifies the range of all characters from the left to the right, in Unicode order. If the left character is greater than or equal to the right character it is a syntax error. If a '-' occurs as the first character after the opening '[' or '[^', or if it occurs as the last character before the closing ']', then it is taken as a literal. Thus "[a\\-b]", "[-ab]", and "[ab-]" all indicate the same set of three characters, 'a', 'b', and '-'.

Sets may be intersected using the '&' operator or the asymmetric set difference may be taken using the '-' operator, for example, "[[:L:]&[\\u0000-\\u0FFF]]" indicates the set of all Unicode letters with values less than 4096. Operators ('&' and '|') have equal precedence and bind left-to-right. Thus "[[:L:]-[a-z]-[\\u0100-\\u01FF]]" is equivalent to "[[[:L:]-[a-z]]-[\\u0100-\\u01FF]]". This only really matters for difference; intersection is commutative.

`[a]`	The set containing 'a'
`[a-z]`	The set containing 'a' through 'z' and all letters in between, in Unicode order
`[^a-z]`	The set containing all characters but 'a' through 'z', that is, U+0000 through 'a'-1 and 'z'+1 through U+10FFFF
`[[pat1][pat2]]`	The union of sets specified by pat1 and pat2
`[[pat1]&[pat2]]`	The intersection of sets specified by pat1 and pat2
`[[pat1]-[pat2]]`	The asymmetric difference of sets specified by pat1 and pat2
`[:Lu:] or \p{Lu}`	The set of characters having the specified Unicode property; in this case, Unicode uppercase letters
`[:^Lu:] or \P{Lu}`	The set of characters not having the given Unicode property

Warning: you cannot add an empty string ("") to a UnicodeSet.

Formal syntax

pattern := ('[' '^'? item* ']') | property

item := char | (char '-' char) | pattern-expr

pattern-expr := pattern | pattern-expr pattern | pattern-expr op pattern

op := '&' | '-'

special := '[' | ']' | '-'

char := any character that is notspecial | ('\\'any character) | ('\u' hex hex hex hex)

hex := any character for which Character.digit(c, 16) returns a non-negative result

property := a Unicode property set pattern

Legend:

a := b a may be replaced by b

a? zero or one instance of a

a* one or more instances of a

a | b either a or b

'a' the literal string between the quotes

To iterate over contents of UnicodeSet, use UnicodeSetIterator class. @author Alan Liu @stable ICU 2.0 @see UnicodeSetIterator

            }
        }
    }
    
    public void TestVariableCharacters() {
        UnicodeSet valid = new UnicodeSet("[G   y   Y   u   Q   q   M   L   w   W   d   D   F   g   E   e   c   a   h   H   K   k   m   s   S   A   z   Z   v   V]");
        for (char c = 0; c < 0xFF; ++c) {
            boolean works = false;
            try {
                VariableField vf = new VariableField(String.valueOf(c), true);
                logln("VariableField " + vf.toString());
                works = true;
            } catch (Exception e) {}
            if (works != valid.contains(c)) {
                if (works) {
                    errln("VariableField can be created with illegal character: " + c);
                } else {
                    errln("VariableField can't be created with legal character: " + c);
                }

View Full Code Here


    public static final String UNUSED = "??";


    public final UnicodeSet getSet(PatternMatcher matcher, UnicodeSet result) {
        if (result == null)
            result = new UnicodeSet();
        if (isType(STRING_OR_MISC_MASK)) {
            for (int i = 0; i <= 0x10FFFF; ++i) {
                String value = getValue(i);
                if (value != null && matcher.matches(value)) {
                    result.add(i);

View Full Code Here

                    System.out.println(propertyName + "=" + propertyValue);
                UnicodeProperty prop = getProperty(propertyName);
                if (prop == null)
                    return false;
                result.clear();
                UnicodeSet x = prop.getSet(propertyValue, result);
                return x.size() != 0;
            }

View Full Code Here

                if (prop == null) {
                    throw new IllegalArgumentException("Invalid Property in: "
                            + s + "\r\nUse " + showSet(getAvailableNames()));
                }
                String value = s.substring(pos + 1);
                UnicodeSet set;
                if (value.startsWith("\u00AB")) { // regex!
                    set = prop.getSet(regexMatcher.set(value.substring(1, value
                            .length() - 1)));
                } else {
                    set = prop.getSet(value);
                }
                if (set.size() == 0) {
                    throw new IllegalArgumentException(
                            "Empty Property-Value in: " + s + "\r\nUse "
                                    + showSet(prop.getAvailableValues()));
                }
                if (DEBUG)
                    System.out.println("\t(" + prefix + ")Returning "
                            + set.toPattern(true));
                return set.toPattern(true).toCharArray(); // really ugly
            }

View Full Code Here

        int flags) 
    {
        if (pw == null) pw = CONSOLE;
        String[] names = { name1, name2 };


        UnicodeSet temp;
        
        if ((flags&1) != 0) {
            temp = new UnicodeSet(set1).removeAll(set2);
            pw.print(lineSeparator);
            pw.print(inOut.format(names));
            pw.print(lineSeparator);
            showSetNames(pw, temp);
        }


        if ((flags&2) != 0) {
            temp = new UnicodeSet(set2).removeAll(set1);
            pw.print(lineSeparator);
            pw.print(outIn.format(names));
            pw.print(lineSeparator);
            showSetNames(pw, temp);
        }


        if ((flags&4) != 0) {
            temp = new UnicodeSet(set2).retainAll(set1);
            pw.print(lineSeparator);
            pw.print(inIn.format(names));
            pw.print(lineSeparator);
            showSetNames(pw, temp);
        }

View Full Code Here

        }


        int numOfEncodings = encodings.size();
        for (int i = 0; i < numOfEncodings; i++) {
            resultsManually[findIndex((String) encodings.get(i))] = true;
            UnicodeSet unicodePointSet = new UnicodeSet();
            Charset testCharset = CharsetICU.forNameICU((String) encodings
                    .get(i));
            ((CharsetICU) testCharset).getUnicodeSet(unicodePointSet,
                    mappingTypes);
            int ch;
            int index = 0;
            while (index < s.length()) {
                ch = UTF16.charAt(s, index);
                if (!excludedEncodings.contains(ch)
                        && !unicodePointSet.contains(ch)) {
                    resultsManually[findIndex((String) encodings.get(i))] = false;
                    break;
                }
                index += UTF16.getCharCount(ch);
            }

View Full Code Here

                171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182,
                183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194,
                195, 196, 197, 198, 199, 200, -1, 1, -1 };


        UnicodeSet[] excludedSets = new UnicodeSet[3];
        excludedSets[0] = new UnicodeSet();
        for (int i = 1; i < 3; i++) {
            excludedSets[i] = new UnicodeSet(i * 30, i * 30 + 500);
        }
        
        for (int excludedSetId = 0; excludedSetId < 3; excludedSetId++) {
            for (int testCaseIdx = 0, prev = 0; testCaseIdx < encodingsTestCases.length; testCaseIdx++) {
                if (encodingsTestCases[testCaseIdx] != -1)

View Full Code Here

        ConversionCase cc = new ConversionCase();
        CharsetProviderICU provider = new CharsetProviderICU();
        CharsetICU charset  ;
       
             
        UnicodeSet mapset = new UnicodeSet();
        UnicodeSet mapnotset = new UnicodeSet();
        UnicodeSet unicodeset = new UnicodeSet();
        String ellipsis = "0x2e";
        cc.charset = ((ICUResourceBundle) testcase.getObject("charset"))
                .getString();
        cc.map = ((ICUResourceBundle) testcase.getObject("map")).getString();
        cc.mapnot = ((ICUResourceBundle) testcase.getObject("mapnot"))
                .getString();
        
     
        int which = ((ICUResourceBundle) testcase.getObject("which")).getInt(); // only checking for ROUNDTRIP_SET
        
        // ----for debugging only
        logln("");
        logln("TestGetUnicodeSet[" + cc.charset + "] ");
        logln("...............................................");
        
        try{
           // if cc.charset starts with '*', obtain it from com/ibm/icu/dev/data/testdata
           charset = (cc.charset != null && cc.charset.length() > 0 && cc.charset.charAt(0) == '*')
                    ? (CharsetICU) provider.charsetForName(cc.charset.substring(1),
                        "com/ibm/icu/dev/data/testdata", this.getClass().getClassLoader())
                    : (CharsetICU) provider.charsetForName(cc.charset);
           
           //checking for converter that are not supported at this point        
           try{
               if(charset.name()=="BOCU-1" ||charset.name()== "SCSU"|| charset.name()=="lmbcs1" || charset.name()== "lmbcs2" ||
                      charset.name()== "lmbcs3" || charset.name()== "lmbcs4" || charset.name()=="lmbcs5" || charset.name()=="lmbcs6" ||
                      charset.name()== "lmbcs8" || charset.name()=="lmbcs11" || charset.name()=="lmbcs16" || charset.name()=="lmbcs17" || 
                      charset.name()=="lmbcs18"|| charset.name()=="lmbcs19"){
                   
                   logln("Converter not supported at this point :" +charset.displayName());
                   return;
               }
                             
               if(which==1){
                   logln("Fallback set not supported at this point for converter : "+charset.displayName());
                  return;
               }
               
           }catch(Exception e){
               return;
           }
           
           mapset.clear();
           mapnotset.clear();
                   
           mapset.applyPattern(cc.map,false);
           mapnotset.applyPattern(cc.mapnot,false);
           
           charset.getUnicodeSet(unicodeset, which);
           UnicodeSet diffset = new UnicodeSet();
           
           //are there items that must be in unicodeset but are not?           
           (diffset = mapset).removeAll(unicodeset);
           if(!diffset.isEmpty()){
               StringBuffer s = new StringBuffer(diffset.toPattern(true));
               if(s.length()>100){
                   s.replace(0, 0x7fffffff, ellipsis);
               }
               errln("error in missing items - conversion/getUnicodeSet test case "+cc.charset + "\n" + s.toString());
           }
           
          //are the items that must not be in unicodeset but are?
           (diffset=mapnotset).retainAll(unicodeset);
           if(!diffset.isEmpty()){
               StringBuffer s = new StringBuffer(diffset.toPattern(true));
               if(s.length()>100){
                   s.replace(0, 0x7fffffff, ellipsis);
               }
               errln("contains unexpected items - conversion/getUnicodeSet test case "+cc.charset + "\n" + s.toString());
           }

View Full Code Here

  public void testOptimizer() throws Exception {
    String rules = "a > b; b > c;"; // convert a's to b's and b's to c's
    Transliterator custom = Transliterator.createFromRules("test", rules, Transliterator.FORWARD);
    assertTrue(custom.getFilter() == null);
    new ICUTransformFilter(new KeywordTokenizer(new StringReader("")), custom);
    assertTrue(custom.getFilter().equals(new UnicodeSet("[ab]")));
  }

View Full Code Here

  public void testOptimizerSurrogate() throws Exception {
    String rules = "\\U00020087 > x;"; // convert CJK UNIFIED IDEOGRAPH-20087 to an x
    Transliterator custom = Transliterator.createFromRules("test", rules, Transliterator.FORWARD);
    assertTrue(custom.getFilter() == null);
    new ICUTransformFilter(new KeywordTokenizer(new StringReader("")), custom);
    assertTrue(custom.getFilter().equals(new UnicodeSet("[\\U00020087]")));
  }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of com.ibm.icu.text.UnicodeSet

com.ibm.icu.charset.CharsetISCII

com.ibm.icu.charset.CharsetSelector

com.ibm.icu.dev.demo.translit.CaseIterator

com.ibm.icu.dev.demo.translit.TransliterationChart

com.ibm.icu.dev.test.charset.TestConversion

com.ibm.icu.dev.test.charset.TestSelection

com.ibm.icu.dev.test.collator.RandomCollator

com.ibm.icu.dev.test.format.DateTimeGeneratorTest

com.ibm.icu.dev.test.lang.UCharacterCaseTest

com.ibm.icu.dev.test.lang.UCharacterTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

`pattern :=`	`('[' '^'? item* ']') \| property`
`item :=`	`char \| (char '-' char) \| pattern-expr`
`pattern-expr :=`	`pattern \| pattern-expr pattern \| pattern-expr op pattern`
`op :=`	`'&' \| '-'`
`special :=`	`'[' \| ']' \| '-'`
`char :=`	any character that is not`special \| ('\\'`any character`) \| ('\u' hex hex hex hex)`
`hex :=`	any character for which `Character.digit(c, 16)` returns a non-negative result
`property :=`	a Unicode property set pattern