Package net.sf.okapi.steps.tokenization.tokens

Examples of net.sf.okapi.steps.tokenization.tokens.Tokens


                        "can't understand '{}' as a BCP-47 locale; defaulting to English",
                        bcp47Locale);
                locale = LocaleId.ENGLISH;
            }

            Tokens tokens = StringTokenizer.tokenizeString(s, locale, "WORD");
            return tokens.size();
        } catch (Exception e) {
            Object[] args = new Object[] { s, bcp47Locale, e };
            log.error("unable to count words in string '{}' for locale '{}'",
                    args);
            return 0;
View Full Code Here

TOP

Related Classes of net.sf.okapi.steps.tokenization.tokens.Tokens

Copyright © 2018 www.massapicom. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.