Examples of cue.lang.NGramIterator

cue.lang.NGramIterator

Construct with a {@link String}, some integer n, and a {@link Locale}; retrieve a sequence of {@link String}s, each of which has n words that appear contiguously within a sentence. "Words" are as defined by the {@link WordIterator}.

If you don't provide a {@link Locale}, you get the default {@link Locale} for your system, which may or may not be whatyou want. The {@link Locale} is used by a {@link SentenceIterator}to find sentence breaks.

Example:

final String lyric = "This happened once before. I came to your door. No reply."; for (final String s : new NGramIterator(3, lyric)) { System.out.println(s); } for (final String s : new NGramIterator(2, lyric)) { System.out.println(s); } This happened once happened once before I came to came to your to your door This happened happened once once before I came came to to your your door No reply

@author Jonathan Feinberg

     */
    private Counter<String> createWordList(File fileIn) throws IOException {
        String contents = processInput(fileIn).toLowerCase();
        if (!lemmatize) {
            Counter<String> words = new Counter<String>();
            for (final String word : new NGramIterator(1, contents, Locale.ENGLISH, StopWords.English)) {
                words.note(word);
            }
            return words;
        } else {
            StanfordLemmatizer lemmatizer = new StanfordLemmatizer();

Examples of cue.lang.NGramIterator

Related Classes of cue.lang.NGramIterator