Examples of org.carrot2.examples.clustering.ClusteringNonEnglishContent

org.carrot2.workbench.core.ui.actions.IControlProvider
[[[start:clustering-non-english-content-intro]]]
This example shows how to cluster non-English content. By default Carrot2 assumes that the documents provided for clustering are written in English. When clustering content written in some different language, it is important to indicate the language to Carrot2, so that it can use the lexical resources (stop words, tokenizer, stemmer) appropriate for that language.

There are two ways to indicate the desired clustering language to Carrot2:
1. By setting the language of each document in their {@link org.carrot2.core.Document#LANGUAGE} field. The language does not necessarilyhave to be the same for all documents on the input, Carrot2 can handle multiple languages in one document set as well. Please see the {@link org.carrot2.text.clustering.MultilingualClustering#languageAggregationStrategy}attribute for more details.
2. By setting the fallback language. For documents with undefined {@link org.carrot2.core.Document#LANGUAGE} field, Carrot2 will assume the some fallbacklanguage, which is English by default. You can change the fallback language by setting the {@link org.carrot2.text.clustering.MultilingualClustering#defaultLanguage}attribute.
Additionally, a number of document sources automatically set the {@link org.carrot2.core.Document#LANGUAGE} of documents they produce based on theirspecific language-related attributes. Currently, three documents support this scenario:
1. {@link org.carrot2.source.microsoft.Bing3WebDocumentSource} through the{@link org.carrot2.source.microsoft.Bing3WebDocumentSource#market} attribute,
2. {@link org.carrot2.source.etools.EToolsDocumentSource} through the{@link org.carrot2.source.etools.EToolsDocumentSource#language} attribute.
For the document sources that do not set the documents' language automatically, the easiest way to set the clustering language is through the {@link org.carrot2.text.clustering.MultilingualClustering#defaultLanguage} attribute.
[[[end:clustering-non-english-content-intro]]]

                  + carrot2ResourcesDir);
              final InputStream resourceStream = resourceLoader
                  .openResource(carrot2ResourcesDir + "/" + resource);
              
              log.info(resource + " loaded from " + carrot2ResourcesDir);
              final IResource foundResource = new IResource() {
                public InputStream open() throws IOException {
                  return resourceStream;
                }
              };
              return new IResource[] { foundResource };

View Full Code Here

    initAttributes.put("solrIndexSchema", core.getSchema());


    // Customize Carrot2's resource lookup to first look for resources
    // using Solr's resource loader. If that fails, try loading from the classpath.
    DefaultLexicalDataFactoryDescriptor.attributeBuilder(initAttributes)
        .resourceLookup(new ResourceLookup(new IResourceLocator() {
          public IResource[] getAll(final String resource) {
            final SolrResourceLoader resourceLoader = core.getResourceLoader();
            final String carrot2ResourcesDir = resourceLoader.getConfigDir()
                + initParams.get(CarrotParams.LEXICAL_RESOURCES_DIR, CARROT_RESOURCES_PREFIX);
            try {

View Full Code Here

    initAttributes.put("solrIndexSchema", core.getSchema());


    // Customize Carrot2's resource lookup to first look for resources
    // using Solr's resource loader. If that fails, try loading from the classpath.
    DefaultLexicalDataFactoryDescriptor.attributeBuilder(initAttributes)
        .resourceLookup(new ResourceLookup(new IResourceLocator() {
          public IResource[] getAll(final String resource) {
            final SolrResourceLoader resourceLoader = core.getResourceLoader();
            final String carrot2ResourcesDir = resourceLoader.getConfigDir()
                + initParams.get(CarrotParams.LEXICAL_RESOURCES_DIR, CARROT_RESOURCES_PREFIX);
            try {

View Full Code Here

                    .defaultLanguage(LanguageCode.ENGLISH);




                    File resourcesDir = new File(environment.configFile(), "carrot2/resources");


                    ResourceLookup resourceLookup = new ResourceLookup(new DirLocator(resourcesDir));


                    DefaultLexicalDataFactoryDescriptor.attributeBuilder(attributes)
                    .mergeResources(true);
                    LexicalDataLoaderDescriptor.attributeBuilder(attributes)
                    .resourceLookup(resourceLookup);

View Full Code Here




    @Override
    protected AbstractBrowserVisualizationViewPage wrappedCreatePage(IWorkbenchPart part)
    {
        final SearchEditor editor = (SearchEditor) part;
        final CirclesViewPage page = new CirclesViewPage(editor);
        initPage(page);
        page.createControl(getPageBook());
        return page;
    }

View Full Code Here


    private void doGroupSelection(int [] selectedGroups)
    {
        logger.debug("Selection visualization->editor: " + Arrays.toString(selectedGroups));


        SearchEditorSelectionProvider prov = 
          (SearchEditorSelectionProvider) editor.getSite().getSelectionProvider();


        prov.setSelected(selectedGroups, selectionListener);
    }

View Full Code Here


        IPreferenceStore store = Activator.getInstance().getPreferenceStore();
        store.addPropertyChangeListener(listener);


        pageSite.getActionBars().getToolBarManager().add(
            new ExportImageAction(new IControlProvider()
            {
                public Control getControl()
                {
                    return FoamTreeViewPage.this.getControl();
                }

View Full Code Here

    public void init(IPageSite pageSite)
    {
        super.init(pageSite);


        pageSite.getActionBars().getToolBarManager().add(
            new ExportImageAction(new IControlProvider()
            {
                @Override
                public Control getControl()
                {
                    return CirclesViewPage.this.getControl();

View Full Code Here


        IPreferenceStore store = Activator.getInstance().getPreferenceStore();
        store.addPropertyChangeListener(listener);


        pageSite.getActionBars().getToolBarManager().add(
            new ExportImageAction(new IControlProvider()
            {
                public Control getControl()
                {
                    return FoamTreeViewPage.this.getControl();
                }

View Full Code Here

    public void init(IPageSite pageSite)
    {
        super.init(pageSite);


        pageSite.getActionBars().getToolBarManager().add(
            new ExportImageAction(new IControlProvider()
            {
                @Override
                public Control getControl()
                {
                    return CirclesViewPage.this.getControl();

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.carrot2.examples.clustering.ClusteringNonEnglishContent

com.carrotsearch.hppc.BitSet

com.carrotsearch.hppc.IntArrayList

com.carrotsearch.hppc.IntIntOpenHashMap

com.carrotsearch.hppc.IntStack

com.tamingtext.carrot2.Carrot2ExampleTest

org.apache.http.message.BasicNameValuePair

org.apache.lucene.search.IndexSearcher

org.apache.mahout.math.matrix.DoubleMatrix2D

org.carrot2.clustering.kmeans.BisectingKMeansClusteringAlgorithm

org.carrot2.clustering.kmeans.BisectingKMeansClusteringAlgorithmTest

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.