Examples of com.ngdata.hbaseindexer.indexer.DirectSolrClassicInputDocumentWriter

com.ngdata.hbaseindexer.indexer.DirectSolrClassicInputDocumentWriter
Writes updates (new documents and deletes) directly to a set of SolrServer (one for each shard).
There are two main pieces of functionality that this class provides, both related to error handling in Solr:
Selective swallowing of errors
If a write to Solr throws an exception signifying that the underlying problem is related to Solr, then the exception will be thrown up the stack. The intention of this behaviour is to allow the write to be re-tried indefinitely until the Solr issue is resolved.
If a write to Solr throws an exception signifying that the underlying problem lies with the document being written, then the exception will be logged, but otherwise ignored. The intention of this behaviour is to stop a single bad document from holding up the whole indexing process for other documents.
Individual retry of documents
If a single document in a batch causes an exception to be thrown that is related to the document itself, then each update will be retried individually.

            if (solrMode.equals("cloud")) {
                DirectSolrInputDocumentWriter writer = createCloudSolrWriter(context, indexConnectionParams);
                solrDocWriter = wrapInBufferedWriter(context, writer);
                return Indexer.createIndexer(indexName, indexerConf, tableName, mapper, null, null, solrDocWriter);
            } else if (solrMode.equals("classic")) {
                DirectSolrClassicInputDocumentWriter classicSolrWriter = createClassicSolrWriter(context, indexConnectionParams);
                Sharder sharder = createSharder(indexConnectionParams, classicSolrWriter.getNumServers());
                solrDocWriter = wrapInBufferedWriter(context, classicSolrWriter);
                return Indexer.createIndexer(indexName, indexerConf, tableName, mapper, null, sharder, solrDocWriter);
            } else {
                throw new RuntimeException("Only 'cloud' and 'classic' are valid values for solr.mode, but got " + solrMode);
            }

View Full Code Here

        connectionManager.setMaxTotal(getSolrMaxConnectionsTotal(indexConnectionParams));


        HttpClient httpClient = new DefaultHttpClient(connectionManager);
        List<SolrServer> solrServers = createHttpSolrServers(indexConnectionParams, httpClient);


        return new DirectSolrClassicInputDocumentWriter(
                context.getConfiguration().get(INDEX_NAME_CONF_KEY), solrServers);
    }

View Full Code Here

                    connectionManager.setDefaultMaxPerRoute(getSolrMaxConnectionsPerRoute(connectionParams));
                    connectionManager.setMaxTotal(getSolrMaxConnectionsTotal(connectionParams));


                    httpClient = new DefaultHttpClient(connectionManager);
                    List<SolrServer> solrServers = createHttpSolrServers(connectionParams, httpClient);
                    solrWriter = new DirectSolrClassicInputDocumentWriter(indexerDef.getName(), solrServers);
                    sharder = createSharder(connectionParams, solrServers.size());
                } else {
                    throw new RuntimeException("Only 'cloud' and 'classic' are valid values for solr.mode, but got " + solrMode);
                }
            } else {

View Full Code Here

TOP

Related Classes of com.ngdata.hbaseindexer.indexer.DirectSolrClassicInputDocumentWriter

com.ngdata.hbaseindexer.mr.HBaseIndexerMapper

com.ngdata.hbaseindexer.supervisor.IndexerSupervisor

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of com.ngdata.hbaseindexer.indexer.DirectSolrClassicInputDocumentWriter

Selective swallowing of errors

Individual retry of documents

Related Classes of com.ngdata.hbaseindexer.indexer.DirectSolrClassicInputDocumentWriter