Examples of com.ngdata.hbaseindexer.indexer.DirectSolrInputDocumentWriter

com.ngdata.hbaseindexer.indexer.DirectSolrInputDocumentWriter
Writes updates (new documents and deletes) directly to a SolrServer.
There are two main pieces of functionality that this class provides, both related to error handling in Solr:
Selective swallowing of errors
If a write to Solr throws an exception signifying that the underlying problem is related to Solr, then the exception will be thrown up the stack. The intention of this behaviour is to allow the write to be re-tried indefinitely until the Solr issue is resolved.
If a write to Solr throws an exception signifying that the underlying problem lies with the document being written, then the exception will be logged, but otherwise ignored. The intention of this behaviour is to stop a single bad document from holding up the whole indexing process for other documents.
Individual retry of documents
If a single document in a batch causes an exception to be thrown that is related to the document itself, then each update will be retried individually.

            throws IOException, SharderException {
        Configuration conf = context.getConfiguration();
        if (conf.getBoolean(INDEX_DIRECT_WRITE_CONF_KEY, false)) {
            String solrMode = getSolrMode(indexConnectionParams);
            if (solrMode.equals("cloud")) {
                DirectSolrInputDocumentWriter writer = createCloudSolrWriter(context, indexConnectionParams);
                solrDocWriter = wrapInBufferedWriter(context, writer);
                return Indexer.createIndexer(indexName, indexerConf, tableName, mapper, null, null, solrDocWriter);
            } else if (solrMode.equals("classic")) {
                DirectSolrClassicInputDocumentWriter classicSolrWriter = createClassicSolrWriter(context, indexConnectionParams);
                Sharder sharder = createSharder(indexConnectionParams, classicSolrWriter.getNumServers());

View Full Code Here

            throw new IllegalStateException("No collection name defined");
        }
        CloudSolrServer solrServer = new CloudSolrServer(indexZkHost);
        solrServer.setDefaultCollection(collectionName);


        return new DirectSolrInputDocumentWriter(context.getConfiguration().get(INDEX_NAME_CONF_KEY), solrServer);
    }

View Full Code Here


            if (indexerDef.getConnectionType() == null || indexerDef.getConnectionType().equals("solr")) {
                Map<String, String> connectionParams = indexerDef.getConnectionParams();
                String solrMode = SolrConnectionParamUtil.getSolrMode(connectionParams);
                if (solrMode.equals("cloud")) {
                    solrWriter = new DirectSolrInputDocumentWriter(indexerDef.getName(), createCloudSolrServer(connectionParams));
                } else if (solrMode.equals("classic")) {
                    connectionManager = new PoolingClientConnectionManager();
                    connectionManager.setDefaultMaxPerRoute(getSolrMaxConnectionsPerRoute(connectionParams));
                    connectionManager.setMaxTotal(getSolrMaxConnectionsTotal(connectionParams));

View Full Code Here

TOP

Related Classes of com.ngdata.hbaseindexer.indexer.DirectSolrInputDocumentWriter

com.ngdata.hbaseindexer.mr.HBaseIndexerMapper

com.ngdata.hbaseindexer.supervisor.IndexerSupervisor

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of com.ngdata.hbaseindexer.indexer.DirectSolrInputDocumentWriter

Selective swallowing of errors

Individual retry of documents

Related Classes of com.ngdata.hbaseindexer.indexer.DirectSolrInputDocumentWriter