Examples of Shard

com.amazonaws.services.kinesis.model.Shard

A uniquely identified group of data records in an Amazon Kinesis stream.
com.senseidb.indexing.hadoop.keyvalueformat.Shard
This class represents the metadata of a shard. Version is the version number of the entire index. Directory is the directory where this shard resides in. Generation is the Lucene index's generation. Version and generation are reserved for future use. Note: Currently the version number of the entire index is not used and defaults to -1.
com.volantis.mcs.dissection.impl.Shard
This class contains all the shard related information needed by the dissector.
There is one instance of this class for each shard in each dissectable area.
When the shard contents are being selected this object is in the pending state. Once this has been completed it is moved to the ready state.
net.sf.katta.protocol.metadata.IndexMetaData.Shard
org.apache.hadoop.contrib.index.mapred.Shard
This class represents the metadata of a shard. Version is the version number of the entire index. Directory is the directory where this shard resides in. Generation is the Lucene index's generation. Version and generation are reserved for future use. Note: Currently the version number of the entire index is not used and defaults to -1.
org.broadinstitute.gatk.engine.datasources.reads.Shard
@author aaron @version 1.0 @date Apr 10, 2009
Interface Shard
The base abstract class for shards.
org.elasticsearch.hadoop.serialization.dto.Shard
org.elasticsearch.river.mongodb.MongoConfig.Shard
org.makersoft.shards.Shard
每个物理分区为一个Shard

Examples of org.broadinstitute.gatk.engine.datasources.reads.Shard

        GATKSAMRecord read2 = buildSAMRecord("read2","chr1", 1, 5);
        GATKSAMRecord read3 = buildSAMRecord("read3","chr1", 6, 10);
        GATKSAMRecord read4 = buildSAMRecord("read4","chr1", 6, 10);
        SAMRecordIterator iterator = new SAMRecordIterator(read1, read2, read3, read4);


        Shard shard = new MockLocusShard(genomeLocParser,Collections.singletonList(genomeLocParser.createGenomeLoc("chr1", 1, 10)));
        WindowMaker windowMaker = new WindowMaker(shard,genomeLocParser,iterator,shard.getGenomeLocs());
        WindowMaker.WindowMakerIterator window = windowMaker.next();
        LocusShardDataProvider dataProvider = new LocusShardDataProvider(shard, window.getSourceInfo(), genomeLocParser, window.getLocus(), window, null, null);
        LocusView view = createView(dataProvider);


        List<GATKSAMRecord> expectedReads = new ArrayList<GATKSAMRecord>();
        Collections.addAll(expectedReads, read1, read2, read3, read4);
        testReadsInContext(view, shard.getGenomeLocs(), expectedReads);
    }

View Full Code Here

Examples of org.broadinstitute.gatk.engine.datasources.reads.Shard

        GATKSAMRecord read2 = buildSAMRecord("read2","chr1", 3, 7);
        GATKSAMRecord read3 = buildSAMRecord("read3","chr1", 4, 8);
        GATKSAMRecord read4 = buildSAMRecord("read4","chr1", 5, 9);
        SAMRecordIterator iterator = new SAMRecordIterator(read1, read2, read3, read4);


        Shard shard = new MockLocusShard(genomeLocParser,Collections.singletonList(genomeLocParser.createGenomeLoc("chr1", 1, 10)));
        WindowMaker windowMaker = new WindowMaker(shard,genomeLocParser,iterator,shard.getGenomeLocs());
        WindowMaker.WindowMakerIterator window = windowMaker.next();
        LocusShardDataProvider dataProvider = new LocusShardDataProvider(shard, window.getSourceInfo(), genomeLocParser, window.getLocus(), window, null, null);
        LocusView view = createView(dataProvider);


        List<GATKSAMRecord> expectedReads = new ArrayList<GATKSAMRecord>();
        Collections.addAll(expectedReads, read1, read2, read3, read4);
        testReadsInContext(view, shard.getGenomeLocs(), expectedReads);
    }

View Full Code Here

Examples of org.broadinstitute.gatk.engine.datasources.reads.Shard

        GATKSAMRecord read4 = buildSAMRecord("read4","chr1", 4, 8);
        GATKSAMRecord read5 = buildSAMRecord("read5","chr1", 5, 9);
        GATKSAMRecord read6 = buildSAMRecord("read6","chr1", 6, 10);
        SAMRecordIterator iterator = new SAMRecordIterator(read1, read2, read3, read4, read5, read6);


        Shard shard = new MockLocusShard(genomeLocParser,Collections.singletonList(genomeLocParser.createGenomeLoc("chr1", 1, 10)));
        WindowMaker windowMaker = new WindowMaker(shard,genomeLocParser,iterator,shard.getGenomeLocs());
        WindowMaker.WindowMakerIterator window = windowMaker.next();
        LocusShardDataProvider dataProvider = new LocusShardDataProvider(shard, window.getSourceInfo(), genomeLocParser, window.getLocus(), window, null, null);
        LocusView view = createView(dataProvider);


        List<GATKSAMRecord> expectedReads = new ArrayList<GATKSAMRecord>();
        Collections.addAll(expectedReads, read1, read2, read3, read4, read5, read6);
        testReadsInContext(view, shard.getGenomeLocs(), expectedReads);
    }

View Full Code Here

Examples of org.broadinstitute.gatk.engine.datasources.reads.Shard

        GATKSAMRecord read11 = buildSAMRecord("read11","chr1", 11, 15);
        GATKSAMRecord read12 = buildSAMRecord("read12","chr1", 12, 16);
        SAMRecordIterator iterator = new SAMRecordIterator(read01, read02, read03, read04, read05, read06,
                                                           read07, read08, read09, read10, read11, read12);


        Shard shard = new MockLocusShard(genomeLocParser,Collections.singletonList(genomeLocParser.createGenomeLoc("chr1", 6, 15)));
        WindowMaker windowMaker = new WindowMaker(shard,genomeLocParser,iterator,shard.getGenomeLocs());
        WindowMaker.WindowMakerIterator window = windowMaker.next();
        LocusShardDataProvider dataProvider = new LocusShardDataProvider(shard, window.getSourceInfo(), genomeLocParser, window.getLocus(), window, null, null);
        LocusView view = createView(dataProvider);


        List<GATKSAMRecord> expectedReads = new ArrayList<GATKSAMRecord>();
        Collections.addAll(expectedReads, read01, read02, read03, read04, read05, read06,
                           read07, read08, read09, read10, read11, read12);
        testReadsInContext(view, shard.getGenomeLocs(), expectedReads);
    }

View Full Code Here

Examples of org.broadinstitute.gatk.engine.datasources.reads.Shard

    /**
     * Make sure binding to an empty list produces an empty tracker.
     */
    @Test
    public void testNoBindings() {
        Shard shard = new MockLocusShard(genomeLocParser,Collections.singletonList(genomeLocParser.createGenomeLoc("chrM",1,30)));
        LocusShardDataProvider provider = new LocusShardDataProvider(shard, null, genomeLocParser, shard.getGenomeLocs().get(0), null, seq, Collections.<ReferenceOrderedDataSource>emptyList());
        ReferenceOrderedView view = new ManagingReferenceOrderedView( provider );


        RefMetaDataTracker tracker = view.getReferenceOrderedDataAtLocus(genomeLocParser.createGenomeLoc("chrM",10));
        Assert.assertEquals(tracker.getValues(Feature.class).size(), 0, "The tracker should not have produced any data");
    }

View Full Code Here

Examples of org.broadinstitute.gatk.engine.datasources.reads.Shard

    public void testSingleBinding() {
        String fileName = privateTestDir + "TabularDataTest.dat";
        RMDTriplet triplet = new RMDTriplet("tableTest","Table",fileName,RMDStorageType.FILE,new Tags());
        ReferenceOrderedDataSource dataSource = new ReferenceOrderedDataSource(triplet,builder,seq.getSequenceDictionary(),genomeLocParser,false);


        Shard shard = new MockLocusShard(genomeLocParser,Collections.singletonList(genomeLocParser.createGenomeLoc("chrM",1,30)));


        LocusShardDataProvider provider = new LocusShardDataProvider(shard, null, genomeLocParser, shard.getGenomeLocs().get(0), null, seq, Collections.singletonList(dataSource));
        ReferenceOrderedView view = new ManagingReferenceOrderedView( provider );


        RefMetaDataTracker tracker = view.getReferenceOrderedDataAtLocus(genomeLocParser.createGenomeLoc("chrM",20));
        TableFeature datum = tracker.getFirstValue(new RodBinding<TableFeature>(TableFeature.class, "tableTest"));

View Full Code Here

Examples of org.broadinstitute.gatk.engine.datasources.reads.Shard

        ReferenceOrderedDataSource dataSource1 = new ReferenceOrderedDataSource(testTriplet1,builder,seq.getSequenceDictionary(),genomeLocParser,false);


        RMDTriplet testTriplet2 = new RMDTriplet("tableTest2","Table",file.getAbsolutePath(),RMDStorageType.FILE,new Tags());
        ReferenceOrderedDataSource dataSource2 = new ReferenceOrderedDataSource(testTriplet2,builder,seq.getSequenceDictionary(),genomeLocParser,false);


        Shard shard = new MockLocusShard(genomeLocParser,Collections.singletonList(genomeLocParser.createGenomeLoc("chrM",1,30)));


        LocusShardDataProvider provider = new LocusShardDataProvider(shard, null, genomeLocParser, shard.getGenomeLocs().get(0), null, seq, Arrays.asList(dataSource1,dataSource2));
        ReferenceOrderedView view = new ManagingReferenceOrderedView( provider );


        RefMetaDataTracker tracker = view.getReferenceOrderedDataAtLocus(genomeLocParser.createGenomeLoc("chrM",20));
        TableFeature datum1 = tracker.getFirstValue(new RodBinding<TableFeature>(TableFeature.class, "tableTest1"));

View Full Code Here

Examples of org.elasticsearch.hadoop.serialization.dto.Shard

        info.put("index", "index");
        info.put("relocating_node", "none");
        info.put("node", "1");
        info.put("primary", "true");


        Shard sh1 = new Shard(info);
        Node node1 = new Node("1", info);


        info.put("name", "2");
        info.put("shard", 2);


        Shard sh2 = new Shard(info);
        Node node2 = new Node("2", info);


        info.put("name", "3");
        info.put("shard", 3);


        Shard sh3 = new Shard(info);
        Node node3 = new Node("3", info);


        info.put("name", "4");
        info.put("shard", 4);


        Shard sh4 = new Shard(info);
        Node node4 = new Node("4", info);


        info.put("name", "5");
        info.put("shard", 5);


        Shard sh5 = new Shard(info);
        Node node5 = new Node("5", info);


        info.put("name", "6");
        info.put("shard", 6);


        Shard sh6 = new Shard(info);
        Node node6 = new Node("6", info);


        pd1 = new PartitionDefinition(sh1, node1, null, null);
        pd2 = new PartitionDefinition(sh2, node2, null, null);
        pd3 = new PartitionDefinition(sh3, node3, null, null);

View Full Code Here

Examples of org.elasticsearch.hadoop.serialization.dto.Shard

        // if there's no task info, just pick a random bucket
        if (currentInstance <= 0) {
            currentInstance = new Random().nextInt(targetShards.size()) + 1;
        }
        int bucket = currentInstance % targetShards.size();
        Shard chosenShard = orderedShards.get(bucket);
        Node targetNode = targetShards.get(chosenShard);


        // pin settings
        SettingsUtils.pinNode(settings, targetNode.getIpAddress(), targetNode.getHttpPort());
        String node = SettingsUtils.getPinnedNode(settings);
        repository = new RestRepository(settings);


        if (log.isDebugEnabled()) {
            log.debug(String.format("Partition writer instance [%s] assigned to primary shard [%s] at address [%s]",
                    currentInstance, chosenShard.getName(), node));
        }


        return repository;
    }

View Full Code Here

Examples of org.elasticsearch.hadoop.serialization.dto.Shard

        List<List<Map<String, Object>>> info = client.targetShards(resourceR.index());


        for (List<Map<String, Object>> shardGroup : info) {
            // find the first started shard in each group (round-robin)
            for (Map<String, Object> shardData : shardGroup) {
                Shard shard = new Shard(shardData);
                if (shard.getState().isStarted()) {
                    Node node = nodes.get(shard.getNode());
                    Assert.notNull(node, String.format("Cannot find node with id [%s] (is HTTP enabled?) from shard [%s] in nodes [%s]; layout [%s]", shard.getNode(), shard, shardGroup, info));
                    shards.put(shard, node);
                    break;
                }
            }
        }

View Full Code Here

0 1 2 3 4

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.