Examples of skipHttpHeader()

Examples of org.archive.io.arc.ARCRecord.skipHttpHeader()

380
381
382
383
384
385
386
387
388
389
390
    reporter.setStatus(getStatus(url, oldUrl, recordLengthAsStr, noSpacesMimetype));

    // This is a nutch 'more' field.
    metaData.set("contentLength", recordLengthAsStr);

    rec.skipHttpHeader();
    reporter.setStatusIfElapse("read headers on " + url);

    // TODO: Skip if unindexable type.
    int total = 0;
   
View Full Code Here

Examples of org.archive.io.arc.ARCRecord.skipHttpHeader()

181
182
183
184
185
186
187
188
189
190
191
    final ARCRecordMetaData arcData = rec.getMetaData();
    // Look at ARCRecord meta data line mimetype. It can be empty. If so,
    // two more chances at figuring it either by looking at HTTP headers or
    // by looking at first couple of bytes of the file. See below.
    String mimetype = getMimetype(arcData.getMimetype(), this.mimeTypes, url);
    rec.skipHttpHeader();
    reporter.setStatusIfElapse("read headers on " + url);

    // Read in first block. If mimetype still null, look for MAGIC.
    int len = rec.read(this.buffer, 0, this.buffer.length);
    // check mimetype
View Full Code Here

Examples of org.archive.io.arc.ARCRecord.skipHttpHeader()

366
367
368
369
370
371
372
373
374
375
376
   * @throws IOException
   */
  private void replayRecord(ARCReader arcReader) throws IOException {
      arcReader.setStrict(true);
      ARCRecord arcRecord = (ARCRecord) arcReader.get(this.offset);
      arcRecord.skipHttpHeader();
      if (arcRecord.hasErrors()) {
        logger.warning("record has errors: " + arcRecord.getErrors());
      }
      byte[] buffer = new byte[BUFFER_SIZE];
      if (arcRecord.available() > 0) {
View Full Code Here
TOP
Copyright © 2018 www.massapi.com. All rights reserved.
All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.