Examples of com.scaleunlimited.helpful.operations.MboxSplitterFunction

Package com.scaleunlimited.helpful.operations

Examples of com.scaleunlimited.helpful.operations.MboxSplitterFunction

com.scaleunlimited.helpful.operations.MboxSplitterFunction
Cascading function to split a single mbox file into N entries, one per each email. This then lets us parse them as individual emails, which works better with the current Tika API, as we can then use the metadata returned from the Parser.parse() call to get subject, etc.

    when(_funcCall.getOutputCollector()).thenReturn(_collector);
  }
  
  @Test
  public void testSplitterWithNonMbox() {
    MboxSplitterFunction splitter = new MboxSplitterFunction();
    
    FetchedDatum datum = new FetchedDatum("baseUrl", "redirectedUrl", 0, new HttpHeaders(), new ContentBytes(), "text/ascii", 0);
    TupleEntry value = new TupleEntry(datum.getTupleEntry());
    
    when(_funcCall.getArguments()).thenReturn(value);
    splitter.operate(_process, _funcCall);
    
    verify(_collector).add(value);
  }

View Full Code Here

    verify(_collector).add(value);
  }
  
  @Test
  public void testSplitterTwoEmails() throws UnsupportedEncodingException {
    MboxSplitterFunction splitter = new MboxSplitterFunction();


    final String mboxString = "From 1\r\rContent 1\r\rFrom 2\r\rContent 2";
    byte[] mboxContent = mboxString.getBytes("us-ascii");
    FetchedDatum datum = new FetchedDatum("baseUrl", "redirectedUrl", 0, new HttpHeaders(), new ContentBytes(mboxContent), "application/mbox", 0);
    TupleEntry value = new TupleEntry(datum.getTupleEntry());
    
    when(_funcCall.getArguments()).thenReturn(value);
    splitter.operate(_process, _funcCall);


    verify(_collector, times(2)).add(any(TupleEntry.class));
  }

View Full Code Here

    @SuppressWarnings("serial")
  private static class SplitEmails extends SubAssembly {


    public SplitEmails(FetchPipe fetchPipe) {
            Pipe splitPipe = new Pipe(SPLITTER_PIPE_NAME, fetchPipe.getContentTailPipe());
            splitPipe = new Each(splitPipe, new MboxSplitterFunction());
            // TODO KKr - code currently relies on splitPipe being first tail pipe.
            setTails(splitPipe, fetchPipe.getStatusTailPipe());
      }

View Full Code Here

TOP

Related Classes of com.scaleunlimited.helpful.operations.MboxSplitterFunction

bixo.datum.ContentBytes

bixo.datum.FetchedDatum

cascading.tuple.TupleEntry

com.scaleunlimited.helpful.operations.MboxSplitterFunctionTest

com.scaleunlimited.helpful.tools.AnalyzeEmail$SplitEmails

java.io.InputStream

java.io.InputStreamReader

java.io.BufferedReader

java.io.ByteArrayInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.