Examples of net.bpiwowar.mg4j.extensions.utils.StructuredTextExtractor

Package net.bpiwowar.mg4j.extensions.utils

Examples of net.bpiwowar.mg4j.extensions.utils.StructuredTextExtractor

net.bpiwowar.mg4j.extensions.utils.StructuredTextExtractor
A callback extracting text and titles for structured documents. This implementation also keeps track of the different segments within the text. It also stores the positions of elements and tags.
This callbacks extracts all text in the page, and the title. The resulting text is available through {@link #text}, and the title through {@link #title}. Furthermore, the segments of the resulting text are preserved.
@author Ingo Frommholz <ingo@dcs.gla.ac.uk> @author Benjamin Piwowarski

                    //System.out.println(w.getHTMLContent());


                    // See how the parsed content looks like
                    BulletParser parser = new BulletParser(TRECParsingFactory.INSTANCE);
                    ComposedCallbackBuilder composedBuilder = new ComposedCallbackBuilder();
                    StructuredTextExtractor textExtractor = new StructuredTextExtractor();
                    composedBuilder.add(textExtractor);
                    parser.setCallback(composedBuilder.compose());
                    parser.parse(w.getHTMLContent().toCharArray());
                    System.out.println(textExtractor.getText());
                }
            }
            in.close();
            stream.close();
        }

View Full Code Here

        // The parser is a SGML BulletParser with TREC vocabulary
        this.parser = new BulletParser(TRECParsingFactory.INSTANCE);


        ComposedCallbackBuilder composedBuilder = new ComposedCallbackBuilder();


        composedBuilder.add(this.textExtractor = new StructuredTextExtractor());


        this.textExtractor.ignore(
                TRECParsingFactory.ELEMENT_DOCNO,
                TRECParsingFactory.ELEMENT_FILEID,
                TRECParsingFactory.ELEMENT_FIRST,

View Full Code Here

TOP

Related Classes of net.bpiwowar.mg4j.extensions.utils.StructuredTextExtractor

net.bpiwowar.mg4j.extensions.TagPointer

net.bpiwowar.mg4j.extensions.trec.TRECDocumentFactory

net.bpiwowar.mg4j.extensions.warc.WarcRecord

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.