Examples of org.lilyproject.repository.bulk.jython.JythonLineMapper

org.lilyproject.repository.bulk.jython.JythonLineMapper

Maps single input records (lines of text) into Lily {@link Record} objects using a user-supplied callable.

The signature of the callable is as follows:

 __call__(inputLine, mappingContext)

The inputLine argument is a single string, with line-ending characters removed. The mappingContext argument is a {@link LineMappingContext} object which can be used to create records and ids, as well as writingcreated records.

The callable can write zero, one, or many {@code Record} objects to the context.

The callable must be indentifiable as a single symbol. In the case of a global function, this is just the name of a function. If the callable is a method in a class, an instance of the class must be created and put in the global scope.

An example of a global function would be something like the following:

 def myMapFunction(inputLine, mappingContext): rec = mappingContext.newRecord() rec.setRecordType(ctx.qn("{org.lilyproject}Person")) firstName, lastName = inputLine.split(",") rec.setField("{org.lilyproject}FirstName", firstName) rec.setField("{org.lilyproject}LastName", lastName) mappingContext.writeRecord(rec)

The mapping function in the above example would be provided as "myMapFunction".

Using a method on an instance of the class can be done as follows:

 class MyMapperClass(object): def myMapMethod(self, inputLine, mappingContext): rec = mappingContext.newRecord() rec.setRecordType(ctx.qn("{org.lilyproject}Person")) firstName, lastName = inputLine.split(",") rec.setField("{org.lilyproject}FirstName", firstName) rec.setField("{org.lilyproject}LastName", lastName) mappingContext.writeRecord(rec) mapMethod = MyMapperClass().myMapMethod

The mapping function in this example would be provided as "mapMethod".

        }
        long start = System.currentTimeMillis();
        int numLines = 0;


        try {
            LineMapper lineMapper = new JythonLineMapper(Files.toString(new File(pythonMapperPath), Charsets.UTF_8),
                    pythonSymbol);
            LineMappingContext mappingContext = new LineMappingContext(bulkIngester, recordWriter);
            String line;
            while ((line = bufferedReader.readLine()) != null) {
                lineMapper.mapLine(line, mappingContext);
                numLines++;
            }
        } catch (PyException pe) {
            pe.printStackTrace();    // Print the Jython-native stack trace
            log.error("Exception encountered in Python code", pe);

    private MapReduceRecordWriter recordWriter;


    @Override
    protected void setup(Context context) throws IOException, InterruptedException {
        Configuration conf = context.getConfiguration();
        lineMapper = new JythonLineMapper(conf.get(MAPPER_CODE), conf.get(MAPPER_SYMBOL_NAME));
        bulkIngester = BulkIngester.newBulkIngester(conf.get(LILY_ZK_STRING), 30000, conf.get(REPOSITORY_NAME),
                conf.get(TABLE_NAME), Boolean.valueOf(conf.get(BULK_MODE)));
        recordWriter = new MapReduceRecordWriter(bulkIngester);
        lineMappingContext = new LineMappingContext(bulkIngester, recordWriter);
    }

Examples of org.lilyproject.repository.bulk.jython.JythonLineMapper

Related Classes of org.lilyproject.repository.bulk.jython.JythonLineMapper