Examples of com.pinterest.secor.common.LogFilePath

com.pinterest.secor.common.LogFilePath

LogFilePath represents path of a log file. It contains convenience method for building and decomposing paths. Log file path has the following form: prefix/topic/partition1/.../partitionN/generation_kafkaParition_firstMessageOffset where: prefix is top-level directory for log files. It can be a local path or an s3 dir, topic is a kafka topic, partition1, ..., partitionN is the list of partition names extracted from message content. E.g., the partition may describe the message date such as dt=2014-01-01, generation is the consumer version. It allows up to perform rolling upgrades of non-compatible Secor releases, kafkaPartition is the kafka partition of the topic, firstMessageOffset is the offset of the first message in a batch of files committed atomically. @author Pawel Garbacki (pawel@pinterest.com)

    private SecorConfig mConfig;


    @Override
    public void setUp() throws Exception {
        super.setUp();
        mLogFilePath = new LogFilePath("/some_parent_dir", PATH);
        mLogFilePathGz = new LogFilePath("/some_parent_dir", PATH_GZ);
    }

        String prefix = getPrefix();
        String topicPrefix = getTopicPrefix();
        String[] paths = FileUtil.listRecursively(topicPrefix);
        for (String path : paths) {
            if (!path.endsWith("/_SUCCESS")) {
                LogFilePath logFilePath = new LogFilePath(prefix, path);
                TopicPartition topicPartition = new TopicPartition(logFilePath.getTopic(),
                    logFilePath.getKafkaPartition());
                SortedMap<Long, HashSet<LogFilePath>> offsetToFiles =
                    mTopicPartitionToOffsetToFiles.get(topicPartition);
                if (offsetToFiles == null) {
                    offsetToFiles = new TreeMap<Long, HashSet<LogFilePath>>();
                    mTopicPartitionToOffsetToFiles.put(topicPartition, offsetToFiles);
                }
                long offset = logFilePath.getOffset();
                HashSet<LogFilePath> logFilePaths = offsetToFiles.get(offset);
                if (logFilePaths == null) {
                    logFilePaths = new HashSet<LogFilePath>();
                    offsetToFiles.put(offset, logFilePaths);
                }

Examples of com.pinterest.secor.common.LogFilePath

Related Classes of com.pinterest.secor.common.LogFilePath