List of globStatus() Examples

Examples of globStatus()

org.apache.hadoop.fs.FileSystem.globStatus()

Return all the files that match filePattern and are not checksum files. Results are sorted by their names.
A filename pattern is composed of regular characters and special pattern matching characters, which are:

?
Matches any single character.

*
Matches zero or more characters.

[abc]
Matches a single character from character set {a,b,c}.

[a-b]
Matches a single character from the character range {a...b}. Note that character a must be lexicographically less than or equal to character b.

[^a]
Matches a single character that is not from character set or range {a}. Note that the ^ character must occur immediately to the right of the opening bracket.

\c
Removes (escapes) any special meaning of character c.

{ab,cd}
Matches a string from the string set {ab, cd}

{ab,c{de,fh}}
Matches a string from the string set {ab, cde, cfh}

@param pathPattern a regular expression specifying a pth pattern @return an array of paths that match the path pattern @throws IOException

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

    
    List<PriorityQueue<StringDoublePair>> queues = new ArrayList<PriorityQueue<StringDoublePair>>();
    
    IntPairWritable key = new IntPairWritable();
    DoubleWritable value = new DoubleWritable();
    for (FileStatus status : fs.globStatus(new Path(dir, "*"))) {
      Path path = status.getPath();
      SequenceFile.Reader reader = new SequenceFile.Reader(fs, path, job);
      while (reader.next(key, value)) {
        int topic = key.getFirst();
        int word = key.getSecond();

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

    IntWritable key = new IntWritable();
    LongWritable value = new LongWritable();
    Configuration conf = new Configuration();
    
    FileSystem fs = FileSystem.get(featureCountPath.toUri(), conf);
    FileStatus[] outputFiles = fs.globStatus(new Path(featureCountPath.toString()
                                                      + OUTPUT_FILES_PATTERN));
    
    long chunkSizeLimit = chunkSizeInMegabytes * 1024 * 1024;
    int chunkIndex = 0;
    Path chunkPath = getPath(dictionaryPathBase + FREQUENCY_FILE, chunkIndex);

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

    
    try {
      // get all filtered file names in result list
      FileSystem fs = clusterPath.getFileSystem(job);
      FileStatus[] matches = fs.listStatus(
        FileUtil.stat2Paths(fs.globStatus(clusterPath, clusterFileFilter)), clusterFileFilter);
      
      for (FileStatus match : matches) {
        result.add(fs.makeQualified(match.getPath()));
      }

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

    
    double ll = 0.0;
    
    IntPairWritable key = new IntPairWritable();
    DoubleWritable value = new DoubleWritable();
    for (FileStatus status : fs.globStatus(new Path(dir, "part-*"))) {
      Path path = status.getPath();
      SequenceFile.Reader reader = new SequenceFile.Reader(fs, path, job);
      while (reader.next(key, value)) {
        if (key.getFirst() == LOG_LIKELIHOOD_KEY) {
          ll = value.get();

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

    double[] logTotals = new double[numTopics];
    double ll = 0.0;
    
    IntPairWritable key = new IntPairWritable();
    DoubleWritable value = new DoubleWritable();
    for (FileStatus status : fs.globStatus(new Path(dir, "part-*"))) {
      Path path = status.getPath();
      SequenceFile.Reader reader = new SequenceFile.Reader(fs, path, job);
      while (reader.next(key, value)) {
        int topic = key.getFirst();
        int word = key.getSecond();

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

       * Performing glob pattern matching
       */
      List<Path> result = new ArrayList<Path>(paths.length);
      for (Path p : paths) {
        FileSystem fs = p.getFileSystem(jobConf);
        FileStatus[] matches = fs.globStatus(p);
        if (matches == null) {
          LOG.warn("Input path does not exist: " + p);
        }
        else if (matches.length == 0) {
          LOG.warn("Input Pattern " + p + " matches 0 files");

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

       * Performing glob pattern matching
       */
      List<Path> result = new ArrayList<Path>(paths.length);
      for (Path p : paths) {
        FileSystem fs = p.getFileSystem(jobConf);
        FileStatus[] matches = fs.globStatus(p);
        if (matches == null) {
          throw new IOException("Input path does not exist: " + p);
        }
        else if (matches.length == 0) {
          LOG.warn("Input Pattern " + p + " matches 0 files");

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

        }
        return true;
      }
    };


    FileStatus[] statusList = fs.globStatus(queryPath, pathFilter);


    if (null == statusList) {
      return;
    } else {
      LOG.debug("Query Path: " + queryPath + " ; # list of files: " +

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

            // separate them out
            for(String location : LoadFunc.getPathStrings(input)){
                if(! UriUtil.isHDFSFileOrLocal(location)){
                    continue;
                }
                FileStatus[] status=fs.globStatus(new Path(location));
                if (status != null){
                    for (FileStatus s : status){
                        size += getPathLength(fs, s);
                    }
                }

View Full Code Here

Examples of org.apache.hadoop.fs.FileSystem.globStatus()

    System.err.println("Patterns:" + patterns);
    ArrayList<Path> filesToSearch = new ArrayList<Path>();


    FileSystem fs = getFS(conf, args[filterArg + 1]);
    for(int i=filterArg + 1; i < args.length; ++i){
      Path[] globbedPaths = FileUtil.stat2Paths(fs.globStatus(new Path(args[i])));
      if(globbedPaths != null)
        for(Path p: globbedPaths)
          filesToSearch.add(p);
    }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.