这篇来介绍一下往HDFS写数据与查看文件信息的操作.
1.读取本地文件,并写入至HDFS.
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import java.io.*;
import java.net.URI;
public class CopyToHdfs {
public static void main(String[] args) throws Exception{
String hdfsPath = "hdfs://192.168.133.153:9000/test/write/hdfs-site.xml";//hdfs位置
String localPath = "E:\\Hadoop\\hdfs-site.xml";//本地磁盘位置
Configuration configuration = new Configuration();
FileSystem fs = FileSystem.get(URI.create(hdfsPath), configuration);
FSDataOutputStream out = fs.create(new Path(hdfsPath));//创建一个输出流
InputStream in = new FileInputStream(new File(localPath));//从本地读取文件
IOUtils.copyBytes(in, out, 100, true);
System.out.println("上传完毕");
}
}
之后使用在虚拟机使用 hdfs dfs -ls /test/write 命令查看,可以发现已经写入的文件:
[scott@master hadoop]$ hdfs dfs -ls /test/write
Found 1 items
-rw-r--r-- 3 scott supergroup 2275 2019-01-20 13:11 /test/write/hdfs-site.xml
2.利用getFileStatus()展示目录信息
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import java.net.URI;
public class ShowFileStatus {
public static void main(String[] args) throws Exception{
String hdfsPath = "hdfs://192.168.133.153:9000/test/";
FileSystem fs = FileSystem.get(URI.create(hdfsPath), new Configuration());
Path path = new Path(hdfsPath);
FileStatus status = fs.getFileStatus(path);
System.out.println(status);
}
}
输出结果:
FileStatus{
path=hdfs://192.168.133.153:9000/test;
isDirectory=true;
modification_time=1547954859269;
access_time=0;
owner=scott;
group=supergroup;
permission=rwxr-xr-x;
isSymlink=false
}
这里做了一点处理,实际的输出结果这些信息全部都挤在一行.可以发现,展示的信息还是比较全面的:是否是目录.修改时间,所有者,所在组,权限,是否是符号链接等.
3.利用listStatus()显示文件夹内都有什么
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import java.net.URI;
public class ShowSubDirs {
public static void main(String[] args) throws Exception{
String dir1 = "hdfs://192.168.133.153:9000/test";
FileSystem fs = FileSystem.get(new URI(dir1), new Configuration());
Path path = new Path(dir1);
FileStatus[] status = fs.listStatus(path);
Path[] paths = FileUtil.stat2Paths(status);//又一个工具类
for (Path p: paths) {
System.out.println(p.toString());
}
}
}
输出结果:
hdfs://192.168.133.153:9000/test/read
hdfs://192.168.133.153:9000/test/write
Hadoop提供了非常实用的工具类,多多使用它们会让程序变得简单一些.