1、首先下载windows x64的hadoop客户端依赖包
hadoop客户端依赖包
2、配置环境变量
需要将依赖包解压缩,配置为HADOOP_HOME,同时将bin文件夹设置到PATH中
3、使用IntellijIDEA2017选择maven工程
下面是一个参考maven配置
<repositories>
<repository>
<id>nexus-aliyun</id>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</repository>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
</dependencies>
其中,阿里云不是必须的,只是可以加速。cloudera是必须的,因为我用的是CDH版本的hadoop。
4、hdfs测试代码
public class HDFSApp {
public static final String HDFS_PATH="hdfs://192.168.190.133:8020";
FileSystem fileSystem=null;
Configuration configuration=null;
@Before
public void setUp() throws Exception{
configuration=new Configuration();
fileSystem=FileSystem.get(new URI(HDFS_PATH),configuration);
System.out.println("HDFS APP SETUP");
}
@Test
public void create() throws IOException {
FSDataOutputStream outputStream=fileSystem.create(new Path("/hdfsapi/test/a.txt"));
outputStream.write("hello hadoop".getBytes());
outputStream.flush();
outputStream.close();
}
@Test
public void cat() throws IOException {
FSDataInputStream inputStream=fileSystem.open(new Path("/hdfsapi/test/a.txt"));
IOUtils.copyBytes(inputStream,System.out,1024);
inputStream.close();
}
@Test
public void mkdir() throws IOException {
fileSystem.mkdirs(new Path("/hdfsapi/test"));
}
@After
public void tearDown() throws Exception{
configuration=null;
configuration=null;
System.out.println("HDFS APP SHUTDOWN");
}
}
5、运行测试代码,unit test pass即为配置完成