此前配置好分布式Hadoop环境,此篇主要讲解通过Intellij IDEA编写分布式MapReduce程序以及利用Hadoop实现词频统计
系统环境
- 虚拟机:VirtualBox
- Linux:Ubuntu 16.04 LTS
- Hadoop 2.7.5
- IDE:Intellij IDEA
- JDK 1.8.0_151
安装Intellij及破解
创建Hadoop工程
创建新工程
打开Intellij IDEA,创建一个新工程
选择Java项目,并添加JDK路径
添加依赖包
点击File-Project Structure
打开后点击左侧Modules
,然后点击Dependencies
点击右侧+
,选择JARs or directories
,将下图所有依赖包的目录导入
【注】/usr/hadoop-2.7.5
是Hadoop安装目录
编写代码
新建一个类名为WordCount
,代码如下
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
public class WordCount {
public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();
public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
配置编译环境
点击Run-Edit Configuration
点击左上角+
,然后点击Application
Name
,即该运行配置的名字,这里命名为RunHadoop
Main Class
,即需要运行的主类,这里使用的默认包,所以填写WordCount
Program arguments
,即运行时需要输入的参数,此处填写参数为hdfs://master:9000/data/input/README.txt hdfs://master:9000/output/
此处,第一个参数为输入文件路径,第二个参数为输出文件路径。
master:9000
为分布式Hadoop环境中core-site.xml
配置文件中fs.default.name
的值/data/input/README.txt
为分布式Hadoop环境中利用hdfs dfs -mkdir -p /data/input hdfs dfs -put README.txt /data/input
创建并上传到HDFS系统中的文件路径。
【注】若hdfs://master:9000/output
已经存在,需要手动删除
hdfs dfs -rm -r /output
查看运行结果
web界面查看
通过http://localhost:50070
查看各个结点运行状况