版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/qq_22830285/article/details/84206306
使用MRunit 测试 MapReduce ,就比如我们写java 程序是的junit单元测试,其实,mrunit 和标准的junit框架一起使用,可以将mapreduce作业的测试作为正常开发环境的一部分运行。
准备测试的包:mrunit-1.1.0-hadoop2.jar ,下载地址;https://repository.apache.org/content/repositories/releases/org/apache/mrunit/mrunit/1.1.0/
单单一个 还不够,因为我测试的时候报了很多错,发现是缺少
apache-mrunit-1.1.0-hadoop2-bin.tar.gz lib 文件目录下相关的包
下载地址:http://archive.apache.org/dist/mrunit/mrunit-1.0.0/
将上面相关的jar文件导入eclipse 项目中,同时导入junit4.
以wordcount 为例:
对WordCountMap 右键,新建 junit test case 工程。
我这里用 MapDriver 测试驱动。
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.junit.Test;
public class WordCountMapTest {
@Test
public void testMapLongWritableTextContext() throws IOException {
Text value=new Text("a good man ");
new MapDriver<LongWritable, Text, Text, IntWritable>()
.withMapper(new WordCountMap())
.withInput(new LongWritable(1),value)
.withOutput(new Text("a"),new IntWritable(1))
.withOutput(new Text("good"),new IntWritable(1))
.withOutput(new Text("man"),new IntWritable(1))
.runTest();
}
}
右键run as---->junit
WordCountReduce的测试,使用ReduceDriver 驱动。代码如下:
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
import org.junit.Before;
import org.junit.Test;
public class WordCountReduceTest {
@Before
public void setUp() throws Exception {
}
@Test
public void testReduceTextIterableOfIntWritableContext() throws IOException {
List<IntWritable> values = new ArrayList<IntWritable>();
values.add(new IntWritable(1));
values.add(new IntWritable(1));
new ReduceDriver<Text, IntWritable, Text, IntWritable>()
.withReducer(new WordCountReduce())
.withInput(new Text("6"), values)
.withOutput(new Text("6"), new IntWritable(2))
.runTest();
}
}
测试方法,如同map。