Hadoop学习-编译并运行maxTemperature
按照书中样例编写代码
其中的MaxTemperatureMapper类定义有错误,应该写为
public class MaxTemperatureMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
使用mvn命令编译打包
mvn clean compile package
打包成功后生成jar包
maxTemperature-1.0-SNAPSHOT.jar
在Hadoop上创建输入路径及文件
hdfs dfs -mkdir input/ncdc
hdfs dfs -put input/ncdc/sample.txt /user/lianyz/input/ncdc/sample.txt
使用以下命令查看hadoop路径及文件
hdfs dfs -ls -R
删除多余的文件
hdfs dfs -rm -r ncdc
运行程序计算sample.txt
hadoop jar maxTemperature-1.0-SNAPSHOT.jar com.lianyz.hadoop.MaxTemperature input/ncdc/sample.txt output/maxTemperature
查看计算结果
hdfs dfs -cat output/maxTemperature/part-r-00000
计算结果如下
2020-05-03 08:07:07,462 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-05-03 08:07:08,044 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1949 111
1950 22
运行程序计算all
hdfs dfs -put input/ncdc/all/*.gz /user/lianyz/input/ncdc/
hdfs dfs -rm input/ncdc/sample.txt
hdfs dfs -rm output/maxTemperature
hadoop jar maxTemperature-1.0-SNAPSHOT.jar com.lianyz.hadoop.MaxTemperature input/ncdc output/maxTemperature
查看计算结果
hdfs dfs -cat output/maxTemperature/part-r-00000
计算结果如下
2020-05-03 08:13:31,537 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-05-03 08:13:32,133 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1901 333
1902 328