MapReduce之job提交过程源码分析-03
其他
2019-04-20 21:30:59
阅读次数: 0
MapReduce序列化即案例演示-02
下面源码分析是在上面这个案例上进行的.
![](https://img-blog.csdnimg.cn/20190414124740808.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414125138265.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414130144615.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414130201210.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414130230558.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
可以看到这里动态代理生成一个本地job对象,因为在driver中第一行没填什么配置信息,默认连接的就是本地
![](https://img-blog.csdnimg.cn/20190414130309651.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
这个是提交job的具体方法
![](https://img-blog.csdnimg.cn/20190414130812554.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414131112331.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414131232657.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
当输出路径存在时它就会抛出这个异常
![](https://img-blog.csdnimg.cn/20190414131255620.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
退出来到提交job方法
它会创建这么一个路径,原先是没有的,会在idea在哪个盘下,它就会在哪个盘下创建
![](https://img-blog.csdnimg.cn/20190414131612642.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414131632411.png)
给job生成一个id
![](https://img-blog.csdnimg.cn/20190414132134597.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
在这个方法里提交一些文件和jar包
![](https://img-blog.csdnimg.cn/20190414132427720.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414132441336.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
这个方法里就是提交一些文件和jar包,在本地的话不需要,因为已经有了,在yarn上运行的话就需要了
![](https://img-blog.csdnimg.cn/2019041413262948.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414132639623.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
当这个方法执行完的时候就会在刚才生成的路径下创建jodid文件夹
![](https://img-blog.csdnimg.cn/20190414132758952.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414132837397.png)
这个方法是生成切片信息的
![](https://img-blog.csdnimg.cn/2019041413291395.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414133125774.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
这个方法就是具体生成多个个切片的逻辑
![](https://img-blog.csdnimg.cn/2019041413321774.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
声明了两个变量最小值为1,最大值为long的最大值
![](https://img-blog.csdnimg.cn/20190414133316998.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
它会遍历这个路径下有几个文件,按照文件进行分片.
![](https://img-blog.csdnimg.cn/20190414133625291.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
分片大小
![](https://img-blog.csdnimg.cn/20190414133818498.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
是根据这个算法进行文件分片的,所以可以根据调节minSize和maxSize的大小来决定每片的大小
![](https://img-blog.csdnimg.cn/20190414133841261.png)
这个是分为几片,文件的大小/片的大小如果大于1.1那么就再进行分片
![](https://img-blog.csdnimg.cn/20190414134213521.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
这个方法执行完后就在这个路径下生成了分片信息了
![](https://img-blog.csdnimg.cn/20190414134504672.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
因为我这里就一个文件并且大小不够128兆所以只分了一片,如果是多个文件的话就是多片,它这是按照文件个数然后再根据文件大小进行分片的.
![](https://img-blog.csdnimg.cn/20190414134606954.png)
这个是往里写配置信息的
![](https://img-blog.csdnimg.cn/20190414134852984.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414135030182.png)
![](https://img-blog.csdnimg.cn/20190414135101813.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
以上操作只是提交job的过程,这句代码提交后,才开始mapreduce程序,也就是mapper,reducer程序
![](https://img-blog.csdnimg.cn/20190414140139119.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
最后它会把生成的那些文件删除掉
![](https://img-blog.csdnimg.cn/20190414135132693.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
![](https://img-blog.csdnimg.cn/20190414135438105.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2t4ajE5OTgwNTI0,size_16,color_FFFFFF,t_70)
转载自blog.csdn.net/kxj19980524/article/details/89295146