在Windows 7中编译Hadoop 2.7.7

一、下载Hadoop2.7.7源码

解压后,查看BUILDING.txt文件,其中详细描述了在windows下编译hadoop需要的条件和步骤,具体见下:

----------------------------------------------------------------------------------

Building on Windows

----------------------------------------------------------------------------------
Requirements:

* Windows System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer
* Windows SDK 7.1 or Visual Studio 2010 Professional
* Windows SDK 8.1 (if building CPU rate control for the container executor)
* zlib headers (if building native code bindings for zlib)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* Unix command-line tools from GnuWin32: sh, mkdir, rm, cp, tar, gzip. These
  tools must be present on your PATH.

Unix command-line tools are also included with the Windows Git package which
can be downloaded from http://git-scm.com/download/win.

If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012).
Do not use Visual Studio Express.  It does not support compiling for 64-bit,
which is problematic if running a 64-bit system.  The Windows SDK 7.1 is free to
download here:

http://www.microsoft.com/en-us/download/details.aspx?id=8279

The Windows SDK 8.1 is available to download at:

http://msdn.microsoft.com/en-us/windows/bg162891.aspx

Cygwin is neither required nor supported.

----------------------------------------------------------------------------------
Building:

Keep the source code tree in a short path to avoid running into problems related
to Windows maximum path length limitation.  (For example, C:\hdc).

Run builds from a Windows SDK Command Prompt.  (Start, All Programs,
Microsoft Windows SDK v7.1, Windows SDK 7.1 Command Prompt.)

JAVA_HOME must be set, and the path must not contain spaces.  If the full path
would contain spaces, then use the Windows short path instead.

You must set the Platform environment variable to either x64 or Win32 depending
on whether you're running a 64-bit or 32-bit system.  Note that this is
case-sensitive.  It must be "Platform", not "PLATFORM" or "platform".
Environment variables on Windows are usually case-insensitive, but Maven treats
them as case-sensitive.  Failure to set this environment variable correctly will
cause msbuild to fail while building the native code in hadoop-common.

set Platform=x64 (when building on a 64-bit system)
set Platform=Win32 (when building on a 32-bit system)

Several tests require that the user must have the Create Symbolic Links
privilege.

All Maven goals are the same as described above with the exception that
native code is built by enabling the 'native-win' Maven profile. -Pnative-win
is enabled by default when building on Windows since the native components
are required (not optional) on Windows.

If native code bindings for zlib are required, then the zlib headers must be
deployed on the build machine.  Set the ZLIB_HOME environment variable to the
directory containing the headers.

set ZLIB_HOME=C:\zlib-1.2.7

At runtime, zlib1.dll must be accessible on the PATH.  Hadoop has been tested
with zlib 1.2.7, built using Visual Studio 2010 out of contrib\vstudio\vc10 in
the zlib 1.2.7 source tree.

http://www.zlib.net/

----------------------------------------------------------------------------------
Building distributions:

 * Build distribution with native code    : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]

二、环境准备

1.mvn

下载apache-maven-3.3.9-bin.zip,解压到指定目录,配置环境变量MAVEN_HOME,在Path中添加%MAVEN_HOME%\bin。

配置完成后, 在CMD中运行mvn -version检查是否安装成功。

2.jdk

下载jdk-8u152-windows-x64.exe,根据安装指南进行安装,配置环境变量JAVA_HOME,在Path中添加%JAVA_HOME%\bin和%JAVA_HOME%\jre\bin,在CLASSPATH中添加%JAVA_HOME%\lib;%JAVA_HOME%\lib\dt.jar;%JAVA_HOME%\lib\tools.jar。

配置完成后, 在CMD中运行java -version检查是否安装成功。

3.findbug

下载findbugs-3.0.1.zip,https://nchc.dl.sourceforge.net/project/findbugs/findbugs/3.0.1/findbugs-3.0.1.tar.gz,解压到指定目录,配置环境变量FINDBUGS_HOME,在Path中添加%FINDBUGS_HOME%\bin。

配置完成后, 在CMD中运行findbugs -version检查是否安装成功。

4.cmake

下载cmake-3.12.4-win64-x64.ziphttps://cmake.org/files/v3.12/cmake-3.12.4-win64-x64.zip,解压到指定目录,配置环境变量CMAKE_HOME,在Path中添加%CMAKE_HOME%\bin。

配置完成后, 在CMD中运行cmake -version检查是否安装成功。

5.zlib

下载zlib128-dll.zip,http://zlib.net/zlib128-dll.zip,解压到指定目录,配置环境变量,将zlib128-dll的目录添加到Path。

6.git

git安装后可以在windows上运行linux
下载Git-2.19.1-64-bit.exe,https://github-production-release-asset-2e65be.s3.amazonaws.com/23216272/081b0800-c8c2-11e8-9f25-2e5ceff4ec71?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20181107%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181107T072232Z&X-Amz-Expires=300&X-Amz-Signature=0307ffb083ee4c28ee0892840fd321972f45172768d9c7ed05da8bcbcdbcfaad&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3DGit-2.19.1-64-bit.exe&response-content-type=application%2Foctet-stream 。点击,根据安装指南进行安装,在安装过程中,记得要勾选那些跟命令行工具相关的选项。

扫描二维码关注公众号,回复: 4209265 查看本文章

安装完成后打开cmd,输入linux的命令测试下,比如ls,find等

7.protobuf

下载protobuf-2.5.0.zip和protoc-2.5.0-win32.zip,解压到指定目录。

解压protoc-2.5.0-win32.zip会得到一个protoc.exe文件;

解压protobuf-2.5.0.tar.gz,我的解压路径是E:\work\hadoop\protobuf-2.5.0;
a) 将protoc.exe文件拷贝到C:\Windows\System32目录下;
b) 将protoc.exe拷贝到解压后的E:\work\hadoop\protobuf-2.5.0\src目录中
c) 在windows的cmd中进入E:\work\hadoop\protobuf-2.5.0\java 目录,执行 "mvn package"命令,开始编译,最终会在E:\work\hadoop\protobuf-2.5.0\java\target目录下生成一个protobuf-java-2.5.0.jar包;
d) 如果命令行界面出现"BUILD SUCCESS"结果说明protobuf安装成功,使用"protoc --version"命令来查看安装是否成功:


8.Windows SDK7.1

下载 window SDK 7.1, https://go.microsoft.com/fwlink/p/?LinkId=323507,点击进行安装。

9. Visual Studio 2010 Professional

ps: 刚开始只安装了Windows SDK7.1,编译时报错

E:\work\hadoop\hadoop-2.7.7-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj(44,3): error MSB4019: 未找到导入的项目“E:\Microsoft.Cpp.Default.props”。请确认 <Import> 声明中的路径正确,且磁盘上存在该文件。

在网上查,网友回答可能是vs没安装,故又安装vs2010(参考https://blog.csdn.net/huohuanyu1/article/details/73291800),但安装后该问题仍然存在,不知是否是安装问题。

三、编译Hadoop源码

在所有程序里,找到 Microsoft Visual Studio --> Visual Studio Tools --> Visual Studio x64 win64 命令提示(2010),然后进入到解压后的hadoop 的主目录中(要先进入盘符,再进入目录),输入命令编译:

mvn package -Pdist -Pdocs -DskipTests -Dtar

编译时仍报错:

E:\work\hadoop\hadoop-2.7.7-src\hadoop-common-project\hadoop-common\src\main\winutils\libwinutils.vcxproj(44,3): error MSB4019: 未找到导入的项目“E:\Microsoft.Cpp.Default.props”。请确认 <Import> 声明中的路径正确,且磁盘上存在该文件。

最后无奈之下,手动设置了VCTargetsPath

set Platform=x64
set VCTargetsPath=C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0

至此,编译正常进行。

参考文章:

https://blog.csdn.net/nidaye13123/article/details/78872329

https://blog.csdn.net/buxinchun/article/details/71335937

猜你喜欢

转载自blog.csdn.net/LiuQQu/article/details/83825304