CentOS7编译Hadoop 2.7.3

编译环境

  • CentOS Linux release 7.2.1511 (Core)
  • Hadoop 2.7.3
  • JDK 1.7.0_80

编译准备

本文参考Hadoop源码中官方编译文档BUILDING.txt编写。

安装Maven

  • 安装命令
1
yum -y install maven
  • 检查命令
1
rpm -q maven

安装依赖包

  • 安装命令
1
yum -y install build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
  • 检查命令
1
rpm -q build-essential autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev

注意:

执行安装命令后,会出现以下提示信息:

No package build-essential available.
Package autoconf-2.69-11.el7.noarch already installed and latest version
Package automake-1.13.4-3.el7.noarch already installed and latest version
Package libtool-2.4.2-21.el7_2.x86_64 already installed and latest version
Package cmake-2.8.11-5.el7.x86_64 already installed and latest version
No package zlib1g-dev available.
No package pkg-config available.
No package libssl-dev available.

  • No package build-essential available.解决方案
1
yum groupinstall 'Development Tools'
  • No package zlib1g-dev available.解决方案
1
2
yum install zlib
yum install zlib-devel
  • No package pkg-config available.解决方案
1
yum install pkgconfig
  • No package libssl-dev available.解决方案
1
2
yum install openssl
yum install openssl-devel

安装ProtocolBuffer

  • 安装命令
1
yum -y install libprotobuf-dev protobuf-compiler
  • 检查命令
1
rpm -q libprotobuf-dev protobuf-compiler

安装Findbugs

  • 下载Findbugs
1
wget http://kent.dl.sourceforge.net/project/findbugs/findbugs/3.0.1/findbugs-3.0.1.tar.gz
  • 解压Findbugs到/usr/share/findbugs目录
1
2
3
4
5
# 解压findbugs-3.0.1
tar -zxvf findbugs-3.0.1.tar.gz

# 重命名为findbugs
mv findbugs-3.0.1 findbugs
  • 配置环境变量
1
2
3
4
5
6
7
8
9
# 编辑 ~/.bashrc文件
vi ~/.bashrc

# 配置Findbugs环境变量
export FINDBUGS_HOME=/usr/share/findbugs
export PATH=$FINDBUGS_HOME/bin:$PATH

# 编译.bashrc文件使其生效
source ~/.bashrc
  • 检查是否安装成功
1
2
3
4
findbug -version

# 输出以下结果表示安装成功
3.0.1

开始编译

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Create binary distribution without native code and without documentation:

$ mvn package -Pdist -DskipTests -Dtar

# Create binary distribution with native code and with documentation:

$ mvn package -Pdist,native,docs -DskipTests -Dtar

# Create source distribution:

$ mvn package -Psrc -DskipTests

# Create source and binary distributions with native code and documentation:

$ mvn package -Pdist,native,docs,src -DskipTests -Dtar

# Create a local staging version of the website (in /tmp/hadoop-site)

$ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

编辑完成后,会在[HADOOP_SOURCE_HOME]/hadoop-dist/target目录生成相关发布包,如下图所示:

493e0c0dbe75ec2d/ca77fa67656f46818ead75716102d1df.png

常见问题

Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7

1
2
3
4
5
6
7
8
9
10
11
12
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="/opt/software/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:134 in /opt/software/hadoop-2.7.3-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-common
  • 问题原因

缺少zlib1g-devlibssl-dev依赖库