/
Build and Install Apache Ambari-2.2.0 on AArch64

Build and Install Apache Ambari-2.2.0 on AArch64

Oct 24, 2018

Introduction

Apache Ambari project is aimed at making Hadoop management simpler by developing software for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive, easy-to-use Hadoop management web UI backed by its RESTful APIs.

Ambari enables System Administrators to:

  • Provision a Hadoop Cluster

  • Manage a Hadoop Cluster

  • Monitor a Hadoop Cluster

Ambari is officially confirmed to work on X86_64 platform with JDK7 on Ubuntu 12/14, CentOS 5/6, SuSE 11, while version for JDK8/Ubuntu16 is still in development.

The challenges for AArch64 porting are:

  • Node.js dependencies: Ambari uses old version of node.js (v0.10.44) while AArch64 support only comes to v4.x and later.

  • Phantomjs dependencies: Ambari uses phantomjs 1.9.8, while AArch64 support only comes to v2.1.0 and later.

  • Jars dependencies: some Java projects which referenced by Ambari have no AArch64 support. 

Build

Sources

Setup Environment

  • Ubuntu 16.04 64bit for AArch64, or CentOS7 64bit for AArch64

  • Oracle JDK 1.8.0_111-b14

Pre-Requisites

Oracle JDK 1.8.0_111-b14

Dependencies

maven@v3.0.5, nodejs@v4.2.6, npm@3.5.2, brunch@1.7.10, phantomjs@1.9.8, python>=2.6, python-dev, rpm, yum, g++

Build Steps

Install Pre-requisities

For Ubuntu:

sudo apt install python python-dev rpm yum build-essential libfreetype6 libfreetype6-dev fontconfig fontconfig-config libfontconfig1-dev libssl-dev openssl findbugs -y

For CentOS7

sudo yum groupinstall "Development Tools" sudo yum install python python-devel openssl-devel openssl openssl-libs freetype freetype-devel fontconfig-devel fontconfig gcc gcc-c++ make build autoconf automake cppunit-devel cmake bzip2 rpm-build

Setup maven

The maven comes along with Ubuntu 16.04 is v3.3.9. Ambari requires maven to be 3.0.5. To build v3.0.5, do as:

wget https://archive.apache.org/dist/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz tar xvf apache-maven-3.0.5-bin.tar.gz cd apache-maven-3.0.5/bin export PATH=$PWD:$PATH

Make sure the version of Maven is 3.0.5 when the following command is issued.

mvn --version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2016-11-02 01:10:51+0000) Maven home: /opt/maven Java version: 1.8.0_101, vendor: Oracle Corporation Java home: /opt/jdk/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "4.4.0-45-generic", arch: "aarch64", family: "unix"

Setup python tools

  • For python 2.6, download 

    wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg#md5=bfa92100bd772d5a213eedd356d64086 sudo sh setuptools-0.6c11-py2.6.egg

  • For python 2.7, download 

    wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea sudo sh setuptools-0.6c11-py2.7.egg

Setup nodejs/npm

Nodejs and npm come along with Ubuntu 16.04 are: nodejs@v4.2.6, npm@3.5.2.

For Ubuntu, nodejs/npm can be installed by:

sudo apt-get install -y nodejs npm cd /usr/bin && sudo ln -s nodejs node sudo npm install -g brunch@1.7.10

For CentOS7, nodejs/npm need to be built from source.

git clone https://github.com/nodejs/node.git cd node git checkout -b 4.2.6 v4.2.6 ./configure --prefix=/usr && make -j8 sudo make install sudo npm install -g brunch@1.7.10

The version of built out binaries are: nodejs@v4.2.6, npm@2.14.12.

As long as they are installed, pom.xml in ambari-admin needs to be changed to reflect these versions. The target nodejs/npm version are defined in "configuration" field of "frontend-maven-plugin".

Build PhantomJS

AArch64 is not supported in phantomjs v1.9.8. Attached patch needs to be applied.

git clone https://github.com/ariya/phantomjs.git cd phantomjs git checkout -b 1.9.8 1.9.8 git apply 0001-Patch-to-build-phantomjs-on-AArch64.patch ./build.sh --confirm --jobs 4

When the build is finished, create tar file for deployment

cd deploy ./package.sh

You can test phantomjs build by issuing:

./bin/phantomjs test/run-tests.js

Install phantomjs-1.9.8-linux-aarch64.tar.bz2 to the system and add phantomjs to $PATH. Check if phantomjs is properly installed by doing:

ent-user@ambari-dk:~$ phantomjs --version 1.9.8

Replace frontend-maven-plugin

Ambari uses fronend-maven-plugin@v0.0.16, which doesn't support AArch64. Do following to rebuild this plugin for AArch64.

git clone https://github.com/eirslett/frontend-maven-plugin.git cd frontend-maven-plugin git checkout -b 0.0.16 frontend-plugins-0.0.16 git apply frontend-maven.patch mvn clean -DskipTests install

Replace leveldbjni

levedbjni is used in Ambari-metrics. It only provides x86/x86_64 version in maven repo. So AArch64 version of leveldbjni needs to be built and installed. Sometimes one leveldbjni test may fail, just ignore the failure.

wget http://pkgs.fedoraproject.org/repo/pkgs/snappy/snappy-1.0.5.tar.gz/4c0af044e654f5983f4acbf00d1ac236/snappy-1.0.5.tar.gz tar -xf snappy-1.0.5.tar.gz cd snappy-1.0.5 ./configure --disable-shared --with-pic --host aarch64-unknown-linux --build aarch64-unknown-linux make -j4 cd .. git clone git://github.com/chirino/leveldb.git git clone git://github.com/fusesource/leveldbjni.git export SNAPPY_HOME=`cd snappy-1.0.5; pwd` export LEVELDB_HOME=`cd leveldb; pwd` export LEVELDBJNI_HOME=`cd leveldbjni; pwd` cd leveldb export LIBRARY_PATH=${SNAPPY_HOME} export C_INCLUDE_PATH=${LIBRARY_PATH} export CPLUS_INCLUDE_PATH=${LIBRARY_PATH} git apply ../leveldbjni/leveldb.patch wget https://raw.githubusercontent.com/google/leveldb/master/port/atomic_pointer.h -O port/atomic_pointer.h make libleveldb.a Note: If you face following error on CentOS7 error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options. You have to edit Makefile and add CXXFLAGS +="-std=c++11" cd ${LEVELDBJNI_HOME} git checkout -b 1.8 leveldbjni-1.8 mvn clean install -P all -P linux64 -DskipTests=truebn

Build Ambari

To build Ambari, a certain version number should be provided. This version number IS 5-DIGITS, not "4-digits" mentioned on Ambari's Wiki Page. The last digit may vary but the first 3 digits should be same as Ambari source version. In our case this is 2.2.0. Patch is provided to make Ambari built on AArch64. 

git clone https://git-wip-us.apache.org/repos/asf/ambari.git cd <ambari_root_dir> git checkout release-2.2.0 git apply ambari_build_aarch64.patch mvn versions:set -DnewVersion=2.2.0.0.1 pushd ambari-metrics mvn versions:set -DnewVersion=2.2.0.0.1 popd mvn -B clean install package rpm:rpm -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl -Drat.ignoreErrors=true

Note 1: Ambari 2.4.2 should also work, just checkout release-2.4.2 and set correct version string in above command lines. For building ambari-2.4.2, apply the attached patch ambari-build_aarch64_2.4.2.patch instead of ambari_build_aarch64.patch.

Note 2: Building Ambari in CentOS may require python2.6. You can install by following above "Setup python tools" section, or by creating a symbol link "sudo ln -s /usr/bin/python2.7 /usr/bin/python2.6".

Ambari Server will create following packages

  • RPM will be created under AMBARI_DIR/ambari-server/target/rpm/ambari-server/RPMS/aarch64.

Ambari Agent will create following packages

  • RPM will be created under AMBARI_DIR/ambari-agent/target/rpm/ambari-agent/RPMS/aarch64.

Ambari Metrics will create following packages

  • RPM will be created under  AMBARI_DIR/ambari-metrics/ambari-metrics-timelineservice/target/rpm/ambari-metrics-collector/RPMS/noarch.

 

Patches

Running Ambari

Run Ambari Server

First, install Pre-Requisities

yum install postgresql

Then install the Ambari Server RPM.

yum install ambari-server/target/rpm/ambari-server/RPMS/aarch64/ambari-server-*.rpm

Initialize Ambari Server:

ambari-server setup

Start up Ambari Server:

ambari-server start

To access Ambari, go to: http://{ambari-server-hostname}:8080

The initial username/password is admin/admin.

Run Ambari Agent

Install the Ambari Agent RPM.

yum install ambari-agent/target/rpm/ambari-agent/RPMS/aarch64/ambari-agent-2.2.0.0-1.aarch64.rpm

Then edit the location of Ambari Server in /etc/ambari-agent/conf/ambari-agent.ini by editing the hostname line.

Start Ambari Agent:

sudo ambari-agent start