ERP 18.06 Building and testing BigData components using Bigtop on Debian-9:AArch64

Owned by Jun He
Jun 15, 2018
2 min read

Pre-requisites

Install distro packages

sudo apt-get install unzip openjdk-8-jdk git autoconf curl iputils-ping net-tools build-essential ruby

Set JAVA_HOME

export JAVA_HOME=`readlink -f /usr/bin/java | sed "s:jre/bin/java::"`

Install docker CE

sudo apt-get install apt-transport-https ca-certificates gnupg2 software-properties-common
# download docker's gpg key
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -

# verify key
# the output should be:
# pub   rsa4096 2017-02-22 [SCEA]
#      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
# uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
# sub   rsa4096 2017-02-22 [S]
sudo apt-key fingerprint 0EBFCD88

# import docker's repo
echo "deb [arch=arm64] https://download.docker.com/linux/debian \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list

# now update sources
sudo apt update

# now install docker-ce and docker-compose
sudo apt install -y docker-ce docker-compose

Get Source

git clone https://git.linaro.org/leg/bigdata/bigtop-trunk.git -b erp18.06

Build

Build Bigtop dev images

Bigtop-puppet

./gradlew -POS=debian-9 -Pprefix=erp18.06 bigtop-puppet

This will generate bigtop/puppet:erp18.06-debian-9-aarch64 image.

Bigtop-slaves

./gradlew -POS=debian-9 -Pprefix=erp18.06 bigtop-slaves

This will generate bigtop/slaves:erp18.06-debian-9-aarch64 image.

Build Components

# components to be built:
# ambari bigtop-groovy bigtop-jsvc bigtop-tomcat bigtop-utils hadoop hbase hive spark zookeeper
# commmand to build a certain comp:
# docker run -v `pwd`:/ws bigtop/slaves:erp18.06-debian-9-aarch64 bash -l -c 'cd /ws ; ./gradlew <comp>-deb'
# use hadoop as an example
docker run -v `pwd`:/ws bigtop/slaves:erp18.06-debian-9-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-deb'

Tests

Generate local repo

docker run -v `pwd`:/ws bigtop/slaves:erp18.06-debian-9-aarch64 bash -l -c 'cd /ws ; ./gradlew apt'

Create test yaml file

There are several sample smoke test yaml files in <BIGTOP_ROOT>/provisioner/docker/, you may use one as the start point. Following is an example for hive smoke test.

docker:
        memory_limit: "8g"
        image: "bigtop/puppet:erp18.06-debian-9-aarch64"

#repo: "http://repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64"
repo: "file:///bigtop-home/output/apt"
distro: debian
components: [hdfs, yarn, mapreduce, hive, zookeeper]
enable_local_repo: false
smoke_test_components: [hive]
# you may also execute multiple test cases in one run with following settings:
#smoke_test_components: [hdfs, yarn, mapreduce, hive]

Run smoke test

To run smoke test provided by Bigtop, do as following

cd provisioner/docker
# smoke.yaml is the one user created for specific component(s)
# For detail usage of docker-hadoop.sh, you may just issue "./docker-hadoop.sh", it will show help messages
./docker-hadoop.sh -C smoke.yaml -c 3 -s -d