ERP 18.06 Building and testing BigData components using Bigtop on Debian-9:AArch64
ERP 18.06 Building and testing BigData components using Bigtop on Debian-9:AArch64
Pre-requisites
Install distro packages
sudo apt-get install unzip openjdk-8-jdk git autoconf curl iputils-ping net-tools build-essential ruby
Set JAVA_HOME
export JAVA_HOME=`readlink -f /usr/bin/java | sed "s:jre/bin/java::"`
Install docker CE
sudo apt-get install apt-transport-https ca-certificates gnupg2 software-properties-common # download docker's gpg key curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add - # verify key # the output should be: # pub rsa4096 2017-02-22 [SCEA] # 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 # uid [ unknown] Docker Release (CE deb) <docker@docker.com> # sub rsa4096 2017-02-22 [S] sudo apt-key fingerprint 0EBFCD88 # import docker's repo echo "deb [arch=arm64] https://download.docker.com/linux/debian \ $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list # now update sources sudo apt update # now install docker-ce and docker-compose sudo apt install -y docker-ce docker-compose
Get Source
git clone https://git.linaro.org/leg/bigdata/bigtop-trunk.git -b erp18.06
Build
Build Bigtop dev images
Bigtop-puppet
./gradlew -POS=debian-9 -Pprefix=erp18.06 bigtop-puppet
This will generate bigtop/puppet:erp18.06-debian-9-aarch64
image.
Bigtop-slaves
./gradlew -POS=debian-9 -Pprefix=erp18.06 bigtop-slaves
This will generate bigtop/slaves:erp18.06-debian-9-aarch64
image.
Build Components
# components to be built: # ambari bigtop-groovy bigtop-jsvc bigtop-tomcat bigtop-utils hadoop hbase hive spark zookeeper # commmand to build a certain comp: # docker run -v `pwd`:/ws bigtop/slaves:erp18.06-debian-9-aarch64 bash -l -c 'cd /ws ; ./gradlew <comp>-deb' # use hadoop as an example docker run -v `pwd`:/ws bigtop/slaves:erp18.06-debian-9-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-deb'
Tests
Generate local repo
docker run -v `pwd`:/ws bigtop/slaves:erp18.06-debian-9-aarch64 bash -l -c 'cd /ws ; ./gradlew apt'
Create test yaml file
There are several sample smoke test yaml files in <BIGTOP_ROOT>/provisioner/docker/
, you may use one as the start point. Following is an example for hive smoke test.
docker: memory_limit: "8g" image: "bigtop/puppet:erp18.06-debian-9-aarch64" #repo: "http://repos.bigtop.apache.org/releases/1.2.1/debian/8/x86_64" repo: "file:///bigtop-home/output/apt" distro: debian components: [hdfs, yarn, mapreduce, hive, zookeeper] enable_local_repo: false smoke_test_components: [hive] # you may also execute multiple test cases in one run with following settings: #smoke_test_components: [hdfs, yarn, mapreduce, hive]
Run smoke test
To run smoke test provided by Bigtop, do as following
cd provisioner/docker # smoke.yaml is the one user created for specific component(s) # For detail usage of docker-hadoop.sh, you may just issue "./docker-hadoop.sh", it will show help messages ./docker-hadoop.sh -C smoke.yaml -c 3 -s -d
, multiple selections available,
Related content
ERP 18.06 Building and testing BigData components using Bigtop on Debian-9:AArch64
ERP 18.06 Building and testing BigData components using Bigtop on Debian-9:AArch64
More like this
BigData components
BigData components
More like this
Building BigTop using Docker container
Building BigTop using Docker container
More like this
Bigtop (v1.3.0) Build, Smoketest, and Deploy on Multiple Physical Machines
Bigtop (v1.3.0) Build, Smoketest, and Deploy on Multiple Physical Machines
More like this
ERP 17.08 - Build, Configure, Install and Test Hadoop, Spark, Hive, Hbase in containerized environment
ERP 17.08 - Build, Configure, Install and Test Hadoop, Spark, Hive, Hbase in containerized environment
More like this
ERP 17.08 - Build, Configure, Install and Test Hadoop, Spark, Hive, Hbase in containerized environment
ERP 17.08 - Build, Configure, Install and Test Hadoop, Spark, Hive, Hbase in containerized environment
More like this