Bigtop Smoke Tests
Introduction
List of Smoke Tests
Sources
- Upstream: https://git.linaro.org/leg/bigdata/bigtop-trunk.git/
- Root upstream: https://github.com/apache/bigtop
- Release-1.2.0 is used in this Wiki
Setup Environment
- Debian Jessie 8 on AArch64
$ lsb_release -idrc Distributor ID: Debian Description: Debian GNU/Linux 8.8 (jessie) Release: 8.8 Codename: jessie
- Misc software need tobe installed and check for sudo permission
# apt-get install vim git sudo # visudo Add the below line with your username <username> ALL=(ALL) ALL # push 'Ctrl + x' key to quit visudo now log-in as username and check for the access $ sudo ls /root $ sudo apt-get update $ sudo apt-get upgrade
Install Pre-Requisites
- Open JDK
$ sudo apt-get install -t jessie-backports openjdk-8-jre-headless $ sudo apt-get install openssh-server openssh-clients java-1.8.0-openjdk* $ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64$ java -version openjdk version "1.8.0_131" OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-1~bpo8+1-b11) OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
- Docker
$ sudo apt-get install docker docker.io $ docker --version Docker version 1.12.6, build 78d1802
- Docker-compose
$ curl -L --fail https://github.com/docker/compose/releases/download/1.14.0/run.sh > /usr/local/bin/docker-compose $ sudo chmod +x /usr/local/bin/docker-compose $ docker-compose --version docker-compose version 1.14.0, build c7bdf9e Note that if you hit an error "Not found" when you check for docker-compose version. Just clone docker repository and copy the docker-composer binary under /usr/local/bin/docker-compose $ git clone https://github.com/docker/compose.git $ cd compose $ sudo cp ./bin/docker-compose /usr/local/bin/docker-compose
- Maven
$ sudo apt-get install maven $ mvn -version Apache Maven 3.0.5 Maven home: /usr/share/maven Java version: 1.8.0_131, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-8-openjdk-arm64/jre Default locale: en_IN, platform encoding: UTF-8 OS name: "linux", version: "4.12.0-219-arm64", arch: "aarch64", family: "unix" $
Smoke Test Components
- Hadoop
- Hive
- Hbase
- Spark
Test Steps
Install Pre-requisities
- Clone the bigtop-trunk git repository and branch out to erp17.08
$ git clone https://git.linaro.org/leg/bigdata/bigtop-trunk.git $ cd bigtop-trunk $ git branch erp17.08 origin/erp17.08 $ git checkout erp17.08
- Generate a docker image using build script in docker-puppet. This will create a docker image from aarch64/debian:jessie
$ cd docker/bigtop-puppet/debian-8-aarch64 $ sudo ./build.sh $ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE bigtop/puppet debian-8-aarch64 4b127acf355b 3 minutes ago 278.8 MB $ cd <BIGTOP_SRC_TOP> $ docker build -t bigtop/slaves:erp17.08-debian-8-aarch64 -f docker/bigtop-slaves/debian-8-aarch64/Dockerfile .
- Basic packages are required for all big data components. They are: bigtop-groovy, bigtop-utils, bigtop-jsvc and bigtop-tomcat
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew bigtop-groovy-deb bigtop-utils-deb bigtop-jsvc-deb bigtop-tomcat-deb'
- Build the target BigTop components like Hadoop, Hbase, Spark and Hive debian and packages
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-deb hbase-deb spark-deb hive-deb' $ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-pkg hbase-pkg spark-pkg hive-pkg'
- Build smoke test dependent packages
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew flume-deb mahout-deb pig-deb sqoop-deb zookeeper-deb'
- Create a local repo
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew apt'
Create smoke test configuration file.
Bigtop smoke test uses yaml to configure test repo/environment/components. A typical configuration yaml can be found at <BIGTOP_SRC_TOP>/provisioner/docker/erp17.08-debian-8-aarch64
Generally following properties should be customized for certain smoke test:
- docker image
- repo url
- distro type
- components to install
- components to test
Following is configuration yaml for Hive:
# Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. docker: memory_limit: "4g" image: "bigtop/puppet:debian-8-aarch64" repo: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/debian/8-aarch64/aarch64" distro: debian components: [hdfs, yarn, mapreduce, hive] enable_local_repo: true smoke_test_components: [hive] jdk: "openjdk-8-jdk"
Execute smoke tests
It is quite straight forward to execute smoke in bigtop
cd provisioner/docker ./docker-hadoop.sh -C <smoke_test_cfg_yaml> -c <node_count> -s -d for example: # ./docker-hadoop.sh -C config_debian-8-aarch64.yaml -c 3 -s -d
Before running smoke test, it is suggested to run environment check first
./docker-hadoop.sh -E
For detailed usage, just issue: ./docker-hadoop.sh
usage: docker-hadoop.sh [-C file ] args -C file Use alternate file for config.yaml commands: -c NUM_INSTANCES, --create NUM_INSTANCES Create a Docker based Bigtop Hadoop cluster -d, --destroy Destroy the cluster -e, --exec INSTANCE_NO|INSTANCE_NAME Execute command on a specific instance. Instance can be specified by name or number. For example: docker-hadoop.sh --exec 1 bash docker-hadoop.sh --exec docker_bigtop_1 bash -E, --env-check Check whether required tools has been installed -l, --list List out container status for the cluster -p, --provision Deploy configuration changes -s, --smoke-tests Run Bigtop smoke tests -h, --help
- To run the Spark smoke tests you need to apply 2 patches https://issues.apache.org/jira/browse/BIGTOP-2860 and https://issues.apache.org/jira/browse/BIGTOP-2866
$ vim provisioner/docker/config_debian-8-aarch64.yaml Do the below changes components: [hdfs, yarn, mapreduce, hive, spark] enable_local_repo: true smoke_test_components: [spark] jdk: "openjdk-8-jdk" save and quit. $ vim provisioner/utils/smoke-tests.sh Add the below lines export HADOOP_HOME=/usr/lib/hadoop export HADOOP_PREFIX=$HADOOP_HOME export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib/native" export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec export HADOOP_CONF_DIR=/etc/hadoop/conf export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce export HADOOP_HDFS_HOME=/usr/lib/hadoop-hdfs export YARN_HOME=/usr/lib/hadoop-yarn export HADOOP_YARN_HOME=/usr/lib/hadoop-yarn/ export HADOOP_USER_NAME=hdfs export CLASSPATH=$CLASSPATH:. export CLASSPATH=$CLASSPATH:$HADOOP_HOME/hadoop-common-2.7.2.jar:$HADOOP_HOME/client/hadoop-hdfs-2.7.2.jar:$HADOOP_HOME/hadoop-auth-2.7.2.jar:/usr/lib/hadoop-mapreduce/*:/usr/lib/hive/lib/*:/usr/lib/hadoop/lib/*: export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::") export PATH=/usr/lib/hadoop/libexec:/etc/hadoop/conf:$HADOOP_HOME/bin/:$PATH export SPARK_HOME=/usr/lib/spark export PATH=$HADOOP_HOME\bin:$PATH export SPARK_DIST_CLASSPATH=$HADOOP_HOME\bin\hadoop:$CLASSPATH:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/lib/*:/usr/lib/hadoop-mapreduce/*:. export CLASSPATH=$CLASSPATH:/usr/lib/hadoop/lib/*:. export SPARK_HOME=/usr/lib/spark export SPARK_MASTER_IP=172.17.0.2 export SPARK_MASTER_PORT=7077 save and quit. Run the command smoke test commands $ ./docker-hadoop.sh -C config_debian-8-aarch64.yaml -c 3 -s -d If the smoke test is still failing. It might be because of you have't set the "SPARK_MASTER_IP" properly. Another way to do is that remove the $masterMode and add local[*] as below $ vim bigtop-tests/smoke-tests/spark/TestSpark.groovy make the below change final String SPARK_SHELL = SPARK_HOME + "/bin/spark-shell --master local[*]" save and quit. Now run the spark smoke-tests again.
- Using sandbox
$ cd docker/sandbox If you are pointing to any other image e.g. the image located on your local git hub, you can edit the build script and change the "FROM bigtop/puppet:${OS}" $ sudo ./build.sh -a bigtop -o debian-8-aarch64 -c "hadoop, spark, hive, hbase"
- How to Tag and Export bigtop-puppet docker image in your own area. You need to create an account on https://hub.docker.com/ following are the example commands
$ sudo docker tag 4b127acf355b nbhat/puppet:debian-8-aarch64 $ sudo docker login $ sudo docker push nbhat/puppet:debian-8-aarch64
- How to create bigtop-slaves image using the image in your area
$ cd docker/bigtop-slaves/debian-8-aarch64 $ vim Dockerfile Change the image fetching path in Dockerfile for example: "FROM bigtop/puppet:debian-8-aarch64" to "FROM nbhat/puppet:debian-8-aarch64" Note that the image must be available on my area at https://hub.docker.com/ $ cd bigtop-trunk $ git branch * erp17.08 master $ sudo ./gradlew -POS=debian-8-aarch64 -Pprefix=erp17.08 bigtop-slaves $ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE bigtop/slaves erp17.08-debian-8-aarch64 9d5d71be2c8b 2 minutes ago 2.804 GB bigtop/puppet debian-8-aarch64 4b127acf355b About an hour ago 278.8 MB nbhat/puppet debian-8-aarch64 4b127acf355b About an hour ago 278.8 MB
- Check the task you want to build.
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew :task --all'
Patches
Upstream and Resolved:
https://issues.apache.org/jira/browse/BIGTOP-2838
https://issues.apache.org/jira/browse/BIGTOP-2843
https://issues.apache.org/jira/browse/BIGTOP-2841
https://issues.apache.org/jira/browse/BIGTOP-2851
https://issues.apache.org/jira/browse/BIGTOP-2852
https://issues.apache.org/jira/browse/BIGTOP-2853
Patches under discussion:
https://issues.apache.org/jira/browse/BIGTOP-2856
https://issues.apache.org/jira/browse/BIGTOP-2854