Skip to end of banner
Go to start of banner

Bigtop Smoke Tests

Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 19 Current »


Introduction


 

List of Smoke Tests


Sources

Setup Environment

  • Debian Jessie 8 on AArch64
$ lsb_release -idrc
Distributor ID:    Debian
Description:       Debian GNU/Linux 8.8 (jessie)
Release:           8.8
Codename:          jessie
  • Misc software need tobe installed and check for sudo permission
# apt-get install vim git sudo
# visudo

Add the below line with your username

<username>   ALL=(ALL)   ALL

# push 'Ctrl + x' key to quit visudo

now log-in as username and check for the access

$ sudo ls /root
$ sudo apt-get update                                                           
$ sudo apt-get upgrade                                                

Install Pre-Requisites

  • Open JDK
$ sudo apt-get install -t jessie-backports openjdk-8-jre-headless
$ sudo apt-get install openssh-server openssh-clients java-1.8.0-openjdk*
$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
  • Docker
$ sudo apt-get install docker docker.io
$ docker --version
Docker version 1.12.6, build 78d1802
  • Docker-compose
$ curl -L --fail https://github.com/docker/compose/releases/download/1.14.0/run.sh > /usr/local/bin/docker-compose

$ sudo chmod +x /usr/local/bin/docker-compose
$ docker-compose --version
docker-compose version 1.14.0, build c7bdf9e

Note that if you hit an error "Not found" when you check for docker-compose version.  Just clone docker repository and copy the docker-composer 
binary under /usr/local/bin/docker-compose

$ git clone https://github.com/docker/compose.git
$ cd compose
$ sudo cp ./bin/docker-compose /usr/local/bin/docker-compose
  • Maven
$ sudo apt-get install maven
$ mvn -version
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-arm64/jre
Default locale: en_IN, platform encoding: UTF-8
OS name: "linux", version: "4.12.0-219-arm64", arch: "aarch64", family: "unix"
$

Smoke Test Components

  • Hadoop
  • Hive
  • Hbase
  • Spark

Test Steps

Install Pre-requisities

  • Clone the bigtop-trunk git repository and branch out to erp17.08
$ git clone https://git.linaro.org/leg/bigdata/bigtop-trunk.git
$ cd bigtop-trunk
$ git branch erp17.08 origin/erp17.08
$ git checkout erp17.08
  • Generate a docker image using build script in docker-puppet.  This will create a docker image from aarch64/debian:jessie
$ cd docker/bigtop-puppet/debian-8-aarch64
$ sudo ./build.sh
$ sudo docker images
REPOSITORY          TAG                   IMAGE ID            CREATED             SIZE
bigtop/puppet       debian-8-aarch64     4b127acf355b        3 minutes ago       278.8 MB

$ cd <BIGTOP_SRC_TOP>
$ docker build -t bigtop/slaves:erp17.08-debian-8-aarch64 -f docker/bigtop-slaves/debian-8-aarch64/Dockerfile .
  • Basic packages are required for all big data components. They are: bigtop-groovy, bigtop-utils, bigtop-jsvc and bigtop-tomcat
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew bigtop-groovy-deb bigtop-utils-deb bigtop-jsvc-deb bigtop-tomcat-deb'
  • Build the target BigTop components like Hadoop, Hbase, Spark and Hive debian and packages
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-deb hbase-deb spark-deb hive-deb'
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-pkg hbase-pkg spark-pkg hive-pkg'
  • Build smoke test dependent packages
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew flume-deb mahout-deb pig-deb sqoop-deb zookeeper-deb'
  • Create a local repo
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew apt'

Create smoke test configuration file.

Bigtop smoke test uses yaml to configure test repo/environment/components. A typical configuration yaml can be found at <BIGTOP_SRC_TOP>/provisioner/docker/erp17.08-debian-8-aarch64

Generally following properties should be customized for certain smoke test:

  • docker image
  • repo url
  • distro type
  • components to install
  • components to test

Following is configuration yaml for Hive:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

docker:
        memory_limit: "4g"
        image:  "bigtop/puppet:debian-8-aarch64"

repo: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/debian/8-aarch64/aarch64"
distro: debian
components: [hdfs, yarn, mapreduce, hive]
enable_local_repo: true
smoke_test_components: [hive]
jdk: "openjdk-8-jdk"

Execute smoke tests

It is quite straight forward to execute smoke in bigtop

cd provisioner/docker
./docker-hadoop.sh -C <smoke_test_cfg_yaml> -c <node_count> -s -d

Before running smoke test, it is suggested to run environment check first

./docker-hadoop.sh -E

For detailed usage, just issue: ./docker-hadoop.sh

usage: docker-hadoop.sh [-C file ] args
       -C file                                   Use alternate file for config.yaml
  commands:
       -c NUM_INSTANCES, --create NUM_INSTANCES  Create a Docker based Bigtop Hadoop cluster
       -d, --destroy                             Destroy the cluster
       -e, --exec INSTANCE_NO|INSTANCE_NAME      Execute command on a specific instance. Instance can be specified by name or number.
                                                 For example: docker-hadoop.sh --exec 1 bash
                                                              docker-hadoop.sh --exec docker_bigtop_1 bash
       -E, --env-check                           Check whether required tools has been installed
       -l, --list                                List out container status for the cluster
       -p, --provision                           Deploy configuration changes
       -s, --smoke-tests                         Run Bigtop smoke tests
       -h, --help


  • Using sandbox
$ cd docker/sandbox
If you are pointing to any other image e.g. the image located on your local git hub, you can edit the build script and change the "FROM bigtop/puppet:${OS}"
$ sudo ./build.sh -a bigtop -o debian-8-aarch64 -c "hadoop, spark, hive, hbase"
  • How to Tag and Export bigtop-puppet docker image in your own area.  You need to create an account on https://hub.docker.com/ following are the example commands
$ sudo docker tag 4b127acf355b nbhat/puppet:debian-8-aarch64
$ sudo docker login 
$ sudo docker push nbhat/puppet:debian-8-aarch64
  • How to create bigtop-slaves image using the image in your area
$ cd  docker/bigtop-slaves/debian-8-aarch64
$ vim Dockerfile

Change the image fetching path in Dockerfile for example: "FROM bigtop/puppet:debian-8-aarch64" to "FROM nbhat/puppet:debian-8-aarch64" 
Note that the image must be available on my area at https://hub.docker.com/  

$ cd bigtop-trunk
$ git branch
* erp17.08
  master
$ sudo ./gradlew -POS=debian-8-aarch64 -Pprefix=erp17.08 bigtop-slaves
$ sudo docker images
REPOSITORY          TAG                                        IMAGE ID            CREATED             SIZE
bigtop/slaves       erp17.08-debian-8-aarch64                  9d5d71be2c8b        2 minutes ago       2.804 GB
bigtop/puppet       debian-8-aarch64                           4b127acf355b        About an hour ago   278.8 MB
nbhat/puppet        debian-8-aarch64                           4b127acf355b        About an hour ago   278.8 MB
  • Check the task you want to build.
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew :task --all'


  • Patches

Upstream and Resolved:

https://issues.apache.org/jira/browse/BIGTOP-2838
https://issues.apache.org/jira/browse/BIGTOP-2843

https://issues.apache.org/jira/browse/BIGTOP-2841

https://issues.apache.org/jira/browse/BIGTOP-2851

https://issues.apache.org/jira/browse/BIGTOP-2852

https://issues.apache.org/jira/browse/BIGTOP-2853

Patches under discussion:

https://issues.apache.org/jira/browse/BIGTOP-2856

https://issues.apache.org/jira/browse/BIGTOP-2854



  • No labels