Versions Compared

Old Version 17

changes.mady.by.user Former user

Saved on

compared with

New Version 18

changes.mady.by.user Former user

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


--IN-PROGRESS---

Table of Contents

Introduction

 

List of Smoke Tests

Sources

Setup Environment

  • Debian Jessie 8 on AArch64
Code Block
languagebash
$ lsb_release -idrc
Distributor ID:    Debian
Description:       Debian GNU/Linux 8.8 (jessie)
Release:           8.8
Codename:          jessie
  • Misc software need tobe installed and check for sudo permission
Code Block
languagebash
# apt-get install vim git sudo
# visudo

Add the below line with your username

<username>   ALL=(ALL)   ALL

# push 'Ctrl + x' key to quit visudo

now log-in as username and check for the access

$ sudo ls /root
$ sudo apt-get update                                                           
$ sudo apt-get upgrade

Install Pre-Requisites

  • Open JDK
Code Block
languagebash
$ sudo apt-get install -t jessie-backports openjdk-8-jre-headless
$ sudo apt-get install openssh-server openssh-clients java-1.8.0-openjdk*
$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
  • Docker
Code Block
languagebash
$ sudo apt-get install docker docker.io
$ docker --version
Docker version 1.12.6, build 78d1802
  • Docker-compose
Code Block
languagebash
$ curl -L --fail https://github.com/docker/compose/releases/download/1.14.0/run.sh > /usr/local/bin/docker-compose

$ sudo chmod +x /usr/local/bin/docker-compose
$ docker-compose --version
docker-compose version 1.14.0, build c7bdf9e

Note that if you hit an error "Not found" when you check for docker-compose version.  Just clone docker repository and copy the docker-composer 
binary under /usr/local/bin/docker-compose

$ git clone https://github.com/docker/compose.git
$ cd compose
$ sudo cp ./bin/docker-compose /usr/local/bin/docker-compose
  • Maven
Code Block
languagebash
$ sudo apt-get install maven
$ mvn -version
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-arm64/jre
Default locale: en_IN, platform encoding: UTF-8
OS name: "linux", version: "4.12.0-219-arm64", arch: "aarch64", family: "unix"
$

Smoke Test Components

  • Hadoop
  • Hive
  • Hbase
  • Spark

Test Steps

Install Pre-requisities

  • Clone the bigtop-trunk git repository and branch out to erp17.08
Code Block
languagebash
$ git clone https://git.linaro.org/leg/bigdata/bigtop-trunk.git
$ cd bigtop-trunk
$ git branch erp17.08 origin/erp17.08
$ git checkout erp17.08
  • Generate a docker image using build script in docker-puppet.  This will create a docker image from aarch64/debian:jessie
Code Block
languagebash
$ cd docker/bigtop-puppet/debian-8-aarch64
$ sudo ./build.sh
$ sudo docker images
REPOSITORY          TAG                   IMAGE ID            CREATED             SIZE
bigtop/puppet       debian-8-aarch64     4b127acf355b        3 minutes ago       278.8 MB
  • Export bigtop-puppet docker image in your own area.  You need to create an account on https://hub.docker.com/ following are the example commands
Code Block
languagebash
$ sudo docker tag 4b127acf355b nbhat/puppet:debian-8-aarch64
$ sudo docker login 
$ sudo docker push nbhat/puppet:debian-8-aarch64
  • Create bigtop-slaves image
Code Block
languagebash
$ cd  docker/bigtop-slaves/debian-8-aarch64
$ vim Dockerfile

Change the image fetching path in Dockerfile for example: "FROM bigtop/puppet:debian-8-aarch64" to "FROM nbhat/puppet:debian-8-aarch64" 
Note that the image must be available on my area at https://hub.docker.com/  

$ cd bigtop-trunk
$ git branch
* erp17.08
  master
$ sudo ./gradlew -POS=debian-8-aarch64 -Pprefix=erp17.08 bigtop-slaves
$ sudo docker images
REPOSITORY          TAG                                        IMAGE ID            CREATED             SIZE
bigtop/slaves       erp17.08-debian-8-aarch64                  9d5d71be2c8b        2 minutes ago       2.804 GB
bigtop/puppet       debian-8-aarch64                           4b127acf355b        About an hour ago   278.8 MB
nbhat/puppet        debian-8-aarch64                           4b127acf355b        About an hour ago   278.8 MB
  • Check the task you want to build.
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew :task --all'
  • Basic packages are required for all big data components. They are: bigtop-groovy, bigtop-utils, bigtop-jsvc and bigtop-tomcat
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew bigtop-groovy-deb bigtop-utils-deb bigtop-jsvc-deb bigtop-tomcat-deb'
  • Build the target BigTop components like Hadoop, Hbase, Spark and Hive debian and packages
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-deb hbase-deb spark-deb hive-deb'
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-pkg hbase-pkg spark-pkg hive-pkg'
  • Build smoke test dependent packages
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew flume-deb mahout-deb pig-deb sqoop-deb zookeeper-deb'
  • Create a local repo
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew apt'
  • Using sandbox
Code Block
languagebash
$ cd docker/sandbox
If you are pointing to any other image e.g. the image located on your local git hub, you can edit the build script and change the "FROM bigtop/puppet:${OS}"
$ sudo ./build.sh -a bigtop -o debian-8-aarch64 -c "hadoop, spark, hive, hbase"
  • Using provisioner
Code Block
languagebash
$ cd provisioner/docker

Config the provisioner yaml for aarch64 (bigtop/provisioner/docker/config_debian-8-aarch64.yaml)：
image:  "bigtop/puppet:"      ----->      image:  "bigtop/puppet:debian-8-aarch64"
enable_local_repo: false      ------->     enable_local_repo: true

or if you have exported the image in your own docker hub area you can give that image name too.

$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./docker-hadoop.sh --create 5 --smoke-tests --destroy'

-#BigTop Docker provisioner

-
-## Overview
-
-The Vagrantfile definition and wrapper script that creates Bigtop virtual Hadoop cluster on top of Docker containers for you, by pulling from existing publishing bigtop repositories.
-This cluster can be used:
-
-- to test bigtop smoke tests
-- to test bigtop puppet recipes
-
-These containers start sshd daemons, which vagrant uses to provision and install the hadoop cluster.
-
-This has been verified on docker client 1.2.0, with api version 1.15, and vagrant 1.6.5 on Fedora 20 as well as Centos 6.
-
-## Prerequisites
-
-### OS X and Windows
-
-* Install [VirtualBox](https://www.virtualbox.org/wiki/Downloads)
-
-* Install [Vagrant](http://www.vagrantup.com/downloads.html). Need version 1.6.5 or higher.
-
-### Linux
-
-* [Kernel Requirements](http://docker.readthedocs.org/en/v0.5.3/installation/kernel/)
-
-* Install [Docker](https://docs.docker.com/installation/)
-
-* Install [Vagrant](http://www.vagrantup.com/downloads.html)
-
-## Getting Started
-
-* Create a 3 node Bigtop Hadoop cluster from scratch
-
-NOTE : SELinux can PREVENT you from ssh'ing into your docker container.
-As a brute force way to disable it  - remove it from vi /etc/sysconfig/dockerarguments
-(fedora and centos may by default launch docker daemon with the --selinux-enabled option)!
-In the future, lets update this README with the RIGHT way to allow selinux without breaking
-ssh into a docker container!
-
-```
-service docker restart
-docker pull bigtop/deploy:centos-6
-```
-Now, you can start your cluster:
-
-In case of errors you can attempt running as root, or else, ping the mailing list.
-
-## USAGE
-
-1) Create a Bigtop Hadoop cluster by given # of node. (will place a file called config.rb)
-
-```
-./docker-hadoop.sh --create 3
-```
-
-2) Destroy the cluster.
-
-```
-./docker-hadoop.sh --destroy
-```
-
-3) Update your cluster after doing configuration changes. (re-run puppet apply)
-
-```
-./docker-hadoop.sh --provision
-```
-
-4) Run Bigtop smoke tests
-
-```
-./docker-hadoop.sh --smoke-tests
-```
-
-5) Chain your operations with-in one command.
-
-```
-./docker-hadoop.sh --create 5 --smoke-tests --destroy
-```
-
-Commands will be executed by following order:
-
-```
-create 5 node cluster => run smoke tests => destroy the cluster
-```
-
-6) See helper message:
-
-```
-./docker-hadoop.sh -h
-usage: docker-hadoop.sh [options]
-       -c NUM_INSTANCES, --create NUM_INSTANCES  Create a docker based Bigtop Hadoop cluster
-       -p, --provision                           Deploy configuration changes
-       -s, --smoke-tests                         Run Bigtop smoke tests
-       -d, --destroy                             Destroy the cluster
-       -h, --help
-
-```
-
-##Configurations
-
-* There are several parameters can be configured in the vagrantconfig.yaml:
-
-1) Modify memory limit for Docker containers
-
-```
-docker:
-        memory_size: "2048"
-
-```
-
-2)  If you're running Docker provisioner on OS X or Windows, you can customize the boot2docker VM settings
-
-```
-boot2docker:
-        memory_size: "4096"
-        number_cpus: "2"
-```
-
-3) Use different host ports mapping for web UIs
-
-```
-namenode_ui_port: "50070"
-yarn_ui_port: "8088"
-hbase_ui_port: "60010"
-
-```
-Note: If running on OS X or Windows, the boot2docker VM should be reloaded after ports changed
-
-
-##Configure Apache Hadoop ecosystem components
-* Choose the ecosystem you want to be deployed by modifying components in vagrantconfig.yaml
-
-```
-components: "hadoop,hbase,yarn,..."
-```
-
-By default, Apache Hadoop, YARN, and Apache HBase will be installed.
-See `bigtop-deploy/puppet/config/site.csv.example` for more details.
-
-##Notes
-
-* Users currently using vagrant 1.6+ is strongly recommanded to upgrade to 1.6.4+, otherwise you will encounter the [issue](https://github.com/mitchellh/vagrant/issues/3769) when installing plugins
http://git-wip-us.apache.org/repos/asf/bigtop/blob/4ee8f075/bigtop-deploy/vm/vagrant-puppet-docker/Vagrantfile


--IN-PROGRESS---

Table of Contents

Introduction

...

 

List of Smoke Tests

...

Sources

Setup Environment

  • Debian Jessie 8 on AArch64
Code Block
languagebash
$ lsb_release -idrc
Distributor ID:    Debian
Description:       Debian GNU/Linux 8.8 (jessie)
Release:           8.8
Codename:          jessie
  • Misc software need tobe installed and check for sudo permission
Code Block
languagebash
# apt-get install vim git sudo
# visudo

Add the below line with your username

<username>   ALL=(ALL)   ALL

# push 'Ctrl + x' key to quit visudo

now log-in as username and check for the access

$ sudo ls /root
$ sudo apt-get update                                                           
$ sudo apt-get upgrade

Install Pre-Requisites

  • Open JDK
Code Block
languagebash
$ sudo apt-get install -t jessie-backports openjdk-8-jre-headless
$ sudo apt-get install openssh-server openssh-clients java-1.8.0-openjdk*
$ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-arm64$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
  • Docker
Code Block
languagebash
$ sudo apt-get install docker docker.io
$ docker --version
Docker version 1.12.6, build 78d1802
  • Docker-compose
Code Block
languagebash
$ curl -L --fail https://github.com/docker/compose/releases/download/1.14.0/run.sh > /usr/local/bin/docker-compose

$ sudo chmod +x /usr/local/bin/docker-compose
$ docker-compose --version
docker-compose version 1.14.0, build c7bdf9e

Note that if you hit an error "Not found" when you check for docker-compose version.  Just clone docker repository and copy the docker-composer 
binary under /usr/local/bin/docker-compose

$ git clone https://github.com/docker/compose.git
$ cd compose
$ sudo cp ./bin/docker-compose /usr/local/bin/docker-compose
  • Maven
Code Block
languagebash
$ sudo apt-get install maven
$ mvn -version
Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-arm64/jre
Default locale: en_IN, platform encoding: UTF-8
OS name: "linux", version: "4.12.0-219-arm64", arch: "aarch64", family: "unix"
$


Smoke Test Components

  • Hadoop
  • Hive
  • Hbase
  • Spark

Test Steps

Install Pre-requisities

  • Clone the bigtop-trunk git repository and branch out to erp17.08
Code Block
languagebash
$ git clone https://git.linaro.org/leg/bigdata/bigtop-trunk.git
$ cd bigtop-trunk
$ git branch erp17.08 origin/erp17.08
$ git checkout erp17.08
  • Generate a docker image using build script in docker-puppet.  This will create a docker image from aarch64/debian:jessie
Code Block
languagebash
$ cd docker/bigtop-puppet/debian-8-aarch64
$ sudo ./build.sh
$ sudo docker images
REPOSITORY          TAG                   IMAGE ID            CREATED             SIZE
bigtop/puppet       debian-8-aarch64     4b127acf355b        3 minutes ago       278.8 MB

$ cd <BIGTOP_SRC_TOP>
$ docker build -t bigtop/slaves:erp17.08-debian-8-aarch64 -f docker/bigtop-slaves/debian-8-aarch64/Dockerfile .
  • Basic packages are required for all big data components. They are: bigtop-groovy, bigtop-utils, bigtop-jsvc and bigtop-tomcat
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew bigtop-groovy-deb bigtop-utils-deb bigtop-jsvc-deb bigtop-tomcat-deb'
  • Build the target BigTop components like Hadoop, Hbase, Spark and Hive debian and packages
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-deb hbase-deb spark-deb hive-deb'
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew hadoop-pkg hbase-pkg spark-pkg hive-pkg'
  • Build smoke test dependent packages
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew flume-deb mahout-deb pig-deb sqoop-deb zookeeper-deb'
  • Create a local repo
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew apt'

Create smoke test configuration file.

Bigtop smoke test uses yaml to configure test repo/environment/components. A typical configuration yaml can be found at <BIGTOP_SRC_TOP>/provisioner/docker/erp17.08-debian-8-aarch64

Generally following properties should be customized for certain smoke test:

  • docker image
  • repo url
  • distro type
  • components to install
  • components to test

Following is configuration yaml for Hive:

Code Block
languagebash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

docker:
        memory_limit: "4g"
        image:  "bigtop/puppet:debian-8-aarch64"

repo: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/debian/8-aarch64/aarch64"
distro: debian
components: [hdfs, yarn, mapreduce, hive]
enable_local_repo: true
smoke_test_components: [hive]
jdk: "openjdk-8-jdk"

Execute smoke tests

It is quite straight forward to execute smoke in bigtop

Code Block
languagebash
cd provisioner/docker
./docker-hadoop.sh -C <smoke_test_cfg_yaml> -c <node_count> -s -d

Before running smoke test, it is suggested to run environment check first

Code Block
languagebash
./docker-hadoop.sh -E

For detailed usage, just issue: ./docker-hadoop.sh

Code Block
languagebash
usage: docker-hadoop.sh [-C file ] args
       -C file                                   Use alternate file for config.yaml
  commands:
       -c NUM_INSTANCES, --create NUM_INSTANCES  Create a Docker based Bigtop Hadoop cluster
       -d, --destroy                             Destroy the cluster
       -e, --exec INSTANCE_NO|INSTANCE_NAME      Execute command on a specific instance. Instance can be specified by name or number.
                                                 For example: docker-hadoop.sh --exec 1 bash
                                                              docker-hadoop.sh --exec docker_bigtop_1 bash
       -E, --env-check                           Check whether required tools has been installed
       -l, --list                                List out container status for the cluster
       -p, --provision                           Deploy configuration changes
       -s, --smoke-tests                         Run Bigtop smoke tests
       -h, --help


  • Using sandbox
Code Block
languagebash
$ cd docker/sandbox
If you are pointing to any other image e.g. the image located on your local git hub, you can edit the build script and change the "FROM bigtop/puppet:${OS}"
$ sudo ./build.sh -a bigtop -o debian-8-aarch64 -c "hadoop, spark, hive, hbase"


  • How to Tag and Export bigtop-puppet docker image in your own area.  You need to create an account on https://hub.docker.com/ following are the example commands
Code Block
languagebash
$ sudo docker tag 4b127acf355b nbhat/puppet:debian-8-aarch64
$ sudo docker login 
$ sudo docker push nbhat/puppet:debian-8-aarch64
  • How to create bigtop-slaves image using the image in your area
Code Block
languagebash
$ cd  docker/bigtop-slaves/debian-8-aarch64
$ vim Dockerfile

Change the image fetching path in Dockerfile for example: "FROM bigtop/puppet:debian-8-aarch64" to "FROM nbhat/puppet:debian-8-aarch64" 
Note that the image must be available on my area at https://hub.docker.com/  

$ cd bigtop-trunk
$ git branch
* erp17.08
  master
$ sudo ./gradlew -POS=debian-8-aarch64 -Pprefix=erp17.08 bigtop-slaves
$ sudo docker images
REPOSITORY          TAG                                        IMAGE ID            CREATED             SIZE
bigtop/slaves       erp17.08-debian-8-aarch64                  9d5d71be2c8b        2 minutes ago       2.804 GB
bigtop/puppet       debian-8-aarch64                           4b127acf355b        About an hour ago   278.8 MB
nbhat/puppet        debian-8-aarch64                           4b127acf355b        About an hour ago   278.8 MB
  • Check the task you want to build.
Code Block
languagebash
$ sudo docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew :task --all'


  • Patches

Upstream and Resolved:

https://issues.apache.org/jira/browse/BIGTOP-2838
https://issues.apache.org/jira/browse/BIGTOP-2843

https://issues.apache.org/jira/browse/BIGTOP-2841

https://issues.apache.org/jira/browse/BIGTOP-2851

https://issues.apache.org/jira/browse/BIGTOP-2852

https://issues.apache.org/jira/browse/BIGTOP-2853

Patches under discussion:

https://issues.apache.org/jira/browse/BIGTOP-2856

https://issues.apache.org/jira/browse/BIGTOP-2854