Bigtop SmokeTest

Owned by Jun He (Deactivated)
Last updated: Aug 04, 2017
3 min read

Smoke Tests

Setup repo

Bigtop release repo doesn't have arm64 support. To execute smoke test it is required to setup local repo with packages we build out.

docker run -v `pwd`:/ws bigtop/slaves:erp17.08-debian-8-aarch64 bash -l -c 'cd /ws ; ./gradlew apt'

Create test configuration

Bigtop smoke test uses yaml to configure test repo/environment/components. A typical configuration yaml can be found at <BIGTOP_SRC_TOP>/provisioner/docker/erp17.08-debian-8-aarch64

Generally following properties should be customized for certain smoke test:

  • docker image
  • repo url
  • distro type
  • components to install
  • components to test

Following is configuration yaml for Hive:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

docker:
        memory_limit: "4g"
        image:  "bigtop/puppet:debian-8-aarch64"

repo: "http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/debian/8-aarch64/aarch64"
distro: debian
components: [hdfs, yarn, mapreduce, hive]
enable_local_repo: true
smoke_test_components: [hive]
jdk: "openjdk-8-jdk"

Execute test

It is quite straight forward to execute smoke in bigtop

cd provisioner/docker
./docker-hadoop.sh -C <smoke_test_cfg_yaml> -c <node_count> -s -d

Before running smoke test, it is suggested to run environment check first

./docker-hadoop.sh -E

For detailed usage, just issue: ./docker-hadoop.sh

usage: docker-hadoop.sh [-C file ] args
       -C file                                   Use alternate file for config.yaml
  commands:
       -c NUM_INSTANCES, --create NUM_INSTANCES  Create a Docker based Bigtop Hadoop cluster
       -d, --destroy                             Destroy the cluster
       -e, --exec INSTANCE_NO|INSTANCE_NAME      Execute command on a specific instance. Instance can be specified by name or number.
                                                 For example: docker-hadoop.sh --exec 1 bash
                                                              docker-hadoop.sh --exec docker_bigtop_1 bash
       -E, --env-check                           Check whether required tools has been installed
       -l, --list                                List out container status for the cluster
       -p, --provision                           Deploy configuration changes
       -s, --smoke-tests                         Run Bigtop smoke tests
       -h, --help