.Build Apache Drill v1.0

Pre-requisites

OpenJDK8
Zookeeper

Install OpenJDK

$ sudo apt-get install openjdk-8-jdk

Make sure you have the right OpenJDK version

$ java -version

It should display 1.8.0_111

Set JAVA_HOME

$ export JAVA_HOME=`readlink -f /usr/bin/java | sed "s:jre/bin/java::"`

Building Apache Zookeeper

Download and unzip the Zookeeper package from Official Apache archive in all machines that will be used for zookeeper quorum as shown below:

$ wget https://www-us.apache.org/dist/zookeeper/stable/zookeeper-3.4.12.tar.gz

$ tar -xzvf zookeeper-3.4.12.tar.gz

Edit the /etc/hosts file across all the nodes and add the ipaddress and hostname (nodenames). If the hostnames are not right, change them in /etc/hosts file

192.168.1.102 node1
192.168.1.103 node2
192.168.1.105 node3

Create zookeeper user

$ sudo adduser zookeeper

Configure zookeeper

Now, Create the directory zookeeper under /var/lib folder which will serve as Zookeeper data directory and create another zookeeper directory under /var/log where all the Zookeeper logs will be captured. Both of the directory ownership need to be changed as zookeeper.

$ sudo mkdir /var/lib/zookeeper

$ cd /var/lib

$ sudo chown zookeeper:zookeeper zookeeper/

$ sudo mkdir /var/log/zookeeper

$ cd /var/log

$ sudo chown zookeeper:zookeeper zookeeper/

Edit the bashrc for the zookeeper user via setting up the following Zookeeper environment variables.

$ export ZOO_LOG_DIR=/var/log/zookeeper

Source the .bashrc in current login session:

$ source ~/.bashrc

Create the server id for the ensemble. Each zookeeper server should have a unique number in the myid file within the ensemble and should have a value between 1 and 255.

In Node1

$ sudo sh -c "echo '1' > /var/lib/zookeeper/myid"

In Node2

$ sudo sh -c "echo '2' > /var/lib/zookeeper/myid"

In Node3

$ sudo sh -c "echo '3' > /var/lib/zookeeper/myid"

Now, go to the conf folder under the Zookeeper home directory (location of the Zookeeper directory after Archive has been unzipped/extracted).

$ cd /home/zookeeper/zookeeper-3.4.13/conf/

By default, a sample conf file with name zoo_sample.cfg will be present in conf directory. Make a copy of it with name zoo.cfg as shown below, and edit new zoo.cfg as described across all the nodes.

$ cp zoo_sample.cfg zoo.cfg

Edit zoo.cfg and the below

$ vi zoo.cfg

dataDir=/var/lib/zookeeper
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

Now, do the below changes in log4.properties file as follows.

$ vi log4j.properties

zookeeper.log.dir=/var/log/zookeeper

zookeeper.tracelog.dir=/var/log/zookeeper

log4j.rootLogger=INFO, CONSOLE, ROLLINGFILE

After the configuration has been done in zoo.cfg file in all three nodes, start zookeeper in all the nodes one by one, using following command:

$ /home/zookeeper/zookeeper-3.4.12/bin/zkServer.sh start

Zookeeper Service Start on all the Nodes.

ZooKeeper JMX enabled by default
Using config: /home/ganesh/zookeeper-3.4.12/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

The log file will be created in /var/log/zookeeper of zookeeper named zookeeper.log, tail the file to see logs for any errors.

$ tail -f /var/log/zookeeper/zookeeper.log

Verify the Zookeeper Cluster and Ensemble

In Zookeeper ensemble out of three servers, one will be in leader mode and other two will be in follower mode. You can check the status by running the following commands.

$ /home/zookeeper/zookeeper-3.4.13/bin/zkServer.sh status

Zookeeper Service Status Check.

In Zookeeper ensemble If you have 3 nodes, out of them, one will be in leader mode and other two will be in follower mode. You can check the status by running the following commands. If you have just one then it will be standalone.

With three nodes:

node1

ZooKeeper JMX enabled by default
Using config: /home/zookeeper/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: leader

node2

ZooKeeper JMX enabled by default
Using config: /home/zookeeper/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: follower

node3

ZooKeeper JMX enabled by default
Using config: /home/zookeeper/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: follower

standalone

ZooKeeper JMX enabled by default
Using config: /home/zookeeper/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: standalone

$ echo stat | nc node1 2181

Lists brief details for the server and connected clients.

$ echo mntr | nc node1 2181

Zookeeper list of variables for cluster health monitoring.

$ echo srvr | nc localhost 2181

Lists full details for the Zookeeper server.

If you need to check and see the znode, you can connect by using the below command on any of the zookeeper node:

$ /home/zookeeper/zookeeper-3.4.12/bin/zkCli.sh -server `hostname -f`:2181

Connect to Zookeeper data node and lists the contents.

ent:user.name=root
2019-02-18 02:26:36,822 [myid:] - INFO [main:Environment@100] - Client environm
ent:user.home=/root
2019-02-18 02:26:36,822 [myid:] - INFO [main:Environment@100] - Client environm
ent:user.dir=/home/ganesh
2019-02-18 02:26:36,823 [myid:] - INFO [main:ZooKeeper@441] - Initiating client
connection, connectString=:2181 sessionTimeout=30000 watcher=org.apache.zookeep
er.ZooKeeperMain$MyWatcher@4b9af9a9
Welcome to ZooKeeper!
2019-02-18 02:26:36,846 [myid:] - INFO [main-SendThread(localhost:2181):ClientC
nxn$SendThread@1028] - Opening socket connection to server localhost/127.0.0.1:2
181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-02-18 02:26:36,927 [myid:] - INFO [main-SendThread(localhost:2181):ClientC
nxn$SendThread@878] - Socket connection established to localhost/127.0.0.1:2181,
initiating session
2019-02-18 02:26:36,948 [myid:] - INFO [main-SendThread(localhost:2181):ClientC
nxn$SendThread@1302] - Session establishment complete on server localhost/127.0.
0.1:2181, sessionid = 0x1000001cfe00002, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: :2181(CONNECTED) 0]