Build and Test instructions of BigData Apache components
- Former user (Deleted)
Owned by Former user (Deleted)
This post explains about how to build different BigData packages like Flink, Memcached, Kafka, Oozie, Kafka Streams, and Cassandra. You need to install Maven v3.3.9 and Open JDK8u
About BigData components
There are many BigData components individually
Build and Test Instructions
Install OpenJDK 8
$ wget http://openjdk.linaro.org/releases/jdk8u-server-release-1708.tar.xz $ tar xvf jdk8u-server-release-1708.tar.xz $ cd jdk8u-server-release-1708 $ export JAVA_HOME=$PWD $ cd jre/lib/security/ $ rm cacerts (for CentOS 7) $ ln --symbolic /etc/pki/java/cacerts . (for Debian Jessie) $ ln --symbolic /etc/ssl/certs/java/cacerts . $ cd jdk8u-server-release-1708/bin $ export PATH=$PWD:$PATH
Install Maven v3.3.9
Download and export the Maven v3.3.9
$ wget http://www.us.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz $ tar -xvf apache-maven-3.3.9-bin.tar.gz $ cd apache-maven-3.3.9/bin $ export PATH=$PWD:$PATH
Flink
The flink package build instructions are LEG-250
$ git clone https://github.com/apache/flink.git $ cd flink $ mvn clean package -DskipTests
Kafka
The kafka package build instructions are LEG-252 You need to install "gradle" before executing any other instructions. The gradle install instructions are purely based on your distribution.
$ git clone https://github.com/apache/kafka.git $ cd kafka $ ./gradlew jar
Oozie
The oozie package build instructions are LEG-276
$ git clone https://github.com/apache/oozie.git $ cd oozie $ mvn clean package -DskipTests $ cd minitest $ mvn clean test
Nifi
The nifi package build instructions are LEG-677
$ git clone https://github.com/apache/nifi.git $ cd nifi $ mvn clean package -DskipTests Testing: ------- nareshbhat@debian:~/nifi$ cd nifi-assembly/ nareshbhat@debian:~/nifi/nifi-assembly$ ls -lhd target/ drwxr-xr-x 5 nareshbhat nareshbhat 4.0K Feb 16 15:37 target/ nareshbhat@debian:~/nifi/nifi-assembly$ ls -lhd target/nifi* drwxr-xr-x 3 nareshbhat nareshbhat 4.0K Feb 16 15:36 target/nifi-1.6.0-SNAPSHOT-bin rw-rr- 1 nareshbhat nareshbhat 1.2G Feb 16 15:38 target/nifi-1.6.0-SNAPSHOT-bin.tar.gz rw-rr- 1 nareshbhat nareshbhat 1.2G Feb 16 15:37 target/nifi-1.6.0-SNAPSHOT-bin.zip nareshbhat@debian:~/nifi/nifi-assembly$ mkdir ~/example-nifi-deploy nareshbhat@debian:~/nifi/nifi-assembly$ tar xzf target/nifi-*-bin.tar.gz -C ~/example-nifi-deploy nareshbhat@debian:~/nifi/nifi-assembly$ ls -lh ~/example-nifi-deploy/ total 4.0K drwxr-xr-x 6 nareshbhat nareshbhat 4.0K Feb 16 15:49 nifi-1.6.0-SNAPSHOT nareshbhat@debian:~/nifi/nifi-assembly$ cd ~/example-nifi-deploy/nifi-* nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$ ./bin/nifi.sh start Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708 NiFi home: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT Bootstrap Config File: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT/conf/bootstrap.conf nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$ I could able to see the NiFi screen shot using web browser (Attached). I did't use the https://nifi.apache.org/docs/nifi-docs/html/getting-started.html guide to very extended level. But I just make sure that the basic things works fine. nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$ ./bin/nifi.sh stop Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708 NiFi home: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT Bootstrap Config File: /home/nareshbhat/example-nifi-deploy/nifi-1.6.0-SNAPSHOT/conf/bootstrap.conf 2018-02-16 16:02:54,540 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi has accepted the Shutdown Command and is shutting down now 2018-02-16 16:02:54,575 INFO [main] org.apache.nifi.bootstrap.Command Waiting for Apache NiFi to finish shutting down... 2018-02-16 16:02:56,602 INFO [main] org.apache.nifi.bootstrap.Command NiFi has finished shutting down. nareshbhat@debian:~/example-nifi-deploy/nifi-1.6.0-SNAPSHOT$
Mini NiFi
The mini-nifi package build instructions are in Jira issue
$ git clone https://github.com/apache/nifi-minifi.git $ cd nifi-minifi/ $ mvn clean package -DskipTests nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi$ cd minifi-assembly nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ ls -lhd target/minifi* drwxr-xr-x 3 nareshbhat nareshbhat 4.0K Feb 27 11:27 target/minifi-0.5.0-SNAPSHOT-bin -rw-r--r-- 1 nareshbhat nareshbhat 65M Feb 27 11:27 target/minifi-0.5.0-SNAPSHOT-bin.tar.gz -rw-r--r-- 1 nareshbhat nareshbhat 65M Feb 27 11:27 target/minifi-0.5.0-SNAPSHOT-bin.zip -rw-r--r-- 1 nareshbhat nareshbhat 112K Feb 27 11:26 target/minifi.exe -rw-r--r-- 1 nareshbhat nareshbhat 116K Feb 27 11:26 target/minifiw.exe nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ mkdir ~/example-minifi-deploy nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ tar xzf target/minifi-*-bin.tar.gz -C ~/example-minifi-deploy/ nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ ls -lh ~/example-minifi-deploy/ total 4.0K drwxr-xr-x 6 nareshbhat nareshbhat 4.0K Feb 27 11:29 minifi-0.5.0-SNAPSHOT nareshbhat@debian:~/Apache-packages-compilation/test-jdk8u/nifi-minifi/minifi-assembly$ cd ~/example-minifi-deploy/minifi-* nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$ ./bin/minifi.sh start Bootstrap Classpath: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/bootstrap/*:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/* Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708 MiNiFi home: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT Bootstrap Config File: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf/bootstrap.conf 2018-02-27 11:29:49,626 INFO [main] org.apache.nifi.minifi.bootstrap.Command Starting Apache MiNiFi... 2018-02-27 11:29:49,629 INFO [main] org.apache.nifi.minifi.bootstrap.Command Working Directory: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT 2018-02-27 11:29:49,631 INFO [main] org.apache.nifi.minifi.bootstrap.Command Command: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708/bin/java -classpath /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./conf:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-servlets-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/zookeeper-3.4.6.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-schema-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/osgi-resource-locator-1.0.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/quartz-2.2.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpcore-nio-4.4.5.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-client-dto-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-webapp-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/hk2-locator-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-server-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpclient-4.5.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet.jsp-api-2.3.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-authorizer-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-api-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-nar-utils-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/ecj-4.4.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-data-provenance-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-nar-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-framework-core-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-annotations-2.6.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-runtime-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-schemas-3.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/guava-18.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-write-ahead-log-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-xml-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.el-3.0.1-b09.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-codec-1.10.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-collections4-4.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-mapper-asl-1.9.13.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-entity-filtering-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpasyncclient-4.1.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-hk2-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-repository-models-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet.jsp.jstl-api-1.2.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-site-to-site-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/curator-client-2.11.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-api-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-core-2.6.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-cluster-protocol-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-security-utils-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jul-to-slf4j-1.7.25.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-data-model-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/curator-recipes-2.11.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/lucene-analyzers-common-4.10.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-socket-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/httpcore-4.4.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-common-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-client-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/logback-core-1.2.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-registry-flow-diff-0.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/bcprov-jdk15on-1.55.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-user-actions-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/lucene-core-4.10.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-administration-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.ws.rs-api-2.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jline-0.9.94.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-databind-2.6.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/validation-api-2.0.0.Final.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-security-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/asm-1.0.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.inject-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-util-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-security-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-authorization-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-core-api-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/apache-el-8.0.33.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-properties-loader-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-continuation-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-utils-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-io-2.5.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-io-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-logging-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-framework-core-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/antlr-runtime-3.5.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-media-json-jackson-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/hk2-api-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jersey-client-2.26.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/bcpkix-jdk15on-1.55.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jcl-over-slf4j-1.7.25.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-site-to-site-client-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-properties-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.inject-1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/logback-classic-1.2.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-web-utils-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet.jsp-2.3.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-jsp-jdt-2.3.3.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-servlet-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.servlet-api-3.1.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-persistent-provenance-repository-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-module-jaxb-annotations-2.8.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-flowfile-repo-serialization-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.annotation-api-1.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/spring-core-4.2.4.RELEASE.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/aopalliance-repackaged-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/asm-3.3.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/curator-framework-2.11.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/apache-jsp-9.3.9.v20160517.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javassist-3.22.0-CR2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/slf4j-api-1.7.25.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-lang3-3.4.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/json-smart-2.1.1.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/json-path-2.0.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/minifi-runtime-0.5.0-SNAPSHOT.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/apache-jsp-8.0.33.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/javax.el-api-3.0.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-expression-language-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/commons-logging-1.2.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/nifi-security-1.5.0.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/hk2-utils-2.5.0-b42.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/netty-3.7.0.Final.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jetty-http-9.4.3.v20170317.jar:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./lib/jackson-core-asl-1.9.13.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx256m -Xms256m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.security.egd=file:/dev/urandom -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/./conf/nifi.properties -Dnifi.bootstrap.listen.port=35271 -Dapp=MiNiFi -Dorg.apache.nifi.minifi.bootstrap.config.log.dir=/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/logs org.apache.nifi.minifi.MiNiFi nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$ cd ~/example-minifi-deploy/minifi-* nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$ ./bin/minifi.sh stop Bootstrap Classpath: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/bootstrap/*:/home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/lib/* Java home: /home/nareshbhat/Apache-packages-compilation/JDK/jdk8u-server-release-1708 MiNiFi home: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT Bootstrap Config File: /home/nareshbhat/example-minifi-deploy/minifi-0.5.0-SNAPSHOT/conf/bootstrap.conf 2018-02-27 11:30:57,260 INFO [main] org.apache.nifi.minifi.bootstrap.Command Apache MiNiFi has accepted the Shutdown Command and is shutting down now 2018-02-27 11:30:57,368 INFO [main] org.apache.nifi.minifi.bootstrap.Command MiNiFi has finished shutting down. nareshbhat@debian:~/example-minifi-deploy/minifi-0.5.0-SNAPSHOT$
Memcached
The memcached package build instructions are LEG-686
Install dependency package $ sudo apt-get install libevent-dev $ sudo apt-get update $ sudo apt-get install mysql-server php5-mysql php5 php5-memcached Clone and build memcached $ git clone https://github.com/memcached/memcached.git $ cd memcached $ ./autogen.sh $ ./configure $ make all You can run the testapp $ ./testapp You can check memcached version and run $ ./memcached -V memcached 1.5.4 I have started the memcached service on on console with below command nareshbhat@debian:~/memcached$ memcached start and test it on another console nareshbhat@debian:~$ sudo netstat -tupan | grep 11211 tcp 0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 20423/memcached tcp6 0 0 :::11211 :::* LISTEN 20423/memcached udp 0 0 0.0.0.0:11211 0.0.0.0:* 20423/memcached udp6 0 0 :::11211 :::* 20423/memcached nareshbhat@debian:~$ To verify memcached is recognized by the web server: 1. Create a phpinfo.php file in the web server’s docroot: $ sudo nano /var/www/html/phpinfo.php =========phpinfo.php start============ <?php // Show all information, defaults to INFO_ALL phpinfo(); =========phpinfo.php end============ 2. Go to that page in your web browser. For example, http://<IP-address>/phpinfo.php 3. Make sure memcached displays all the information on web browser I have also executed database_test.php, cache_test.php
Cassandra
The cassandra package build instructions are LEG-695 You have to install ant tool to build cassandra.
$ git clone https://github.com/apache/cassandra.git
$ cd cassandra
$ git tag
$ git checkout cassandra-3.11.1
$ ant clean
$ ant
After successful build you can run the cassandra using command
$ ./bin/cassandra -f
On another terminal I run the below example
$ bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.1-SNAPSHOT | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE SCHEMA schema1
... WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
cqlsh> USE schema1;
cqlsh:schema1> CREATE TABLE users (
... user_id varchar PRIMARY KEY,
... first varchar,
... last varchar,
... age int
... );
cqlsh:schema1> INSERT INTO users (user_id, first, last, age)
... VALUES ('jsmith', 'John', 'Smith', 42);
cqlsh:schema1> SELECT * FROM users;
user_id | age | first | last
---------+-----+-------+-------
jsmith | 42 | John | Smith
(1 rows)
cqlsh:schema1>
Difference between NiFi and Mini-NiFi
Apache NiFi is a robust and secure framework for routing, transforming, and delivering data across a multitude of systems. NiFi can run in parallel with other applications, but it performs best when the entire system (or multiple systems in a cluster) are dedicated to it. It often uses SAN or RAID storage at the TB level for the massive amounts of content it ingests and the provenance it generates. The UI allows multiple users to quickly modify flows simultaneously on the same machine or across a cluster. The latest release candidate of NiFi (1.1.0 RC1) includes over 170 processors for custom integration with various systems and operations, and is 762 MB when compressed for download. In other words, NiFi is a server-class application.
Apache MiNiFi was developed out of a recognized need to bring the capabilities of NiFi to the "edge" as "agents" -- accessing data from IoT and desktop-level devices, and applying primary features of NiFi at the earliest possible stage. Now data can be collected from a variety of protocols, have data provenance generated immediately for more holistic governance and transparency, have light transformations applied at source, be encrypted, be prioritized, and be redundantly routed back to the more powerful transformations done in the cloud or data center.
Now, all of these behaviors can be performed with custom scripts, but then the problem of command & control (C2) is encountered. With hundreds, thousands, or even millions of these devices existing, how can each be monitored and exfilled, and what happens when the flow needs to change? It could be to report back to a new endpoint, to update the frequency at which it is collected or transmitted, or to handle new metrics or metadata from the device. This manual process does not scale. With MiNiFi's integration with NiFi, a flow can be developed using the UI in NiFi and transparently translated to a MiNiFi flow and pushed out to classes of agents across the world.
With manual modification to remove unnecessary processors and features, NiFi can be trimmed to fit on a Raspberry Pi. But it still requires the JVM, and there are plenty of devices that won't support it. MiNiFi is offered in Java and C++, and the footprint is on a completely different scale -- 39 MBfor the Java agent (tar) and 310K for the C++ agent (tar).
A great example of the power and usefulness of MiNiFi is a recent demo at the TU-Automotive Detroit exhibition, where MiNiFi was loaded onto a custom Qualcomm modem located in a "connected car". As the car drives, massive amounts of data are generated by components throughout the car and routed via the CANBUS to be processed. Some data is important to stream back to a remote processing center in real-time -- this data is transmitted via an LTE connection. LTE is widely available, but bandwidth is expensive. Meanwhile, data that was much larger but less time-relevant (system diagnostics, etc.) could be batched and compressed, and then sent in bursts over WiFi when the car was in range of a known hotspot. MiNiFi coordinated all of the flow decisions and routing via geo-enrichment and control plane feedback. Here is a short video of Joe Niemiec explaining the process and showing the flow.
You can extrapolate that demo to many other use cases. It is helpful to think of MiNiFi as a "good guest" -- a lightweight agent that runs on hardware that is probably dedicated to a different primary purpose. Whether this is IoT, a cash register/point of sale system, a car modem, physical sensors, etc., is irrelevant to MiNiFi -- its job is to process and exfil this data while not taking unnecessary resources from the primary function. Contrast this with NiFi, which again, can run simultaneously with other applications, but ideally it has dedicated resources which it can maximize for its own performance.
Reference :
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-memcache-on-ubuntu-14-04
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-memcache-on-ubuntu-14-04
http://devdocs.magento.com/guides/v2.0/config-guide/memcache/memcache_ubuntu.html
https://github.com/linux-on-ibm-z/docs/wiki/Building-Apache-Cassandra-3.11
https://github.com/apache/cassandra