ERP 

• ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0 and Hive 2.0.1

• ERP 17.08 Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

• ERP 17.08 - Build, Configure, Install and Test Hadoop, Spark, Hive, Hbase in containerized environment

• ERP 18.06 Building and testing BigData components using Bigtop on Debian-9:AArch64

• Resources

  • ERP 17.08 deb packages

Big Data Components

Apache Bigtop

...

BigData components

ODPI and Apache BigTop

Build

Building BigTop using Docker container

/wiki/spaces/BDTS/pages/20384022610

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi BigTop Hadoop Installation, setup and running]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPiHadoopMultinodeClusterSetup]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run/ODPi_BigTop_Building]

Install

ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0

...

and Hive 2.0.1

...

...

Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

Tests

...

Smoke Tests

...

Resources

• Bigtop 1.2 Components

• Bigtop Docker images

• Upstream github

• CI Loop

• v1.2 Release
• Bigtop AArch64 patches
• Bigtop Project Ideas and Roadmap

...

ODPi

• Building ODPi BigTop packages
• Setup, Configure and Install ODPi Hadoop
• ODPi Hadoop Cluster Setup Guide
• ODPi BigTop Hadoop Installation, setup and running, ODPi BigTop Hadoop Installation, setup and running

...

Big Data Core Components

• Apache Hadoop

  • Setup, Configure and Install ODPi Hadoop
  • Building and Running Apache Hadoop
  • Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache Bigtop

  • Apache Hadoop Build, Install, Configure and Run Guide

  • Building, Running, Configuring and Profiling Apache Hadoop
  • Apache Hadoop Tuning Notes
  • Apache Hadoop Map Reduce Notes
  • OpenJDK javac Nullpointerexception building Hadoop
  • Patch 1/3 Introduce the HyperCrc32C Checksum class
  • Patch 2/3 libhadoop: CRC: ARM NEON Support
  • Patch 3/3 ModifyCRC to target NEON routine
  • hadoop-lca13

• ELK - ElasticSearch, Logstash and Kibana

  • Building ELK (ElasticSearch, LogStash and Kibana) on AArch64

  • ELK Setup and Run on AArch64

• Apache Sqoop

• Apache Arrow

...

Big Data Operations

• Apache Ambari

  • Build Apache Ambari on AArch64
  • Build Apache Ambari on AArch64 (Copy)
  • Install, Setup and Configure Apache Ambari
  • JVM Null pointer error in javac when building Apache Ambari

• Apache Zookeeper

  • Enablement of Apache Zookeeper on AArch64

• Apache Oozie

• Apache Falcon

• Ganglia

Big Data Streaming Tools

Apache Spark

...

Bigtop Smoke Tests

Integration Tests

ODPi Spec Tests

Issues and Resolutions

Apache Ambari

Build

Build Apache Ambari on AArch64

Build and Install Apache Ambari V2.6.1 on AArch64

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-ambari]

Install

Apache Ambari Install, Setup and Configuration

Apache Hadoop

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopBuildInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun]

Install

Apache Spark

Build

Spark 2.0 - Build, configure and Installation steps using Apache

...

Profiling Spark on AArch64

...

Building Apache Spark 1.6 on AArch64

...

Apache Flink

...

Apache Beam

...

Apache Tez

...

Apache Flume

...

Apache Storm

...

Apache Tachyon

...

Apache Kafka

• Apache Kafka Streams

...

Apache NiFi

• Difference between NiFi and MiNiFi

...

Apache MiNiFi

...

Big Data Data warehousing and Database Tools

• Apache Hive

  • Building Apache Hive

• Apache HBase

  • Enablement of Apache HBase on AArch64

• Apache Cassandra

• Postrgres

• Memcached

• MySQL

• Redis

• Apache Drill

...

Big Data Data Governance and Security

• Apache Ranger

• Apache Knox

• Apache Atlas

• Apache Sentry

...

Big Data File Formats

• Apache Parquet

• Apache Avro

...

Big Data Datascience Notebooks

• Apache Jupyter

• Apache Zeppelin

...

Big Data Analytics

• Apache Kudu

...

Big Data ML - Machine Learning

• H20

  • H20 Install and Run Guide

  • H20 Scaling Study
  • Sparkling Water - Spark + H20

...

Big Data component dependencies

• NETLIB-JAVA AArch64 Natives Support

Tests 

• Smoke Tests

  • BigTop smoke tests 

• Integration Tests

• ODPi Spec Tests

Benchmarking

• TeraSort
• Building, Running, Configuring and Profiling Apache Hadoop
• Spark Bench
• TPC-H
• TPCxHS
• Apache Bench
• BigBench
• Building and Running HiBench on AArch64 Platform
• HiveTestBench

• HiBench

  • Spark profiling on Arm64

Build and Port

• Build Apache Ambari on AArch64
• HBase Enablement on AArch64
• Apache Flink on AArch64
• Zookeeper Enablement on AArch64
• NETLIB-JAVA AArch64 Natives Support
• Apache Ambari Install, Setup and Configuration

Machine Learning

• Building and Installing Tensorflow on AArch64

Misc

...

Bigtop

Blogs/Presentations

State of Big Data on Aarch64 - Apache Bigtop

Big Data benchmarking

Himalayan Odyssey

Big Data Roadmap

Strategic Engineering

Big Data and OpenJDK Strategic Engineering - 2018

Big Data and OpenJDK Strategic Engineering - 2017

Big Data Epics

JIRA Inprogress

...

BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkDependencyLibraries]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_1_6]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_With_BigTop]

[https://wiki.linaro.org/LEG/Engineering/BigData/Spark]

Install

Apache Hive

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/Hive]

Install

Apache HBase

Build

HBase Enablement on AArch64

Install

Apache Zookeeper

Build

Zookeeper Enablement on AAarch64

ELK - ElasticSearch, Kibana and Logstash

Build

Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

Install

                              ELK Setup and Run on AArch64

Apache Kafka

H2O

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OScalingStudy]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparklingWaterGuide]

Benchmarks

[http://openjdk.linaro.org/hadoop-terasort-benchmark-results]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopconfig]

[https://wiki.linaro.org/Internal/People/SteveCapper/hadoop-lca13]

[https://wiki.linaro.org/LEG/Engineering/BigData/ApacheBench]

HiBench [wiki]   [Collaborate]

[https://wiki.linaro.org/LEG/Engineering/BigData/BigBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/HiveTestBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/TPCxHS]

[https://wiki.linaro.org/LEG/Engineering/BigData/TPC-H]

Workloads

DataScience

Guidelines

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopTuningGuide]

Misc

NETLIB-JAVA AArch64 Natives support 

[https://wiki.linaro.org/LEG/Engineering/BigData/MapReduceNotes ]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0001-Introduce-the-HyperCrc32C-Checksum-class.patch| PATCH 1/3 Introduce the HyperCrc32C Checksum class]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0002-libhadoop-CRC-ARM-NEON-support.patch| PATCH 2/3 libhadoop: CRC: ARM NEON support]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0003-Modify-HyperCRC-to-target-NEON-routine.patch| PATCH 3/3 Modify HyperCRC to target NEON routine]]

[https://wiki.linaro.org/LEG/Engineering/BigData/Feasibility]

[https://wiki.linaro.org/LEG/Engineering/BigData/CRC32vsNonCRC32Study]

[https://wiki.linaro.org/LEG/Engineering/BigData/configuring-archiva-with-tomcat7]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

== Other Big Data Components in Roadmap ==

=== Apache Flink ===

=== Apache Beam ===

=== Apache Oozie ===

=== Apache Flume ===

=== Apache Storm ===

=== Apache Tez ===

=== Apache Tachyon ===

...

== Big Data Operations ==

=== Apache Ambari ===

...

== Big Data - Data warehousing Tools ==

=== Apache Pig ===

=== Apache Cassandra ===

=== Apache Sqoop ===

...

== Analytical Tools, Machine Learning ==

=== H2O ===

...

== Big Data Governance and Security ==

=== Apache Ranger ===

=== Apache Knox ===

...

== Notebooks ==

=== Apache Jupyter ===

=== Apache Zeppelin ===

...

== File Formats ==

=== Apache Parquet ===

=== Apache Avro ===

...

== Functional, Regression and Workload Testing ==