ERP 

• ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0 and Hive 2.0.1

• ERP 17.08 Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

• ERP 17.08 - Build, Configure, Install and Test Hadoop, Spark, Hive, Hbase in containerized environment

• ERP 18.06 Building and testing BigData components

...

ODPI and Apache BigTop

...

• using Bigtop on Debian-9:AArch64

• Resources

  • ERP 17.08 deb packages

Big Data Components

1. Apache Bigtop

   • Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop
   • Building BigTop using Docker container

...

/wiki/spaces/BDTS/pages/20384022610

...

1. • Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64
   • Bigtop Smoke Tests

   • Resources

     • Bigtop 1.2 Components

     • Bigtop Docker images

     • Upstream github

     • CI Loop

     • v1.2 Release
     • Bigtop AArch64 patches
     • Bigtop Project Ideas and Roadmap

2. ODPi

   • Building ODPi BigTop packages
   • Setup, Configure and Install ODPi Hadoop
   • ODPi Hadoop Cluster Setup Guide
   • ODPi BigTop Hadoop Installation, setup and running, ODPi BigTop Hadoop Installation, setup and running

...

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPiHadoopMultinodeClusterSetup]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run/ODPi_BigTop_Building]

Install

...

2. Big Data Core Components

   • Apache Hadoop

     • Setup, Configure and Install ODPi Hadoop
     • Building and Running Apache Hadoop
     • Building Hadoop 2.7.2, Spark 2.0

...

1. • • , Hive 2.0.1

...

Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

Tests

Smoke Tests

Bigtop Smoke Tests

Integration Tests

ODPi Spec Tests

Issues and Resolutions

Apache Ambari

...

1. • • using Apache Bigtop

     • Apache Hadoop Build, Install, Configure and Run Guide

     • Building, Running, Configuring and Profiling Apache Hadoop
     • Apache Hadoop Tuning Notes
     • Apache Hadoop Map Reduce Notes
     • OpenJDK javac Nullpointerexception building Hadoop
     • Patch 1/3 Introduce the HyperCrc32C Checksum class
     • Patch 2/3 libhadoop: CRC: ARM NEON Support
     • Patch 3/3 ModifyCRC to target NEON routine
     • hadoop-lca13

   • ELK - ElasticSearch, Logstash and Kibana

     • Building ELK (ElasticSearch, LogStash and Kibana) on AArch64

     • ELK Setup and Run on AArch64

   • Apache Sqoop

   • Apache Arrow

2. Big Data Operations

   • Apache Ambari

     • Build Apache Ambari on AArch64
     • Build Apache Ambari on AArch64 (Copy)

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-ambari]

Install

Apache Ambari Install, Setup and Configuration

Apache Hadoop

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopBuildInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun]

Install

Apache Spark

1. • • Install, Setup and Configure Apache Ambari
     • JVM Null pointer error in javac when building Apache Ambari

   • Apache Zookeeper

     • Enablement of Apache Zookeeper on AArch64

   • Apache Oozie

   • Apache Falcon

   • Ganglia

2. Big Data Streaming Tools

   • Apache Spark

     • Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache Bigtop
     • Spark 2.0 - Build, configure and Installation steps using Apache

...

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkDependencyLibraries]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_1_6]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_With_BigTop]

[https://wiki.linaro.org/LEG/Engineering/BigData/Spark]

Install

Apache Hive

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/Hive]

Install

Apache HBase

Build

HBase Enablement on AArch64

Install

Apache Zookeeper

Build

Zookeeper Enablement on AAarch64

ELK - ElasticSearch, Kibana and Logstash

Build

ERP 17.08 Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

Install

                              ELK Setup and Run on AArch64

Apache Kafka

H2O

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OScalingStudy]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparklingWaterGuide]

Benchmarks

[http://openjdk.linaro.org/hadoop-terasort-benchmark-results]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopconfig]

[https://wiki.linaro.org/Internal/People/SteveCapper/hadoop-lca13]

[https://wiki.linaro.org/LEG/Engineering/BigData/ApacheBench]

HiBench [wiki]   [Collaborate]

[https://wiki.linaro.org/LEG/Engineering/BigData/BigBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/HiveTestBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/TPCxHS]

[https://wiki.linaro.org/LEG/Engineering/BigData/TPC-H]

Workloads

DataScience

Guidelines

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopTuningGuide]

Misc

NETLIB-JAVA AArch64 Natives support 

[https://wiki.linaro.org/LEG/Engineering/BigData/MapReduceNotes ]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0001-Introduce-the-HyperCrc32C-Checksum-class.patch| PATCH 1/3 Introduce the HyperCrc32C Checksum class]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0002-libhadoop-CRC-ARM-NEON-support.patch| PATCH 2/3 libhadoop: CRC: ARM NEON support]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0003-Modify-HyperCRC-to-target-NEON-routine.patch| PATCH 3/3 Modify HyperCRC to target NEON routine]]

[https://wiki.linaro.org/LEG/Engineering/BigData/Feasibility]

[https://wiki.linaro.org/LEG/Engineering/BigData/CRC32vsNonCRC32Study]

[https://wiki.linaro.org/LEG/Engineering/BigData/configuring-archiva-with-tomcat7]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

== Other Big Data Components in Roadmap ==

=== Apache Flink ===

=== Apache Beam ===

=== Apache Oozie ===

=== Apache Flume ===

=== Apache Storm ===

=== Apache Tez ===

=== Apache Tachyon ===

== Big Data Operations ==

=== Apache Ambari ===

== Big Data - Data warehousing Tools ==

=== Apache Pig ===

=== Apache Cassandra ===

=== Apache Sqoop ===

== Analytical Tools, Machine Learning ==

=== H2O ===

== Big Data Governance and Security ==

=== Apache Ranger ===

=== Apache Knox ===

== Notebooks ==

=== Apache Jupyter ===

=== Apache Zeppelin ===

== File Formats ==

=== Apache Parquet ===

=== Apache Avro ===

...

1. • • Bigtop

     • Profiling Spark on AArch64

     • Building Apache Spark 1.6 on AArch64

     • Building Apache Spark 2.0
     • Spark Dependency Libraries

   • Apache Flink

   • Apache Beam

   • Apache Tez

   • Apache Flume

   • Apache Storm

   • Apache Tachyon

   • Apache Kafka

     • Apache Kafka Streams

   • Apache NiFi

     • Difference between NiFi and MiNiFi

   • Apache MiNiFi

2. Big Data Data warehousing and Database Tools

   • Apache Hive

     • Building Apache Hive

   • Apache HBase

     • Enablement of Apache HBase on AArch64

   • Apache Cassandra

   • Postrgres

   • Memcached

   • MySQL

   • Redis

   • Apache Drill

3. Big Data Data Governance and Security

   • Apache Ranger

   • Apache Knox

   • Apache Atlas

   • Apache Sentry

4. Big Data File Formats

   • Apache Parquet

   • Apache Avro

5. Big Data Datascience Notebooks

   • Apache Jupyter

   • Apache Zeppelin

6. Big Data Analytics

   • Apache Kudu

7. Big Data ML - Machine Learning

   • H20

     • H20 Install and Run Guide

     • H20 Scaling Study
     • Sparkling Water - Spark + H20

8. Big Data component dependencies

   • NETLIB-JAVA AArch64 Natives Support

Tests 

• Smoke Tests

  • BigTop smoke tests 

• Integration Tests

• ODPi Spec Tests

Benchmarking

• TeraSort
• Building, Running, Configuring and Profiling Apache Hadoop
• Spark Bench
• TPC-H
• TPCxHS
• Apache Bench
• BigBench
• Building and Running HiBench on AArch64 Platform
• HiveTestBench

• HiBench

  • Spark profiling on Arm64

Build and Port

• Build Apache Ambari on AArch64
• HBase Enablement on AArch64
• Apache Flink on AArch64
• Zookeeper Enablement on AArch64
• NETLIB-JAVA AArch64 Natives Support
• Apache Ambari Install, Setup and Configuration

Machine Learning

• Building and Installing Tensorflow on AArch64

Misc

• Onboarding info - Welcome to Team
• Big Data team work summary v1.0
• CRC32 vs Non-CRC32 Study
• Configuring archiva with Tomcat
  
  

Bigtop

...

Blogs/Presentations

State of Big Data on Aarch64 - Apache Bigtop

Big Data benchmarking

Himalayan Odyssey

...

Big Data Roadmap

...

Strategic Engineering

Big Data and OpenJDK Strategic Engineering - 2018

Big Data and OpenJDK Strategic Engineering - 2017

...

Big Data Epics

...

JIRA Inprogress