Introduction

The aim of this project is to make AArch64 a first class citizen in the Big Data, Analytics and Data Science community (e.g., Hadoop, Spark, etc.). Big Data and Data Science technologies are vital and have become mature with various production implementations. Linaro drives engineering activities and ARMv8 builds. for Apache Ambari, BigTop, Spark and Hadoop.

Roadmap

Current Plan

Edit the macro below and add the appropriate project in the JQL query

Backlog

Edit the macro below and add the appropriate project in the JQL query

Accomplished

Edit the macro below and add the appropriate project in the JQL query

Documentation

Misc

• Onboarding info - Welcome to Team

BigData components

Apache Bigtop

Build

Building BigTop using Docker container

Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi BigTop Hadoop Installation, setup and running]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPiHadoopMultinodeClusterSetup]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run/ODPi_BigTop_Building]

Install

ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0 and Hive 2.0.1

Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

Tests

Smoke Tests

Bigtop Smoke Tests

Integration Tests

ODPi Spec Tests

• Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

• Building BigTop using Docker container

• Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

• Bigtop Smoke Tests

Apache Ambari

Build

Build Apache Ambari on AArch64

Build and Install Apache Ambari V2.6.1 on AArch64

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-ambari]

Install

Apache Ambari Install, Setup and Configuration

Apache Hadoop

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopBuildInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun]

• NETLIB-JAVA AArch64 Natives Support

Install

Apache Spark

Build

Spark 2.0 - Build, configure and Installation steps using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkDependencyLibraries]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_1_6]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_With_BigTop]

[https://wiki.linaro.org/LEG/Engineering/BigData/Spark]

Install

Apache Hive

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/Hive]

Install

Apache HBase

Build

HBase Enablement on AArch64

Install

Apache Zookeeper

Build

Zookeeper Enablement on AAarch64

ELK - ElasticSearch, Kibana and Logstash

Build

Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

Install

ELK Setup and Run on AArch64

Apache Kafka

H2O

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OScalingStudy]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparklingWaterGuide]

Benchmarking

• Spark Bench

• TPC-H

• TPCxHS

• BigBench

• Building and Running HiBench on AArch64 Platform

• HiveTestBench

• [http://openjdk.linaro.org/hadoop-terasort-benchmark-results]

  [https://wiki.linaro.org/LEG/Engineering/BigData/hadoopconfig]

  [https://wiki.linaro.org/Internal/People/SteveCapper/hadoop-lca13]

  [https://wiki.linaro.org/LEG/Engineering/BigData/ApacheBench]

  HiBench [wiki]   [Collaborate]

  [https://wiki.linaro.org/LEG/Engineering/BigData/BigBench]

  [https://wiki.linaro.org/LEG/Engineering/BigData/HiveTestBench]

  [https://wiki.linaro.org/LEG/Engineering/BigData/SparkBench]

  [https://wiki.linaro.org/LEG/Engineering/BigData/TPCxHS]

  [https://wiki.linaro.org/LEG/Engineering/BigData/TPC-H]

Machine Learning

• Building and Installing Tensorflow on AArch64

Workloads

DataScience

Guidelines

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopTuningGuide]

Misc

NETLIB-JAVA AArch64 Natives support 

[https://wiki.linaro.org/LEG/Engineering/BigData/MapReduceNotes ]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0001-Introduce-the-HyperCrc32C-Checksum-class.patch| PATCH 1/3 Introduce the HyperCrc32C Checksum class]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0002-libhadoop-CRC-ARM-NEON-support.patch| PATCH 2/3 libhadoop: CRC: ARM NEON support]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0003-Modify-HyperCRC-to-target-NEON-routine.patch| PATCH 3/3 Modify HyperCRC to target NEON routine]]

[https://wiki.linaro.org/LEG/Engineering/BigData/Feasibility]

[https://wiki.linaro.org/LEG/Engineering/BigData/CRC32vsNonCRC32Study]

[https://wiki.linaro.org/LEG/Engineering/BigData/configuring-archiva-with-tomcat7]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

== Other Big Data Components in Roadmap ==

=== Apache Flink ===

=== Apache Beam ===

=== Apache Oozie ===

=== Apache Flume ===

=== Apache Storm ===

=== Apache Tez ===

=== Apache Tachyon ===

== Big Data Operations ==

=== Apache Ambari ===

== Big Data - Data warehousing Tools ==

=== Apache Pig ===

=== Apache Cassandra ===

=== Apache Sqoop ===

== Analytical Tools, Machine Learning ==

=== H2O ===

== Big Data Governance and Security ==

=== Apache Ranger ===

=== Apache Knox ===

== Notebooks ==

=== Apache Jupyter ===

=== Apache Zeppelin ===

== File Formats ==

=== Apache Parquet ===

=== Apache Avro ===

== Functional, Regression and Workload Testing ==