Introduction

The aim of this project is to make AArch64 a first class citizen in the Big Data, Analytics and Data Science community (e.g., Hadoop, Spark, etc.). Big Data and Data Science technologies are vital and have become mature with various production implementations. Linaro drives engineering activities and ARMv8 builds. for Apache Ambari, BigTop, Spark and Hadoop.

Roadmap

Current Plan

Edit the macro below and add the appropriate project in the JQL query

Backlog

Edit the macro below and add the appropriate project in the JQL query

Accomplished

Edit the macro below and add the appropriate project in the JQL query

Documentation

Open

Misc

• Onboarding info - Welcome to Team

BigData components

Apache Bigtop

Build

Building BigTop using Docker container

Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi BigTop Hadoop Installation, setup and running]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPiHadoopMultinodeClusterSetup]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run/ODPi_BigTop_Building]

Install

ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0 and Hive 2.0.1

Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

Tests

Smoke Tests

Bigtop Smoke Tests

Integration Tests

ODPi Spec Tests

• Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

• Building BigTop using Docker container

• Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

• Bigtop Smoke Tests

Apache Ambari

Build

Build Apache Ambari on AArch64

Build and Install Apache Ambari V2.6.1 on AArch64

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-ambari]

Install

Apache Ambari Install, Setup and Configuration

Apache Hadoop

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopBuildInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun]

• NETLIB-JAVA AArch64 Natives Support

Install

Apache Spark

Build

Spark 2.0 - Build, configure and Installation steps using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkDependencyLibraries]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_1_6]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_With_BigTop]

[https://wiki.linaro.org/LEG/Engineering/BigData/Spark]

Install

Apache Hive

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/Hive]

Install

Apache HBase

Build

HBase Enablement on AArch64

Install

Apache Zookeeper

Build

Zookeeper Enablement on AAarch64

ELK - ElasticSearch, Kibana and Logstash

Build

Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

Install

ELK Setup and Run on AArch64

Apache Kafka

H2O

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OScalingStudy]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparklingWaterGuide]

Benchmarking

• Spark Bench

• TPC-H

• TPCxHS

• BigBench

• Building and Running HiBench on AArch64 Platform

• HiveTestBench

• [http://openjdk.linaro.org/hadoop-terasort-benchmark-results]

  [https://wiki.linaro.org/LEG/Engineering/BigData/hadoopconfig]

  [https://wiki.linaro.org/Internal/People/SteveCapper/hadoop-lca13]

  [https://wiki.linaro.org/LEG/Engineering/BigData/ApacheBench]

  HiBench [wiki]   [Collaborate]

  [https://wiki.linaro.org/LEG/Engineering/BigData/BigBench]

  [https://wiki.linaro.org/LEG/Engineering/BigData/HiveTestBench]

  [https://wiki.linaro.org/LEG/Engineering/BigData/SparkBench]

  [https://wiki.linaro.org/LEG/Engineering/BigData/TPCxHS]

  [https://wiki.linaro.org/LEG/Engineering/BigData/TPC-H]

   

Machine Learning

• Building and Installing Tensorflow on AArch64

Workloads

DataScience

Guidelines

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopTuningGuide]

Misc

NETLIB-JAVA AArch64 Natives support 

[https://wiki.linaro.org/LEG/Engineering/BigData/MapReduceNotes ]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0001-Introduce-the-HyperCrc32C-Checksum-class.patch| PATCH 1/3 Introduce the HyperCrc32C Checksum class]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0002-libhadoop-CRC-ARM-NEON-support.patch| PATCH 2/3 libhadoop: CRC: ARM NEON support]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0003-Modify-HyperCRC-to-target-NEON-routine.patch| PATCH 3/3 Modify HyperCRC to target NEON routine]]

[https://wiki.linaro.org/LEG/Engineering/BigData/Feasibility]

[https://wiki.linaro.org/LEG/Engineering/BigData/CRC32vsNonCRC32Study]

[https://wiki.linaro.org/LEG/Engineering/BigData/configuring-archiva-with-tomcat7]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

== Other Big Data Components in Roadmap ==

=== Apache Flink ===

=== Apache Beam ===

=== Apache Oozie ===

=== Apache Flume ===

=== Apache Storm ===

=== Apache Tez ===

=== Apache Tachyon ===

== Big Data Operations ==

=== Apache Ambari ===

== Big Data - Data warehousing Tools ==

=== Apache Pig ===

=== Apache Cassandra ===

=== Apache Sqoop ===

== Analytical Tools, Machine Learning ==

=== H2O ===

== Big Data Governance and Security ==

=== Apache Ranger ===

=== Apache Knox ===

== Notebooks ==

=== Apache Jupyter ===

=== Apache Zeppelin ===

== File Formats ==

=== Apache Parquet ===

=== Apache Avro ===

== Functional, Regression and Workload Testing ==

Project Meetings

Project Contacts

• Ganesh Raju (Project Lead) | ganesh.raju@linaro.org

• Yuqi Gu (Assignee, ARM) | yuqi.gu@linaro.org

• Mailing Address: leg-bigdata@linaro.org 

Blogs / Presentations:

• State of Big Data on Aarch64 - Apache Bigtop

• Big Data benchmarking

• BDDS Review Nov 2021 - 1

• BDDS Review Nov 2021 - 2

• BDDS Strategic Roadmap Sep 2021

• BDDS Strategic Roadmap July 2021

• Plan of Record BDDS Apr 2021

• Apache Bigtop v3.0 Release

• Bigtop v3.0 With the Upgraded Mpack: New Era of Big Data Distribution

• LVC21-F202 Bigtop v3.0 with upgraded mpack

• LVC20-303 State of Big Data and Data Science on ARM

• SAN19-405 How to lift an elephant with ARM

• Make Life easier for Big Data users on ARM

• Big Data on ARM in Practice with ThunderX

• Qualcomm ARM Server Centriq at Linaro Connect SFO2017

• Big Data Benchmarking on ARM

• BKK19-TR01 Bigtop 101

• BDDS Strategic Roadmap Apr 2020

• Exploring GitHub data with Apache Drill on ARM

• AArch64 and Apache Bigtop - Empowering Big Data Everywhere

• Apache Drill Demo

• Apache Bigtop v1.5 and Wikimedia: Empower BigData in the real world

• New Apache Bigtop v1.5 and Wikimedia

• Apache Ambari on ARM

• LVC20-101 Make Big Data Fly on ARM

• BKK19-101 - AArch64 and Apache Bigtop

• SAN19-306 The Convergence of Big Data and AI

• LVC21F-201 Apache Arrow: A unified language across Big Data

• LTD20-104 Make life easier for Bigtop Data users on ARM

• Demo - Smart City Use Case with Hadoop, Spark, H2O and Sparkling Water

• Himalayan Odyssey

Active Members

Open

Documentation