Current Plan

The following items are on the project backlog but not currently planned. If you are interested in contributing to any of these items, please state your intention on the project's mailing list (found above)

Work in Progress

Coming soon…

Plan of Record

Coming soon…

Health Checks

Coming soon…

Portal

Documentations

ERP 

• ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0 and Hive 2.0.1

• ERP 17.08 Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

• ERP 17.08 - Build, Configure, Install and Test Hadoop, Spark, Hive, Hbase in containerized environment

• Resources

  • ERP 17.08 deb packages

Big Data Components

1. Apache Bigtop

   • Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

   • Building BigTop using Docker container

   • Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

   • Bigtop Smoke Tests

   • Resources

     • Bigtop 1.2 Components

     • Bigtop Docker images

     • Upstream github

     • CI Loop

     • v1.2 Release

     • Bigtop AArch64 patches

     • Bigtop Project Ideas and Roadmap

2. Big Data Core Components

   • Apache Hadoop

     • Setup, Configure and Install ODPi Hadoop

     • Building and Running Apache Hadoop

     • Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache Bigtop

     • Apache Hadoop Build, Install, Configure and Run Guide

     • Building, Running, Configuring and Profiling Apache Hadoop

     • Apache Hadoop Tuning Notes

     • Apache Hadoop Map Reduce Notes

     • OpenJDK javac Nullpointerexception building Hadoop

     • Patch 1/3 Introduce the HyperCrc32C Checksum class

     • Patch 2/3 libhadoop: CRC: ARM NEON Support

     • Patch 3/3 ModifyCRC to target NEON routine

     • hadoop-lca13

3. Big Data Operations

   • Apache Ambari

     • Build Apache Ambari on AArch64

     • Build Apache Ambari on AArch64 (Copy)

     • Install, Setup and Configure Apache Ambari

     • JVM Null pointer error in javac when building Apache Ambari

   • Apache Zookeeper

     • Enablement of Apache Zookeeper on AArch64

   • Apache Oozie

   • Apache Falcon

   • Ganglia

4. Big Data Streaming Tools

   • Apache Spark

     • Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache Bigtop

     • Spark 2.0 - Build, configure and Installation steps using Apache Bigtop

     • Profiling Spark on AArch64

     • Building Apache Spark 1.6 on AArch64

     • Building Apache Spark 2.0

     • Spark Dependency Libraries

   • Apache Flink

   • Apache Beam

   • Apache TezJIRA Inprogress

   • Apache Flume

   • Apache Storm

   • Apache Alluxio

   • Apache Kafka

     • Apache Kafka Streams

   • Apache NiFi

     • Difference between NiFi and MiNiFi

   • Apache MiNiFi

5. Big Data Data warehousing and Database Tools

   • Apache Hive

     • Building Apache Hive

   • Apache HBase

     • Enablement of Apache HBase on AArch64

   • Apache Cassandra

   • Postrgres

   • Memcached

   • MySQL

   • Redis

   • Apache Drill

   • Apache Sqoop

6. Big Data Data Governance and Security

   • Apache Ranger

   • Apache Knox

   • Apache Atlas

   • Apache Sentry

7. Big Data File Formats

   • Apache Parquet

   • Apache Avro

8. Big Data Datascience Notebooks

   • Apache Jupyter

   • Apache Zeppelin

9. Big Data Analytics

   • ELK - ElasticSearch, Logstash and Kibana

     • ERP 17.08 Building ELK (ElasticSearch, LogStash and Kibana) on AArch64

     • ELK Setup and Run on AArch64

   • Apache Arrow

   • Apache Kudu

10. Big Data ML - Machine Learning

    • H20

      • H20 Install and Run Guide

      • H20 Scaling Study

      • Sparkling Water - Spark + H20

11. Big Data component dependencies

    • NETLIB-JAVA AArch64 Natives Support

Tests 

• Smoke Tests

  • BigTop smoke tests 

• Integration Tests

• ODPi Spec Tests

Benchmarking

• TeraSort

• Building, Running, Configuring and Profiling Apache Hadoop

• Spark Bench

• TPC-H

• TPCxHS

• Apache Bench

• BigBench

• Building and Running HiBench on AArch64 Platform

• HiveTestBench

• HiBench

  • Spark profiling on Arm64

Build and Port

• Build Apache Ambari on AArch64

• HBase Enablement on AArch64

• Apache Flink on AArch64

• Zookeeper Enablement on AArch64

• NETLIB-JAVA AArch64 Natives Support

• Apache Ambari Install, Setup and Configuration

Machine Learning

• Building and Installing Tensorflow on AArch64

Misc

• Onboarding info - Welcome to Team

• Big Data team work summary v1.0

• CRC32 vs Non-CRC32 Study

• Configuring archiva with Tomcat

Bigtop

Blogs/Presentations

State of Big Data on Aarch64 - Apache Bigtop

Big Data benchmarking

Himalayan Odyssey

Big Data Roadmap

Strategic Engineering

Big Data and OpenJDK Strategic Engineering - 2018

Big Data and OpenJDK Strategic Engineering - 2017

Recently updated

• 2020 Conferences

  Jun 11, 2020 • contributed by Ganesh Raju
• Big Data JIRA status 03 2020

  Mar 31, 2020 • contributed by Ganesh Raju
• Apache Knox

  Mar 18, 2020 • contributed by Former user (Deleted)
• Building and Installing Tensorflow on AArch64

  Oct 03, 2019 • contributed by Ganesh Raju
• Zookeeper Enablement on AArch64

  Aug 28, 2019 • contributed by Ganesh Raju
• Install and Running Jupyter notebook on ARM64

  Aug 28, 2019 • contributed by Ganesh Raju
• Apache Drill with Jupyter Notebook

  Aug 14, 2019 • contributed by Ganesh Raju
• Apache Ranger and MIT Kerberos Enablement - Big Data security

  Aug 09, 2019 • contributed by Ganesh Raju
• Big Data Components and their Latest Release dates/versions

  Jul 29, 2019 • contributed by Ganesh Raju
• Deploy Ambari ODPI-mpack for Bigtop

  Apr 22, 2019 • contributed by Yuqi Gu

Show More