Documentation

Dec 05, 2022

1 BigData components
- 1.1 ODPI and Apache BigTop
  - 1.1.1 Build
  - 1.1.2 Install
  - 1.1.3 Tests
    - 1.1.3.1 Smoke Tests
    - 1.1.3.2 Integration Tests
    - 1.1.3.3 ODPi Spec Tests
- 1.2 Issues and Resolutions
- 1.3 Apache Ambari
  - 1.3.1 Build
  - 1.3.2 Install
- 1.4 Apache Hadoop
  - 1.4.1 Build
  - 1.4.2 Install
- 1.5 Apache Spark
  - 1.5.1 Build
  - 1.5.2 Install
- 1.6 Apache Hive
  - 1.6.1 Build
  - 1.6.2 Install
- 1.7 Apache HBase
  - 1.7.1 Build
  - 1.7.2 Install
- 1.8 Apache Zookeeper
  - 1.8.1 Build
- 1.9 ELK - ElasticSearch, Kibana and Logstash
  - 1.9.1 Build
  - 1.9.2 Install
- 1.10 Apache Kafka
- 1.11 H2O
2 Benchmarks
3 Workloads
4 DataScience
5 Guidelines
6 Misc
7 == Other Big Data Components in Roadmap ==
- 7.1 === Apache Flink ===
- 7.2 === Apache Beam ===
- 7.3 === Apache Oozie ===
- 7.4 === Apache Flume ===
- 7.5 === Apache Storm ===
- 7.6 === Apache Tez ===
- 7.7 === Apache Tachyon ===
8 == Big Data Operations ==
- 8.1 === Apache Ambari ===
9 == Big Data - Data warehousing Tools ==
- 9.1 === Apache Pig ===
- 9.2 === Apache Cassandra ===
- 9.3 === Apache Sqoop ===
10 == Analytical Tools, Machine Learning ==
- 10.1 === H2O ===
11 == Big Data Governance and Security ==
- 11.1 === Apache Ranger ===
- 11.2 === Apache Knox ===
12 == Notebooks ==
- 12.1 === Apache Jupyter ===
- 12.2 === Apache Zeppelin ===
13 == File Formats ==
- 13.1 === Apache Parquet ===
- 13.2 === Apache Avro ===
14 == Functional, Regression and Workload Testing ==

BigData components

ODPI and Apache BigTop

Build

Building BigTop using Docker container

Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi BigTop Hadoop Installation, setup and running]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPiHadoopMultinodeClusterSetup]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run/ODPi_BigTop_Building]

Install

ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0 and Hive 2.0.1

Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

Tests

Smoke Tests

Bigtop Smoke Tests

Integration Tests

ODPi Spec Tests

Issues and Resolutions

Apache Ambari

Build

Build Apache Ambari on AArch64

Build and Install Apache Ambari V2.6.1 on AArch64

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-ambari]

Install

Apache Ambari Install, Setup and Configuration

Apache Hadoop

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopBuildInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun]

Install

Apache Spark

Build

Spark 2.0 - Build, configure and Installation steps using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkDependencyLibraries]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_1_6]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_With_BigTop]

[https://wiki.linaro.org/LEG/Engineering/BigData/Spark]

Install

Apache Hive

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/Hive]

Install

Apache HBase

Build

HBase Enablement on AArch64

Install

Apache Zookeeper

Build

Zookeeper Enablement on AAarch64

ELK - ElasticSearch, Kibana and Logstash

Build

Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

Install

ELK Setup and Run on AArch64

Apache Kafka

H2O

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OScalingStudy]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparklingWaterGuide]

Benchmarks

[http://openjdk.linaro.org/hadoop-terasort-benchmark-results]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopconfig]

[https://wiki.linaro.org/Internal/People/SteveCapper/hadoop-lca13]

[https://wiki.linaro.org/LEG/Engineering/BigData/ApacheBench]

HiBench [wiki] [Collaborate]

[https://wiki.linaro.org/LEG/Engineering/BigData/BigBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/HiveTestBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkBench]

[https://wiki.linaro.org/LEG/Engineering/BigData/TPCxHS]

[https://wiki.linaro.org/LEG/Engineering/BigData/TPC-H]

Workloads

DataScience

Guidelines

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopTuningGuide]

Misc

NETLIB-JAVA AArch64 Natives support

[https://wiki.linaro.org/LEG/Engineering/BigData/MapReduceNotes ]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0001-Introduce-the-HyperCrc32C-Checksum-class.patch| PATCH 1/3 Introduce the HyperCrc32C Checksum class]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0002-libhadoop-CRC-ARM-NEON-support.patch| PATCH 2/3 libhadoop: CRC: ARM NEON support]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0003-Modify-HyperCRC-to-target-NEON-routine.patch| PATCH 3/3 Modify HyperCRC to target NEON routine]]

[https://wiki.linaro.org/LEG/Engineering/BigData/Feasibility]

[https://wiki.linaro.org/LEG/Engineering/BigData/CRC32vsNonCRC32Study]

[https://wiki.linaro.org/LEG/Engineering/BigData/configuring-archiva-with-tomcat7]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

== Other Big Data Components in Roadmap ==

=== Apache Flink ===

=== Apache Beam ===

=== Apache Oozie ===

=== Apache Flume ===

=== Apache Storm ===

=== Apache Tez ===

=== Apache Tachyon ===

== Big Data Operations ==

=== Apache Ambari ===

== Big Data - Data warehousing Tools ==

=== Apache Pig ===

=== Apache Cassandra ===

=== Apache Sqoop ===

== Analytical Tools, Machine Learning ==

=== H2O ===

== Big Data Governance and Security ==

=== Apache Ranger ===

=== Apache Knox ===

== Notebooks ==

=== Apache Jupyter ===

=== Apache Zeppelin ===

== File Formats ==

=== Apache Parquet ===

=== Apache Avro ===

== Functional, Regression and Workload Testing ==