Skip to end of banner

Go to start of banner

Backup of Big Data & Data Science (BDDS)

Skip to end of metadata

Created by Mikael Rolfhamre on Mar 06, 2023

Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Introduction

The aim of this project is to make AArch64 a first class citizen in the Big Data, Analytics and Data Science community (e.g., Hadoop, Spark, etc.). Big Data and Data Science technologies are vital and have become mature with various production implementations. Linaro drives engineering activities and ARMv8 builds. for Apache Ambari, BigTop, Spark and Hadoop.

Roadmap

Current Plan

Edit the macro below and add the appropriate project in the JQL query

key	summary	type	created	updated	due	assignee	reporter	priority	status	resolution

Loading...

Refresh

Backlog

Edit the macro below and add the appropriate project in the JQL query

key	summary	type	created	updated	due	assignee	reporter	priority	status	resolution

Loading...

Refresh

Accomplished

Edit the macro below and add the appropriate project in the JQL query

key	summary	type	created	updated	due	assignee	reporter	priority	status	resolution

Loading...

Refresh

Documentation

Misc

Onboarding info - Welcome to Team

BigData components

Apache Bigtop

Build

Building BigTop using Docker container

Building Hadoop 2.7.2, Spark 2.0, Hive 2.0.1 using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi BigTop Hadoop Installation, setup and running]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPiHadoopMultinodeClusterSetup]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run]

[https://wiki.linaro.org/LEG/Engineering/BigData/ODPi_Setup_Config_Run/ODPi_BigTop_Building]

Install

ERP 16.12: Installing Hadoop 2.7.2, Spark 2.0 and Hive 2.0.1

Bigtop Sandbox with Hadoop, Spark, Hive, HBase for Aarch64

Tests

Smoke Tests

Bigtop Smoke Tests

Integration Tests

ODPi Spec Tests

Apache Ambari

Build

Build Apache Ambari on AArch64

Build and Install Apache Ambari V2.6.1 on AArch64

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-ambari]

Install

Apache Ambari Install, Setup and Configuration

Apache Hadoop

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopBuildInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun]

NETLIB-JAVA AArch64 Natives Support

Install

Apache Spark

Build

Spark 2.0 - Build, configure and Installation steps using Apache BigTop

[https://wiki.linaro.org/LEG/Engineering/BigData/SparkDependencyLibraries]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_1_6]

[https://wiki.linaro.org/LEG/Engineering/BigData/Building_Spark_With_BigTop]

[https://wiki.linaro.org/LEG/Engineering/BigData/Spark]

Install

Apache Hive

Build

[https://wiki.linaro.org/LEG/Engineering/BigData/Hive]

Install

Apache HBase

Build

HBase Enablement on AArch64

Install

Apache Zookeeper

Build

Zookeeper Enablement on AAarch64

ELK - ElasticSearch, Kibana and Logstash

Build

Building ELK (ElasticSearch, LogStash and Kibana) on Aarch64

Install

ELK Setup and Run on AArch64

Apache Kafka

H2O

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OInstallAndRunGuide]

[https://wiki.linaro.org/LEG/Engineering/BigData/H2OScalingStudy]

[https://wiki.linaro.org/LEG/Engineering/BigData/SparklingWaterGuide]

Benchmarking

Machine Learning

Building and Installing Tensorflow on AArch64

Workloads

DataScience

Guidelines

[https://wiki.linaro.org/LEG/Engineering/BigData/HadoopTuningGuide]

Misc

NETLIB-JAVA AArch64 Natives support

[https://wiki.linaro.org/LEG/Engineering/BigData/MapReduceNotes ]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0001-Introduce-the-HyperCrc32C-Checksum-class.patch| PATCH 1/3 Introduce the HyperCrc32C Checksum class]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0002-libhadoop-CRC-ARM-NEON-support.patch| PATCH 2/3 libhadoop: CRC: ARM NEON support]]

[[https://wiki.linaro.org/LEG/Engineering/BigData/hadoopbuildrun?action=AttachFile&do=view&target=0003-Modify-HyperCRC-to-target-NEON-routine.patch| PATCH 3/3 Modify HyperCRC to target NEON routine]]

[https://wiki.linaro.org/LEG/Engineering/BigData/Feasibility]

[https://wiki.linaro.org/LEG/Engineering/BigData/CRC32vsNonCRC32Study]

[https://wiki.linaro.org/LEG/Engineering/BigData/configuring-archiva-with-tomcat7]

[https://wiki.linaro.org/LEG/Engineering/BigData/horrible-jvm-debug-case-for-ed-hadoop]

== Other Big Data Components in Roadmap ==

=== Apache Flink ===

=== Apache Beam ===

=== Apache Oozie ===

=== Apache Flume ===

=== Apache Storm ===

=== Apache Tez ===

=== Apache Tachyon ===

== Big Data Operations ==

=== Apache Ambari ===

== Big Data - Data warehousing Tools ==

=== Apache Pig ===

=== Apache Cassandra ===

=== Apache Sqoop ===

== Analytical Tools, Machine Learning ==

=== H2O ===

== Big Data Governance and Security ==

=== Apache Ranger ===

=== Apache Knox ===

== Notebooks ==

=== Apache Jupyter ===

=== Apache Zeppelin ===

== File Formats ==

=== Apache Parquet ===

=== Apache Avro ===

== Functional, Regression and Workload Testing ==

Project Meetings

Project Contacts

Ganesh Raju (Project Lead) | ganesh.raju@linaro.org
Yuqi Gu (Assignee, ARM) | yuqi.gu@linaro.org
Mailing Address: leg-bigdata@linaro.org

Blogs / Presentations:

Active Members

Documentation

No labels