Ranked awesome lists, all in one place

This list is a copy of youngwookim/awesome-hadoop with ranks

Awesome Hadoop ★87749

A curated list of amazingly awesome Hadoop and Hadoop ecosystem resources. Inspired by Awesome PHP, Awesome Python and Awesome Sysadmin ★17190

Awesome Hadoop
Resources
Other Awesome Lists

Hadoop

Apache Hadoop - Apache Hadoop
Apache Tez - A Framework for YARN-based, Data Processing Applications In Hadoop
SpatialHadoop - SpatialHadoop is a MapReduce extension to Apache Hadoop designed specially to work with spatial data.
GIS Tools for Hadoop - Big Data Spatial Analytics for the Hadoop Framework
Elasticsearch Hadoop ★1262 - Elasticsearch real-time search and analytics natively integrated with Hadoop. Supports Map/Reduce, Cascading, Apache Hive and Apache Pig.
hadoopy ★245 ⏳2Y - Python MapReduce library written in Cython.
mrjob ★2257 - mrjob is a Python 2.5+ package that helps you write and run Hadoop Streaming jobs.
pydoop - Pydoop is a package that provides a Python API for Hadoop.
hdfs-du ★181 ⏳1Y - HDFS-DU is an interactive visualization of the Hadoop distributed file system.
White Elephant ★188 ⏳4Y - Hadoop log aggregator and dashboard
Genie ★907 - Genie provides REST-ful APIs to run Hadoop, Hive and Pig jobs, and to manage multiple Hadoop resources and perform job submissions across them.
Apache Kylin - Apache Kylin is an open source Distributed Analytics Engine from eBay Inc. that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets
Crunch ★164 ⏳3Y - Go-based toolkit for ETL and feature extraction on Hadoop
Apache Ignite - Distributed in-memory platform

YARN

Apache Slider - Apache Slider is a project in incubation at the Apache Software Foundation with the goal of making it possible and easy to deploy existing applications onto a YARN cluster.
Apache Twill - Apache Twill is an abstraction over Apache Hadoop® YARN that reduces the complexity of developing distributed applications, allowing developers to focus more on their application logic.
mpich2-yarn ★95 - Running MPICH2 on Yarn

NoSQL

Next Generation Databases mostly addressing some of the points: being non-relational, distributed, open-source and horizontally scalable.

Apache HBase - Apache HBase
Apache Phoenix - A SQL skin over HBase supporting secondary indices
happybase ★405 - A developer-friendly Python library to interact with Apache HBase.
Hannibal ★160 - Hannibal is tool to help monitor and maintain HBase-Clusters that are configured for manual splitting.
Haeinsa ★142 ⏳1Y - Haeinsa is linearly scalable multi-row, multi-table transaction library for HBase
hindex ★492 ⏳1Y - Secondary Index for HBase
Apache Accumulo - The Apache Accumulo™ sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
OpenTSDB - The Scalable Time Series Database
Apache Cassandra

SQL on Hadoop

SQL on Hadoop

Apache Hive - The Apache Hive data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL
Apache Phoenix A SQL skin over HBase supporting secondary indices
Apache HAWQ (incubating) - Apache HAWQ is a Hadoop native SQL query engine that combines the key technological advantages of MPP database with the scalability and convenience of Hadoop
Lingual - SQL interface for Cascading (MR/Tez job generator)
Apache Impala - Apache Impala is an open source massively parallel processing (MPP) SQL query engine for data stored in a computer cluster running Apache Hadoop. Impala has been described as the open-source equivalent of Google F1, which inspired its development in 2012.
Presto - Distributed SQL Query Engine for Big Data. Open sourced by Facebook.
Apache Tajo - Data warehouse system for Apache Hadoop
Apache Drill - Schema-free SQL Query Engine
Apache Trafodion

Data Management

Apache Calcite - A Dynamic Data Management Framework
Apache Atlas - Metadata tagging & lineage capture suppoting complex business data taxonomies
Apache Kudu - Kudu provides a combination of fast inserts/updates and efficient columnar scans to enable multiple real-time analytic workloads across a single storage layer, complementing HDFS and Apache HBase.
Confluent Schema registry for Kafka ★667 - Schema Registry provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas.
Hortonworks Schema Registry ★77 - Schema Registry is a framework to build metadata repositories.

Workflow, Lifecycle and Governance

Apache Oozie - Apache Oozie
Azkaban
Apache Falcon - Data management and processing platform
Apache NiFi - A dataflow system
Apache AirFlow ★8345 - Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines
Luigi - Python package that helps you build complex pipelines of batch jobs

Data Ingestion and Integration

Apache Flume - Apache Flume
Suro ★675 ⏳2Y - Netflix’s distributed Data Pipeline
Apache Sqoop - Apache Sqoop
Apache Kafka - Apache Kafka
Gobblin from LinkedIn - Universal data ingestion framework for Hadoop

DSL

Apache Pig - Apache Pig
Apache DataFu - A collection of libraries for working with large-scale data in Hadoop
vahara ★52 ⏳4Y - Machine learning and natural language processing with Apache Pig
packetpig ★296 - Open Source Big Data Security Analytics
akela ★80 ⏳4Y - Mozilla’s utility library for Hadoop, HBase, Pig, etc.
seqpig - Simple and scalable scripting for large sequencing data set(ex: bioinfomation) in Hadoop
Lipstick ★423 - Pig workflow visualization tool. Introducing Lipstick on A(pache) Pig
PigPen ★481 ⏳1Y - PigPen is map-reduce for Clojure, or distributed Clojure. It compiles to Apache Pig, but you don’t need to know much about Pig to use it.

Libraries and Tools

Kite Software Development Kit - A set of libraries, tools, examples, and documentation
gohadoop ★287 - Native go clients for Apache Hadoop YARN.
Hue - A Web interface for analyzing data with Apache Hadoop.
Apache Zeppelin - A web-based notebook that enables interactive data analytics
Jumbune - Jumbune is an open-source product built for analyzing Hadoop cluster and MapReduce jobs.
Apache Thrift
Apache Avro - Apache Avro is a data serialization system.
Elephant Bird ★1061 - Twitter’s collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.
Spring for Apache Hadoop
hdfs - A native go client for HDFS ★481
Oozie Eclipse Plugin - A graphical editor for editing Apache Oozie workflows inside Eclipse.
Hydrosphere Mist ★180 - a service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services.
snakebite - A pure python HDFS client
Apache Parquet - Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.
Apache Superset (incubating) - Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application
Schema Registry UI ★187 - Web tool for the Confluent Schema Registry in order to create / view / search / evolve / view history & configure Avro schemas of your Kafka cluster.

Realtime Data Processing

Apache Storm
Apache Samza
Apache Spark
Apache Flink - Apache Flink is a platform for efficient, distributed, general-purpose data processing. It supports exactly once stream processing.
Apache Pulsar (incubating) - Apache Pulsar (incubating) is a highly scalable, low latency messaging platform running on commodity hardware. It provides simple pub-sub semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication.
Druid - Column oriented distributed data store ideal for powering interactive applications

Distributed Computing and Programming

Apache Spark
Spark Packages - A community index of packages for Apache Spark
SparkHub - A community site for Apache Spark
Apache Crunch
Cascading - Cascading is the proven application development platform for building data applications on Hadoop.
Apache Flink - Apache Flink is a platform for efficient, distributed, general-purpose data processing.
Apache Apex (incubating) - Enterprise-grade unified stream and batch processing engine.
Apache Livy (incubating) - Apache Livy (incubating) is web service that exposes a REST interface for managing long running Apache Spark contexts in your cluster. With Livy, new applications can be built on top of Apache Spark that require fine grained interaction with many Spark contexts.

Packaging, Provisioning and Monitoring

Apache Bigtop - Apache Bigtop: Packaging and tests of the Apache Hadoop ecosystem
Apache Ambari - Apache Ambari
Ganglia Monitoring System
ankush - A big data cluster management tool that creates and manages clusters of different technologies.
Apache Zookeeper - Apache Zookeeper
Apache Curator - ZooKeeper client wrapper and rich ZooKeeper framework
Buildoop ★30 ⏳2Y - Hadoop Ecosystem Builder
Deploop - The Hadoop Deploy System
Jumbune - An open source MapReduce profiling, MapReduce flow debugging, HDFS data quality validation and Hadoop cluster monitoring tool.
inviso ★165 ⏳1Y - Inviso is a lightweight tool that provides the ability to search for Hadoop jobs, visualize the performance, and view cluster utilization.

Search

ElasticSearch
Apache Solr - Apache Solr is an open source search platform built upon a Java library called Lucene.
SenseiDB - Open-source, distributed, realtime, semi-structured database
Banana ★531 - Kibana port for Apache Solr

Search Engine Framework

Apache Nutch - Apache Nutch is a highly extensible and scalable open source web crawler software project.

Security

Apache Ranger - Ranger is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform.
Apache Sentry - An authorization module for Hadoop
Apache Knox Gateway - A REST API Gateway for interacting with Hadoop clusters.
Project Rhino ★90 ⏳3Y - Intel’s open source effort to enhance the existing data protection capabilities of the Hadoop ecosystem to address security and compliance challenges, and contribute the code back to Apache.

Benchmark

Big Data Benchmark
HiBench ★674
Big-Bench ★80
hive-benchmarks ★3 ⏳4Y
hive-testbench ★54 - Testbench for experimenting with Apache Hive at any data scale.
YCSB ★2054 - The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare relative performance of NoSQL database management systems.

Machine learning and Big Data analytics

Apache Mahout
Oryx 2 ★1437 - Lambda architecture on Spark, Kafka for real-time large scale machine learning
MLlib - MLlib is Apache Spark’s scalable machine learning library.
R - R is a free software environment for statistical computing and graphics.
RHadoop including RHDFS, RHBase, RMR2, plyrmr
RHive ★123 RHive, for launching Hive queries from R
Apache Lens
Apache SINGA (incubating) - SINGA is a general distributed deep learning platform for training big deep learning models over large datasets
BigDL - BigDL is a distributed deep learning library for Apache Spark; with BigDL, users can write their deep learning applications as standard Spark programs, which can directly run on top of existing Spark or Hadoop clusters.

Misc.

Hive Plugins
UDF
- http://nexr.github.io/hive-udf/
- https://github.com/edwardcapriolo/hive_cassandra_udfs
- https://github.com/livingsocial/HiveSwarm
- https://github.com/ThinkBigAnalytics/Hive-Extensions-from-Think-Big-Analytics
- https://github.com/karthkk/udfs
- https://github.com/twitter/elephant-bird - Twitter
- https://github.com/lovelysystems/ls-hive
- https://github.com/stewi2/hive-udfs
- https://github.com/klout/brickhouse
- https://github.com/markgrover/hive-translate (PostgreSQL translate())
- https://github.com/deanwampler/HiveUDFs
- https://github.com/myui/hivemall (Machine Learning UDF/UDAF/UDTF)
- https://github.com/edwardcapriolo/hive-geoip (GeoIP UDF)
- https://github.com/Netflix/Surus
Storage Handler
- https://github.com/dvasilen/Hive-Cassandra
- https://github.com/yc-huang/Hive-mongo
- https://github.com/balshor/gdata-storagehandler
- https://bitbucket.org/rodrigopr/redisstoragehandler
- https://github.com/zhuguangbin/HiveJDBCStorageHanlder
- https://github.com/chimpler/hive-solr
- https://github.com/bfemiano/accumulo-hive-storage-manager
SerDe
- https://github.com/rcongiu/Hive-JSON-Serde
- https://github.com/mochi/hive-json-serde
- https://github.com/ogrodnek/csv-serde
- https://github.com/parag/HiveJsonSerde
Libraries and tools
- https://github.com/forward3d/rbhive
- https://github.com/synctree/activerecord-hive-adapter
- https://github.com/hrp/sequel-hive-adapter
- https://github.com/forward/node-hive
- https://github.com/recruitcojp/WebHive
- shib ★162 ⏳1Y - WebUI for query engines: Hive and Presto
- clive ★4 ⏳6Y - Clojure library for interacting with Hive via Thrift
- https://github.com/dmorel/Thrift-API-HiveClient2 (Perl - HiveServer2)
- PyHive ★696 - Python interface to Hive and Presto
- https://github.com/recruitcojp/OdbcHive
- Hive-Sharp
- HiveRunner ★116 - An Open Source unit test framework for hadoop hive queries based on JUnit4
- Beetest ★62 ⏳1Y - A super simple utility for testing Apache Hive scripts locally for non-Java developers.
- Hive_test ★66 ⏳1Y- Unit test framework for hive and hive-service
Flume Plugins
- Flume MongoDB Sink ★55 ⏳1Y
- Flume HornetQ Channel ★0 ⏳5Y
- Flume MessagePack Source ★0 ⏳5Y
- Flume RabbitMQ source and sink ★55 ⏳1Y
- Flume UDP Source ★8 ⏳4Y
- Stratio Ingestion ★142 ⏳1Y - Custom sinks: Cassandra, MongoDB, Stratio Streaming and JDBC
- Flume Custom Serializers ★5 ⏳5Y
- Real-time analytics in Apache Flume ★45 ⏳2Y
- .Net FlumeNG Clients ★16 ⏳4Y

Resources

Various resources, such as books, websites and articles.

Websites

Useful websites and articles

Presentations

Hadoop Summit Presentations - Slide decks from Hadoop Summit
Hadoop 24/7
An example Apache Hadoop Yarn upgrade
Apache Hadoop In Theory And Practice
Hadoop Operations at LinkedIn
Hadoop Performance at LinkedIn
Docker based Hadoop provisioning

Books

Hadoop and Big Data Events

Other Awesome Lists

Other amazingly awesome lists can be found in the awesome-awesomeness and awesome ★87749 list. —

This list is a copy of youngwookim/awesome-hadoop with ranks