List of Apache Projects For Big Data

It is possibly confusing to many new users when we talk about combining various big data related softwares. Here is a List of Apache Projects For Big Data With Basic Practical Details Which is Helpful to the Developers Who Are New in Big Data Field. Apache Hadoop and Apache Spark are possibly most known. At present there are total 37 Apache projects which are directly related to Big Data.

List of Apache Projects For Big Data

There are the 37 Big Data related Apache projects:

1. Apache Airavata
2. Apache Ambari
3. Apache Apex
4. Apache Avro
5. Apache Beam
6. Apache Bigtop
7. Apache BookKeeper
8. Apache Calcite
9. Apache CouchDB
10. Apache Crunch
11. Apache DataFu
12. Apache DirectMemory
13. Apache Drill
14. Apache Edgent
15. Apache Falcon
16. Apache Flink
17. Apache Flume
18. Apache Giraph
19. Apache Hama
20. Apache Helix
21. Apache Ignite
22. Apache Kafka
23. Apache Knox
24. Apache MetaModel
25. Apache Oozie
26. Apache ORC
27. Apache Parquet
28. Apache Phoenix
29. Apache REEF
30. Apache Samza
31. Apache Spark
32. Apache Sqoop
33. Apache Storm
34. Apache Tajo
35. Apache Tez
36. Apache VXQuery
37. Apache Zeppelin

Where is Apache Hadoop? Hadoop is officially under database. Most known and used are Apache Hadoop, Apache Spark, Apache Tez, Apache CouchDB, Apache Bigtop, Apache REEF. In next tier would be Apache Kafka, Apache Flume, Apache Drill, Apache Samza, Apache Storm, and the newer one Apache Edgent.

Big data is a generic terminology for all the softwares using newer strategies for large datasets. The most essential component of a big data system is processing framework. It is processing framework which computes over the data in the system.

In that way we can divide th frameworks as Batch-only frameworks (Apache Hadoop), Stream-only frameworks (Apache Storm, Apache Samza and Hybrid frameworks (Apache Spark, Apache Flink).

The reason of having so much softwares is lack of universal common way of data processing for all existing types of data. If you are new in Big Data and searching for starting with something, Apache Hadoop and Apache Spark are correct to start with. Reason is nothing but that is commonly suggested to understand the limitations.