Skip to content
African Wealths Development
← Newsroom

Insight2 min read

As Big Data grows in importance, the tools built for it keep evolving. Organisations use tools such as Hadoop, Hive, Cassandra, Spark or Kafka depending on their needs.

As Big Data grows in importance, the tools built for it keep evolving and improving. Organisations use tools such as Hadoop, Hive, Cassandra, Spark or Kafka depending on their needs. There is a huge number of solutions, many of them open source. A foundation, the Apache Software Foundation (ASF), supports many of these projects. Given how important these tools are for Big Data, let's briefly discuss some of them.

One of the best-known tools for Big Data analysis is Apache Hadoop, an open-source framework for storing and processing large datasets. Apache Spark is another tool with many followers. One of Spark's great advantages is that it can keep much of the processing data in memory (RAM) as well as on disk, which can be much faster — around 10 GB per second versus an average of 600 MB for Hadoop. Spark is compatible with Hadoop (Hadoop Distributed File System), Apache Cassandra, OpenStack Swift and many other data storage solutions. One of its most interesting features is that it can run on a single local machine, which makes it much easier to use.

There is also Apache Kafka, which lets users publish and subscribe to real-time data streams. Kafka's main goal is to bring the reliability of other messaging systems to streaming data.

Other Big Data tools

  • Apache Lucene can be used for all kinds of recommendation engines, as it relies on full-text indexing and search libraries.
  • Apache Zeppelin is an incubating project that enables interactive data analysis with SQL and other programming languages.
  • Elasticsearch is more of an enterprise search engine. Its greatest advantage is that it can generate insights from both structured and unstructured data.
  • TensorFlow is a software library that is attracting more and more attention, as it is used for machine learning.

Big Data keeps growing and evolving, and so do the tools. As mentioned, some tools work with structured data, others with unstructured data.

In the next article, we will look at these notions of structured and unstructured data. In the meantime, if you have just joined us, you can read the first part of this series.

Newsletter

Enjoyed this article? Get the next ones.

A selection of our insights, once a month, straight to your inbox.

One email a month, no advertising. One-click unsubscribe. Privacy

Request a quoteWhatsAppCall