Apache Spark is all the rage now in the Hadoop ecosystem. It has recently become an Apache top level project, and many people are looking at it as a successor to MapReduce.
You probably want to try this out yourself in your cluster. There are some nuances of Spark that are difficult to figure out. Here’s how you can get past them and start figuring out what Spark is all about and what it could do for you in your cluster.
The Apache group has given approval for the release of Apache Hadoop 2.3.0. It is available for download at your favorite mirror. It includes:
…along with the expected bug fixes. One particularly visual difference is in the Name Node web UI. It has been improved to be more visually appealing.
If you have been following my instructions to set up your own cluster, not to worry…I verified that they also work for this version.
Hive is one of the most popular components of the Hadoop ecosystem…a Hadoop system seems almost bare without it. It provides a good jump start with Hadoop, especially for those with previous SQL experience; however, as you grow in your experience with Hadoop, you’ll come to realize that it isn’t the most optimal tool for your Hadoop jobs. But that’s a story for another post…it remains a great way to get started with any kind of job in Hadoop. On to the instructions! Continue reading →
By now I’ve shown you how to install a single node Hadoop cluster. This configures the cluster with HDFS and YARN functionality, but you may have noticed that submitting a MapReduce job doesn’t show anything in the YARN resource manager. If you are trying to understand how MapReduce interacts with YARN, this doesn’t help you…and it breaks the principle we’ve been trying to follow of trying to set up a cluster that works like a regular cluster that just happens to be on one node.
This post will show you the steps you need to set up MapReduce support in YARN in your cluster. Continue reading →
Here’s how you can get started with your first Hadoop cluster. These instructions will walk you through the process of getting started with Hadoop using:
This will get you started using a single node cluster in pseudo distributed mode. The benefits of this approach is that it is quite similar to how a fully distributed Hadoop cluster will work, except it just happens to be running on only one server. Continue reading →