Be a Dooper!

  • Home
  • Hadoop
  • Cassandra
  • Hadoop Reference
    • core-site.xml (Hadoop 2.7)
    • core-site.xml (Hadoop 2.6)
    • hdfs-site.xml (Hadoop 2.7)
    • hdfs-site.xml (Hadoop 2.6)
    • hive-site.xml (1.1)
    • hive-site.xml (1.0)
    • hive-site.xml (0.14)
    • mapred-site.xml (Hadoop 2.7)
    • mapred-site.xml (Hadoop 2.6)
    • oozie-site.xml (4.0)
    • yarn-site.xml (Hadoop 2.7)
    • yarn-site.xml (Hadoop 2.6)
    • Older Versions
  • About

Tag Archives: opensource

Adding Spark to your Single Node Cluster

Posted on March 2, 2014 by dmontroy

Apache Spark is all the rage now in the Hadoop ecosystem.  It has recently become an Apache top level project, and many people are looking at it as a successor to MapReduce.

You probably want to try this out yourself in your cluster.  There are some nuances of Spark that are difficult to figure out.  Here’s how you can get past them and start figuring out what Spark is all about and what it could do for you in your cluster.

Continue reading →

Posted in Hadoop, YARN | Tags: opensource, singlenode, spark | Leave a comment |

Apache Hadoop 2.3.0 Released!

Posted on February 24, 2014 by dmontroy

The Apache group has given approval for the release of Apache Hadoop 2.3.0.  It is available for download at your favorite mirror.  It includes:

  • Support for Heterogeneous Storage hierarchy in HDFS.
  • In-memory cache for HDFS data with centralized administration and management.
  • Simplified distribution of MapReduce binaries via HDFS in YARN Distributed Cache.

…along with the expected bug fixes.  One particularly visual difference is in the Name Node web UI.  It has been improved to be more visually appealing.

If you have been following my instructions to set up your own cluster, not to worry…I verified that they also work for this version.

Posted in Hadoop | Tags: beginner, opensource, singlenode | Leave a comment |

Setting up Hive

Posted on February 4, 2014 by dmontroy

Hive is one of the most popular components of the Hadoop ecosystem…a Hadoop system seems almost bare without it.  It provides a good jump start with Hadoop, especially for those with previous SQL experience; however, as you grow in your experience with Hadoop, you’ll come to realize that it isn’t the most optimal tool for your Hadoop jobs.  But that’s a story for another post…it remains a great way to get started with any kind of job in Hadoop.  On to the instructions! Continue reading →

Posted in Hive | Tags: hive, opensource, singlenode | Leave a comment |

Adding MapReduce support to YARN

Posted on February 2, 2014 by dmontroy

By now I’ve shown you how to install a single node Hadoop cluster.  This configures the cluster with HDFS and YARN functionality, but you may have noticed that submitting a MapReduce job doesn’t show anything in the YARN resource manager.  If you are trying to understand how MapReduce interacts with YARN, this doesn’t help you…and it breaks the principle we’ve been trying to follow of trying to set up a cluster that works like a regular cluster that just happens to be on one node.

This post will show you the steps you need to set up MapReduce support in YARN in your cluster. Continue reading →

Posted in Hadoop, MapReduce, YARN | Tags: opensource, singlenode | Leave a comment |

Getting Started with Hadoop

Posted on February 1, 2014 by dmontroy

Here’s how you can get started with your first Hadoop cluster.  These instructions will walk you through the process of getting started with Hadoop using:

  • A Linux server with OpenSuSE installed (12.3 was used here) in text mode
  • Apache Hadoop 2.2.0

This will get you started using a single node cluster in pseudo distributed mode.  The benefits of this approach is that it is quite similar to how a fully distributed Hadoop cluster will work, except it just happens to be running on only one server. Continue reading →

Posted in Hadoop | Tags: beginner, opensource, singlenode | Leave a comment |

Categories

  • Building (1)
  • Cassandra (2)
  • Cloud (4)
  • Flume (1)
  • Hadoop (14)
  • Hive (2)
  • MapReduce (1)
  • Oozie (1)
  • Other (1)
  • Spark (5)
  • YARN (3)

Archives

  • January 2018 (1)
  • May 2016 (1)
  • April 2016 (1)
  • December 2015 (2)
  • July 2015 (1)
  • June 2015 (1)
  • April 2015 (2)
  • January 2015 (2)
  • November 2014 (1)
  • October 2014 (1)
  • July 2014 (1)
  • June 2014 (1)
  • April 2014 (2)
  • March 2014 (1)
  • February 2014 (5)
  • Prev
  • 1
  • 2
© Be a Dooper!