I mentioned this before in my general setup instructions. With the second release of Hadoop this year, it’s now time to talk about how to build the native Hadoop libraries. Here are the instructions you need to build the native libraries.
Why is This Necessary?
If you recall, you see a message similar to this one if you are running on a 64 bit server using the Apache distribution without modification:
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/share/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'. 14/02/01 17:02:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
To get around this message, you need to build the libhadoop library for your system. While the procedure to do this isn’t completely obvious, it’s also not that difficult.
To get started, I started with a system that had this setup:
- Minimal server (text mode) install of Linux (in my case, I used OpenSuSE 12.3)
- Basic dev tools already installed: gcc, make
- Kernel source installed
- Java 1.7 installed
- Maven 3.1.1
If you start with this setup, you also need to install these components:
First Step: protobuf 2.5
Building the native libraries requires using protobuf 2.5. You may not have this version in your system. You will need to download and build it yourself. You can get the download from https://developers.google.com/protocol-buffers . Download version 2.5, which is the latest version as of this post.
To build protobuf, run these commands from the main protobuf directory:
> ./configure > make
Once the build has finsihed, run this command to execute the unit tests and verify that protobuf was built successfully:
> make check
Look for this in the output:
================== All 4 tests passed ==================
If you see this, then protobuf was built successfully, and you can move on to building the Hadoop libraries.
Second Step: Building the Hadoop Libraries
To build the Hadoop libraries, start off with a Hadoop distribution archive. (I used Hadoop 2.4 for this post.) Extract the archive, then move into the hadoop-common-project/hadoop-common directory:
> cd hadoop-common-project/hadoop-common
Before building, you need to define the location of protoc in the protobuf code:
> export HADOOP_PROTOC_PATH=[path to protobuf]/src/protoc
From this directory, use Maven to build the native code:
> mvn compile -Pnative
Look for the typical Maven BUILD SUCCESS message to indicate that you have built the libraries properly:
[INFO] -------------------------------------------- [INFO] BUILD SUCCESS [INFO] --------------------------------------------
Maven will generate the libraries in target/native/target/usr/local/lib .
Final step: Copying the libraries into Hadoop
Once the libraries are built, all you need to do is copy them to your Hadoop installation. If you have been following the instructions to set up a cluster on this site, that path is /usr/share/hadoop . Copy the files as the hdfs user since that user has permissions to write to the Hadoop installation:
hdfs> cp target/native/target/usr/local/lib/libhadoop.a /usr/share/hadoop/lib/native hdfs> cp target/native/target/usr/local/lib/libhadoop.so.1.0.0 /usr/share/hadoop/lib/native
A third file, libhadoop.so, does not need to be copied since it is just a symbolic link to libhadoop.so.1.0.0.
Checking It All Out
As a final check, once you put the libraries in place, run a Hadoop HDFS command and verify that you no longer get the native library warning.
That’s all you need to do!