Be a Dooper!

Tag Archives: cloud

Accessing Google Storage from Spark

Posted on by

It’s been a while since my last post at beadooper.  I have had my head in the clouds…really.  Cloud computing has transformed the big data landscape.  Originally the cloud was AWS, and then Azure came along, and now there’s not only the OpenStack stack, but also Google, Rackspace, IBM, and the list goes on.

I’ve already covered Azure and OpenStack in previous posts.  This post I’ll cover how to get to Google Storage from Google Cloud.  The pattern is similar to Azure and OpenStack in that you need a “connector” library and some configuration settings.  In this case, the connector doesn’t come with Hadoop, and you need to get it separately.

And now…Google Storage…

Continue reading →

Accessing Swift storage from Spark

Posted on by

The rise of cloud technologies has resulted in a drive toward the separation of compute resources from storage resources.  Two popular options are Swift (the OpenStack object storage platform) and Azure Blob Storage.  The combination of these technologies allows an organization to think about storage and compute as two different items and plan their budget spends accordingly, since the costs of compute resources tends to be way more expensive than storage resources.

This is the second of two posts discussing Spark and cloud storage.  This post will discuss using Spark to access OpenStack Swift storage.

Continue reading →

Accessing Azure Blob Storage from Spark

Posted on by

The rise of cloud technologies has resulted in a drive toward the separation of compute resources from storage resources.  Two popular options are Swift (the OpenStack object storage platform) and Azure Blob Storage.  The combination of these technologies allows an organization to think about storage and compute as two different items and plan their budget spends accordingly, since the costs of compute resources tends to be way more expensive than storage resources.

This is the first of two posts discussing Spark and cloud storage.  This post will discuss using Spark to access Azure blob storage, and the second will be focused on OpenStack Swift storage.

Continue reading →

How to Connect Hadoop to an Azure storage account

Posted on by

With the 2.7.0 release of Hadoop, Hadoop now includes the ability to connect to a Windows Azure storage account.  There are plenty of advantages to this, including the ability to tap into the resources of Azure for storage.  To be fair to other cloud providers, there is also support for Amazon S3 and OpenStack Swift, but this post is specifically to discuss Azure. Continue reading →