Microsoft, Hadoop and Big Data


In a couple of weeks it will be my one year anniversary here at Microsoft  and  I couldn’t  wish for a better anniversary gift:  now that Microsoft has laid out its roadmap for Big Data, I’m really excited about the role that Apache HadoopTM plays in this.

In case you missed it, Microsoft Corporate Vice President Ted Kummert earlier today announced that we are adopting Hadoop by announcing plans to deliver enterprise class Apache Hadoop based distributions on both Windows Server and Windows Azure.

This news is loaded with goodies for the big data community, broadening the accessibility and usage of Hadoop-based technologies among developers and IT professionals, by making it available on Windows Server and Windows Azure.  

But there is more. Microsoft will be working with the community to offer contributions for inclusion into the Apache Hadoop project and its ecosystem of tools and technologies.

I believe that all of this will really benefit not only the broader Open Source community  by enabling them to take their existing skill sets and assets use them on Windows Azure and Windows Server, but also developers, our customers and partners. It is also another example of our ongoing commitment to providing Interoperability, compatibility and flexibility.

As a proud member of the Apache Software Foundation, I personally could not be happier to see how Microsoft is willing to engage in such an important Open Source project and community.

Technical Considerations

On the more technical front, we have been working on a simplified download, installation and configuration experience of several Hadoop related technologies, including HDFS, Hive, and Pig, which will help broaden the adoption of Hadoop in the enterprise.

The Hadoop based service for Windows Azure will allow any developer or user to submit and run standard Hadoop jobs directly on the Azure cloud with a simple user experience.

Let me stress this once again: it doesn’t matter what platform you are developing your Hadoop jobs on -you will always be able to take a standard Hadoop job and deploy it on our platform, as we strive towards full interoperability with the official Apache Hadoop distribution.

This is great news as it lowers the barrier for building Hadoop based applications while encouraging rapid prototyping scenarios in the Windows Azure cloud for Big Data.

To facilitate all of this, we have also entered into a strategic partnership with Hortonworks that enables us to gain unique experience and expertise to help accelerate the delivery of Microsoft’s Hadoop based distributions on both Windows Server and Windows Azure.

For developers, we will enable integration with Microsoft developer tools as well as invest in making Javascript a first class language for Big Data. We will do this by making it possible to write high performance Map/Reduce jobs using Javascript. Yes, Javascript Map/Reduce, you read it right.

For end users, the Hadoop-based applications targeting the Windows Server and Windows Azure platforms will easily work with Microsoft’s existing BI tools like PowerPivot and recently announced Power View, enabling self-service analysis on business information that was not previously accessible. To enable this we will be delivering an ODBC Driver and an Add-in for Excel, each of which will interoperate with Apache Hive. 

Finally, in line with our commitment to Interoperability and to facilitate the high performance bi-directional movement of enterprise data between Apache Hadoop and Microsoft SQL Server, we have released two Hadoop-based connectors for SQL Server to manufacturing.

The SQL Server connector for Apache Hadoop lets customers move large volumes of data between Hadoop and SQL Server 2008 R2, while the SQL Server PDW connector for Apache Hadoop moves data between Hadoop and SQL Server Parallel Data Warehouse (PDW). These new connectors will enable customers to work effectively with both structured and unstructured data.

I really look forward to sharing updates on all this as we move forward. For now, check out www.microsoft.com/bigdata and check back on the DPI blog tomorrow.

Gianugo

Comments (51)

  1. Anonymous says:

    Is it some kind of Joke or New Marketing tantrum from Microsoft after series of anti-trust law suits all over the world? Microsoft and Open Source? Are you kidding. May be Microsoft has so much free cash to hire people who can portray it as an advocate of Open Source. How about making windows 7 or office 2010 an open source if you are really into open source. Anyways both hadoop and casandra made facebook a success and all attempts from Microsoft in terms of MSN and all fall flat on its head. Is Microsoft using any open source in Bing? If so then can you post the relevant code here.

  2. Java Hater says:

    Wait, isn't Hadoop built on top of Java? I don't want coffee stains and misconfigured classpaths on my shiny .NET servers!

  3. Mat Jaggard says:

    The link you supplied just goes to the SQL 2012 advert 🙁

  4. byron miller says:

    cool stuff.  Does this mean i can run nutch on Azure too? 🙂 A few years ago i started "mozdex.org" and built a 500m page index but i broke the bank trying to pay for services i needed that were a few years ahead of its time.  oh well, i'm glad to see the techology live on. I had a blast working with the product and i'm thrilled to seee MS adopt it. I was one of the first using hadoop and glad not to be the last!

  5. Gianugo Rabellino says:

    @Java Hater: yes, it's Java indeed, and it will come with all the configuration you will expect to keep your servers shiny!

    @Mat: look in the news section – and stay tuned for more updates!

    @byron: send me a note (first dot last at microsoft dot com) if you're interested in running nutch. Or, for that matter, any other OSS on Azure. Will do my best to help make it happen.

  6. Mark says:

    @Java Hater – you can run Java by just inserting a thumbdrive and running java .. . And what is a ".NET server"? LOL

    That being sad, I guess it is nice to know that Hadoop will run on Azure and be supported on Windows/Azure. But … why would you if you have a choice?  

    @Byron – You can run Tomcat on Azure. Not sure why Nutch would not just run. techyfreak.blogspot.com/…/installing-tomcat-in-windows-azure.html

  7. Anonymous says:

    (This blog was originally posted on our Interoperability blog )
    As a proud member of the Apache Software

  8. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – Tom???s DAXOmatic Blog – Microsoft Dynamics AX – Microsoft Dynamics Community

  9. Anonymous says:

    I’m really excited to be able to give you an update on our strategy and product roadmap for Big

  10. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data | Office 365 Singapore for Business – Win-Pro

  11. Anonymous says:

    (This blog was originally posted on our Interoperability blog )
    As a proud member of the Apache Software

  12. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure | Office 365 Singapore for Business – Win-Pro

  13. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – Meng Li's Blog – Site Home – MSDN Blogs

  14. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Meng Li's Blog – Site Home – MSDN Blogs

  15. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Microsoft Lystavlen – Microsoft Dynamics CRM – Microsoft Dynamics Community

  16. Anonymous says:

    (This blog was originally posted on our Interoperability blog )
    As a proud member of the Apache Software

  17. Anonymous says:

    I’m really excited to be able to give you an update on our strategy and product roadmap for Big

  18. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – Microsoft U.S. Partner Team – Partner Community – Microsoft Dynamics Community

  19. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – Dynamics AX Sustained Engineering – Microsoft Dynamics AX – Microsoft Dynamics Community

  20. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  21. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Dynamics AX Sustained Engineering – Microsoft Dynamics AX – Microsoft Dynamics Community

  22. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Windows Virtualization Team Blog – TechNetKlub

  23. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – Exchange Team Blog – TechNetKlub

  24. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Exchange Team Blog – TechNetKlub

  25. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  26. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Windows Server Division weblog – TechNetKlub

  27. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – System Center Team Blog – TechNetKlub

  28. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure : Windows Server team Blog : The Official Microsoft IIS Site

  29. Anonymous says:

    I’m really excited to be able to give you an update on our strategy and product roadmap for Big

  30. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – System Center Team Blog – TechNetKlub

  31. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  32. Anonymous says:

    I’m really excited to be able to give you an update on our strategy and product roadmap for Big

  33. Anonymous says:

    Pingback from Simplifying Big Data Interop – Apache Hadoop on Windows Server & Windows Azure UNI Data Inc.

  34. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  35. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data : Windows Server Customer Engineering (Customer Advisory Team) : The Official Microsoft IIS Site

  36. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – Dynamics AX Sustained Engineering – Microsoft Dynamics AX – Microsoft Dynamics Community

  37. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Dynamics AX Sustained Engineering – Microsoft Dynamics AX – Microsoft Dynamics Community

  38. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  39. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data : Erez Benari's Blog : The Official Microsoft IIS Site

  40. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  41. Anonymous says:

    I’m really excited to be able to give you an update on our strategy and product roadmap for Big

  42. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure | The City Technology Group Blog

  43. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  44. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Information Store – ExchangeBlog – TechNetKlub

  45. Anonymous says:

    (This blog was originally posted on our Interoperability blog ) As a proud member of the Apache Software

  46. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – Safranka M??ty??s szakmai blogja – TechNetKlub

  47. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure – System Center Mindenkinek – TechNetKlub

  48. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data – System Center Mindenkinek – TechNetKlub

  49. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure « NFL Free Agency News

  50. Anonymous says:

    Pingback from Simplifying Big Data Interop ??? Apache Hadoop on Windows Server & Windows Azure « NFL Free Agency News

  51. Anonymous says:

    Pingback from Update: Microsoft, Hadoop and Big Data | UC3

Skip to main content