Peter Christy

Open Source and The Cloud -- The Conundrum(s)

Open Source has always fascinated me. On the one hand it make so much sense; on the other hand it requires imagining a conversation between Karl Mark and John Doerr. Up until recently at least the economic model was understandable:

• Open Source offered free functionality for individuals or students who couldn't afford it otherwise (especially in a modifiable or extensible form).
• It was also free to the largest users (like Amazon using Xen), since they had to support it themselves anyway.
• There was a business to be made supporting the users in middle who were too small to support it themselves but too dependent on correct operation to responsibly go without support.
• It was also possible to add your own proprietary and separate modules, and sell them as separately licensed software (the rules say that improvements to the base software have to be contributed back to the community).

 A recent encounter with old friend Chris Keene (most recently CEO of WaveMaker, now a strategist at acquirer VMware) added new facets to fhis problem (see his earlier post).

• There is a loophole in the basic GPL license that says that if you use Open Source to deliver a service, but you don't sell or distribute the software, you don't have to contribute your changes back to the community.
• When Open Source functionality is delivered as part of a Cloud service (rather than as software), the support proposition vanishes -- you get reliable use in the form of a service but you don't have to maintain it yourself. So as more use moves to the Cloud, and is delivered as a service, the established Open Source economy crumbles.

Ah, but things get stranger still if you think that OpenStack and Hortonworks are leading indicators. Before OpenStack, NASA was using Eucalyptus which consisted of the Open Source bits and the proprietary bits. NASA wanted the proprietary functionality as part of the Open Source base and when that negotiation failed they partnered with RackSpace and just did it themselves. Hortonworks is taking the same approach: they think the management functions that Cloudera adds to Hadoop are valuable base functions so they are going to add that functionality right into base distribution and then give it back for free to the community.

So the big players, using Open Source, have special priviledges (not required to contribute additions back to the community) and can also destroy the part of the Open Source business model that leverages proprietary additional software.

Hmmm... Sounds to me like Karl Marx (or maybe Stalin) 1, John Doerr 0, at the end of this inning.

 

The Hadoop Summit -- The Plot Thickens

The Hadoop Summit this week was a bang up success by any simple measure (like attendance or enthusiasm). The cornerstone announcement was the formation of Hortonworks by Yahoo and Benchmark Capital, spinning out key Yahoo Hadoop resources. Like all Hadoop commercial efforts there are some delightful twists (Cloudera reasonable suggested some clarifications below):

• Like Cloudera, Hortonworks will offer Hadoop support services.
• Unlike Cloudera, Hortonworks will contribute all their developed code back into the Apache Hadoop distribution project (Cloudera has their own distribution; Cloudera contributes all Apache Hadoop changes back to Apache and develops proprietary management modules as well ).
• Very unlike Cloudera, Hortonworks will specifically invest in the management capabilities that is the "secret sauce" by which Cloudera differentiates the not-free Cloudera Enterprise Edition and contribute these additions back to the Apache project.
• For Cloudera, Hortonworks represents incremental validation of the market (good news). But Hortonworks sure smells like a serious competitor to me.

For me the critical business question is how to make money with Hadoop (clearly as a free tool Apache Hadoop provides great value). Since most of the Hadoop dialog is the enthusiasm, I've always felt like a cranky voice and wet blanket,even though how to make money is something everyone in the community should be concerned about because ultimately it is that revenue stream that will really define what Hadoop becomes. So I was just delighted by IBM's Anant Jhingran's keynote (CTO, Information Management) because Anant spoke directly to exactly these issues.  I strongly encourage everyone who wasn't there to watch when the video goes up.

Anant painted what to me is a remarkably insightful view of Hadoop and how it fits into the existing world of analytics, and from that perspective how to better understand the business issues. IBM clearly has an important analytics business already, so it's not surprising that Anant doesn't say that SQL is dead and that Hadoop and nonSQL will take over the world. So you could cynically say it's just legacy company FUD. I think not: I think it's the wisdom of someone with a lot of experience in the business.

Some of his key points (to me) include these:

• Don't get hung up on "big".  Hadoop was key to IBM's Watson solution but after the set-up the core database was just 6 TB which isn't that big ("volume" isn't everything).
• Remember that most analytic solutions use a diversity of data ("variety is important").
• Remember that for some solutions the ability to ingest rapid streams of data is an important element ("velocity").  He showed a slide in which IBM had ranked various important analytics solution are decomposed by the degree to which Variety, Volume and Velocity are important. Anant didn't take the next step and describe the role Hadoop should and can plan, but it certainly seems like a valuable framework for considering that problem.
• He colorfully described the Hadoop community as consisting of "birthers" (those on the leading edge, pushing the boundary with Hadoop) and "adopters" (the larger BI community that would like to integrate Hadoop into their existing and valuable solutions). He reasoned that these quite different groups needed to find a way of working together and working to the broad common good (rather than splintering into factions).
• Anant also pointed out the hidden obvious truth (the ignored elephant in the tent if you forgive the egregious pun): it's really hard to make money from Open Source.

Like many said in one way or another, 2011 is clearly a watershed time for Hadoop and Big Data. A lot of the complexity in the paths to be taken and the decision to be made surfaced at the Hadoop Summit. Now to just consult my crystal ball....

Hadoop Gets Hotter

As if Big Data and analytics needed anything to get hotter, the space heated up with EMC's Greenplum team adding Hadoop to their portfolio. Hadoop is the Open Source project that is building a version of the MapReduce paradigm developed at Google. At the core, Hadoop includes a mechanism for distributing data to many processors and then coordinating relatively simple parallel programs run on that cluster of servers. Compared to traditional database "programming" it is much easier to do arbitrary processing on unstructured data (data that is not already in a database). With servers already cheap, Hadoop simplifies brute force programming to take advantage of cheap parallel computing. The big commercial forces in Hadoop so far have been Cloudera (a Palo Alto startup with some of the Hadoop creators, aiming to be the RedHat of Hadoop) and Yahoo who has driven much of the ongoing core development (although Carol Bartz seems interested in spinning the group out if the market is hot enough). IBM dabbles in Hadoop and has it as an adornment in their much bigger database and analytics business. EMC is interesting because they aren't a big database company; they are much more a big-data database company if you get the nuance, and the addition of Hadoop is more significant in that context. Like Cloudera and others, EMC is going to have their own Hadoop distribution (a specific version of the OpenSource code with specific testing, integration and potential for support). Where EMC expands the Hadoop horizon is by offering a variation of the GreenPlum appliance that can do Hadoop processing, probably as a front-end processing step to be followed by more conventional analysis on the Greenplum appliance.