Peter Christy

The Hadoop Summit -- The Plot Thickens

The Hadoop Summit this week was a bang up success by any simple measure (like attendance or enthusiasm). The cornerstone announcement was the formation of Hortonworks by Yahoo and Benchmark Capital, spinning out key Yahoo Hadoop resources. Like all Hadoop commercial efforts there are some delightful twists (Cloudera reasonable suggested some clarifications below):

• Like Cloudera, Hortonworks will offer Hadoop support services.
• Unlike Cloudera, Hortonworks will contribute all their developed code back into the Apache Hadoop distribution project (Cloudera has their own distribution; Cloudera contributes all Apache Hadoop changes back to Apache and develops proprietary management modules as well ).
• Very unlike Cloudera, Hortonworks will specifically invest in the management capabilities that is the "secret sauce" by which Cloudera differentiates the not-free Cloudera Enterprise Edition and contribute these additions back to the Apache project.
• For Cloudera, Hortonworks represents incremental validation of the market (good news). But Hortonworks sure smells like a serious competitor to me.

For me the critical business question is how to make money with Hadoop (clearly as a free tool Apache Hadoop provides great value). Since most of the Hadoop dialog is the enthusiasm, I've always felt like a cranky voice and wet blanket,even though how to make money is something everyone in the community should be concerned about because ultimately it is that revenue stream that will really define what Hadoop becomes. So I was just delighted by IBM's Anant Jhingran's keynote (CTO, Information Management) because Anant spoke directly to exactly these issues.  I strongly encourage everyone who wasn't there to watch when the video goes up.

Anant painted what to me is a remarkably insightful view of Hadoop and how it fits into the existing world of analytics, and from that perspective how to better understand the business issues. IBM clearly has an important analytics business already, so it's not surprising that Anant doesn't say that SQL is dead and that Hadoop and nonSQL will take over the world. So you could cynically say it's just legacy company FUD. I think not: I think it's the wisdom of someone with a lot of experience in the business.

Some of his key points (to me) include these:

• Don't get hung up on "big".  Hadoop was key to IBM's Watson solution but after the set-up the core database was just 6 TB which isn't that big ("volume" isn't everything).
• Remember that most analytic solutions use a diversity of data ("variety is important").
• Remember that for some solutions the ability to ingest rapid streams of data is an important element ("velocity").  He showed a slide in which IBM had ranked various important analytics solution are decomposed by the degree to which Variety, Volume and Velocity are important. Anant didn't take the next step and describe the role Hadoop should and can plan, but it certainly seems like a valuable framework for considering that problem.
• He colorfully described the Hadoop community as consisting of "birthers" (those on the leading edge, pushing the boundary with Hadoop) and "adopters" (the larger BI community that would like to integrate Hadoop into their existing and valuable solutions). He reasoned that these quite different groups needed to find a way of working together and working to the broad common good (rather than splintering into factions).
• Anant also pointed out the hidden obvious truth (the ignored elephant in the tent if you forgive the egregious pun): it's really hard to make money from Open Source.

Like many said in one way or another, 2011 is clearly a watershed time for Hadoop and Big Data. A lot of the complexity in the paths to be taken and the decision to be made surfaced at the Hadoop Summit. Now to just consult my crystal ball....

"Ticket to Ride"

The Butcher, Baker and Candlestick-Maker Clouds? In the most recent IBM's quarterly Cloud updates for the analyst community, a lot of the discussion was focused on the idea of an industry specific Cloud. IBM provided substance and the focus is quite different from an "Amazon" discussion. The idea is to create synergies with real monetary return when the Cloud tenants share an industry focus: (1) Basic workflow synergies: industries share common subsystems and use common middleware. Consider the advantages if all of this already exists as a service in a Cloud platform. (2) Application structure synergies: For some time IBM has been working to standardize application templates. For all commercial applications there are only a few tens of general templates required, and within a particular industry, only a subset of those. IBM believes there is great value in herding customers and application suppliers toward the use of these standard forms. An industry specific Cloud will include specific support for these templates (application componentization, industry specific middle ware, industry specific SaaS elements). This in turn would reduce the cost and complexity of the applications and their use, and increase the opportunities for cost-effective integration. (3) Security and Compliance: Security and compliance are usually given as inhibitors to the use of Cloud solutions, but IBM argues (quite correctly) they can be solution accelerators. An industry specific Cloud would (by definition) encompass applications and services that were all subject to the same externally imposed security and compliance requirements. IBM will provide a technical and management architecture that is designed to meet these requirements (IBM has extensive experience in this because of all the architecture and development services performed for their customers). Just as Terremark argues correctly that they win Government business because of the security services they imbed in their Cloud offering, IBM is right in seeing this as a real plus rather than a minimized liability. All in all this is a very interesting offering especially building on IBM's business franchise. It wouldn't make much sense for Amazon to offer industry-specific Clouds but neither would it make much sense for IBM to offer an Amazon like service since they choose to focus on higher contribution components.

Autonomic Computing (Redoux)

In the last issue I wrote about how VMware's acquisition of SpringSource gives them a way to interact directly with application developers, and how valuable a framework may turn out to be in Cloud and Utility computing as a way of solving important infrastructure optimization issues (like edge acceleration) without requiring the application developer to understand all the nasty bits. VMware has something even more important in mind I think: what their Analyst Day talks hinted at was that they thought that applications built on the framework could be instrumented and automatically managed as a result. If you remember, IBM tried "Autonomic" computing some time ago, and couldn't really make it work. It seems pretty clear that an executable binary is much too cryptic to make sense of at run time for broader management purposes. Things improve with virtualization because it's much easier to see what the application is doing (if not to understand it). With a framework, things improve still more because you see more of the program structure at the framework abstraction layer, and as VMware notes, you can instrument the framework to give you more useful information about what the program is doing. Are these changes enough to make something like autonomic computing really work? Remember, the goal is a smart data center than automatically dynamically adjusts to accommodate evolving load, hardware and software failure, issues of scaling, ... Our friends at Microsoft would argue I think that this isn't going to work for VMware. Microsoft thinks that by the time you get an executable program all the interesting information about the program (the design intent, how it performs and scales, how you can tell it's in trouble) has been removed or encrypted beyond all hope. Microsoft's approach (Dynamic IT) is based on the idea of capturing and preserving key design and operational data in a declarative form, and using all this in the data center to achieve the goal. Virtualization changes the game enough to make VMware's direction worth looking into, but if you asked me now, I wouldn't bet the rent money they can make it work anymore than IBM could.

Rising Clouds

Cloud announcements don't seem to have been slowed much by the economy. It is still a very complicated and murky topic but there is no doubt that there is something important underneath. Recently VMware introduced the fourth generation of server virtualization infrastructure (now V-Sphere) as a Cloud operating system. At MMS, Microsoft spent a lot of time talking about Cloud as well. We think that Microsoft does the best job positioning Cloud within the spectrum of IT technology and architecture (although it's of course self serving since MS would have the most to lose if the world were to migrate rapidly to primarily Cloud models) and between Windows Server , System Center, Azure, Office 14 and Live, Microsoft is investing heavily in Cloud solutions. VMware still has a clear lead in exploitation of virtualization although there seems to be growing agreement that virtualization is in fact just one pillar of the solution. HP and IBM both are strong players for sure. It will be interesting to see what Google says at their upcoming developer meeting.

Cisco Data Center Competitors

It's interesting times in the data center. Cisco is brewing "California" -- a data center offering that is rumored to include servers, and not surprisingly the competitors are showing signs of friskiness as well ("the best defense is a good offense!"). First HP held a somewhat cryptic introduction of ProCurve partners. Grotesquely oversimplified, HP believes they can build on their large server business and provide switches that are price competitive and integrate well with HP servers. Further, HP believes that because they are a minority vendor (the #2 market share leader but with a miniscule share compared to Cisco) they have to do better at compatibility and partnership (that part must be true) but by meeting those obligations they produce a product with clear TCO advantages over Cisco (the more intriguing claim). Following soon thereafter Juniper held its Analyst Meeting featuring the new team, and as part of that previewed tiny bits of what they call Project Stratus being done in cooperation (of some form) with IBM. Juniper argues they can build switches that are as fast as Cisco (because of their very large system SP business), and at the root of Stratus is a large and flat switch, probably somewhat like what Google is building internally (pure speculation) and maybe like what Andy Bechtolsheim did at Sun (the new Juniper advanced technology leader, David Yen, comes from a long career at Sun). Like HP, Juniper says that it will partner better than Cisco and that will bring TCO value to their customers. They also believe that their cleaner JUNOS software base (one O/S shared by all products; quarterly releases) gives them advantage, specifically the ability to manage the data center fabric as if it were one big switch, again with TCO advantages for the customer. One key argument from both HP and Juniper is that Cisco's complexity and inconclusive progress in network management yields a cost-of-ownership liability competitors can exploit, especially in the "more for less" recessionary time. Cisco's big California shoe is of course yet to drop (although it is starting to block the sun J ).

As The Chip Turns

Steve Jobs might be strange but he sure isn't stupid.  When Apple moved from IBM to Intel for CPU's for the Mac a lot of people wondered whether this was in the best interests of the Mac cultists since Intel iconoclasm was always part of that. Apple understood that since Intel now had the power minimization religion the Intel processors would overtake IBM in power efficiency. It turns out there are other powerful benefits. When Intel brings out a new CPU now it tends to have architectural visibility (e.g. more cores, virtualization capability, …). The pipeline between Intel and the customer on the Windows side is long and hard often requiring some support in Windows and then by the ODM and computer vendor.  In contrast, here in Silicon Valley Apple and Intel have a small team of bright engineers and if it is to Apple's benefit they can easily be the first to market with a MAC that leverages a new Intel CPU. Intel loves it because the not-to-price-sensitive MAC purchaser is a great way to take advantage of Intel's newest. So expect to see some share growth for Apple in areas like laptops especially given the fact that you can run Windows virtualized if you want.  And speaking of competition, we would guess that AMD is in for a pretty hard time now that the CPU competition has moved to mine-has-more-cores-than-yours (naa, naa, naa!). AMD is better than Intel at many things but getting new processes into volume production is not and is not likely to be one of them. Smaller features and more transistors is the way you get more cores, and Intel seems destined to win those battles for the foreseeable future.

IBM buys ISS; Whither Cisco?

For the last month or so various of our network confidants have been asking what we thought of the rumor that IBM would buy ISS.  My (Peter) consistent answer was to doubt that it would happen because of the potential impact on the strategic enterprise security partnership with Cisco. My conclusion was clearly wrong but why?  First, the purchase was by the managed infrastructure part of IBM, not the enterprise part or even Global Services. But still, when the analyst call was done product integration with Tivoli and the earlier MicroMuse acquisitions were discussed, and the Cisco relationship didn't even come up. As background, remember that a couple of years ago IBM did some really nice work identifying generic patterns in security solutions and defining best practices templates for those solutions. As part of that work IBM identified Cisco as a key partner because many of the application security control points were in the network and not on IBM controlled systems.  We still think that Cisco will be a network equipment provider of choice for IBM enterprise security solutions (given Cisco's 70+% share in enterprise networking that is bound to be true anyway).  But remember also that a key objective for Cisco is grabbing more enterprise IT budget share, and security is one of the most promising growth vehicles. The ultimate revenue division question is less who sells the network equipment but rather who sells the management and control systems. If those systems just came with the network devices it would be Cisco's business to lose. But it seems more and more that those solutions look like computer applications (e.g. the Cisco NAC server) in which case Cisco isn't necessarily the most likely server application vendor, especially if the solution has more integration "upward" (e.g. into identity and configuration data) than "downward" to the network devices. So maybe it does mean something for the Cisco IBM relationship, after all, if you "follow the money" as is our want.