Peter Christy

Performance is still an issue -- Velocity 2011

O'Reilly's Velocity 2011 Conference (http://velocityconf.com/velocity2011) is coming up this week. For the last couple of years it's been the scene of a very important reexamination of the issue of network-accessed application performance. Performance has been at the core of a lot of the infrastructure research we've done over the last decade and still an important area for further progress. Here's the story and why it isn't finished yet.

Over 30 years ago a seminal paper was published in the IBM Systems Journal (I'll send you a copy if you ask) in which standards for interactive program response were established to be in the range of .5 seconds (that's where the IBM data ran out). For the next 20 years, mainframe and minicomputer applications were designed to that standard.

Then the Web started to be used for commercial purposes. Early Web performance was pretty bad (20 second page load time), clearly unacceptable by the existing usability standards. But common sense said that 20 seconds was a lot better than (say) a couple of days to get something by USPS, so from that point of view 20 seconds was perfectly acceptable if less than perfect.

Then in 2001 Zona Research really messed things up publishing "research" that said, in effect, 8 seconds was OK.  What the research showed was that if 8 seconds was the commercial state-of-the-art (true at the time) and your Web pages were delivered in 8 seconds, you wouldn't suffer commercially (all true, humans are tolerant). The problem isthat for a lot of the Web generation, 8 seconds was now the "rule," not .5 seconds and that just ain't right.

Thankfully at Velocity there has been an ongoing discussion about the true value of performance and what is really adequate. Two years ago Google and Microsoft presented some real experiment data that showed, chillingly, what the effect on user behavior if just a few tenths of a second of artificial delay were inserted into search requests (video).

And last year, Google's Urs Holzle gave a keynote that went a lot further (Velocity Presentation). He said the right performance target was really .1 second, and certainly not the 6 second page load times that Google believes is typical in the US. Then he went further to point out that the issue was not the access bandwidth (fiber to the home won't fix it) and that even if all the Web performance initiatives currently underway succeeded. the results would still be sub-par (perhaps 1.5 seconds). This doesn't mean that excellent performance for internet-connected applications is impossible, but rather that they need to have some client intelligence (given the speed of light compounded by the nature of TCP/IP and the Web). It's an accurate realization that simple thin-client solutions will never suffice, but given the power in modern devices and technology like HTML5 there are lots of solution approaches possible.

So what? Performance is still an unfinished story.  There are far to many Web-based services with much less than ideal performance from the point of view of human usability.Two important tasks remain. First, we need to re-educate everyone on what excellent performance is (It turns out to be just what IBM identified in 1981.) Nothing since then has changed how humans read and think. The other open issue is getting more Web property owners to build enough performance logging and analysis infrastructure so they can see they relationship of performance to their business. In my experience when the relationship can been seen the importance of performance is clear; the issue is that too few business can measure these critical interactions.

Off to Velocity 2011!

Hadoop Gets Hotter

As if Big Data and analytics needed anything to get hotter, the space heated up with EMC's Greenplum team adding Hadoop to their portfolio. Hadoop is the Open Source project that is building a version of the MapReduce paradigm developed at Google. At the core, Hadoop includes a mechanism for distributing data to many processors and then coordinating relatively simple parallel programs run on that cluster of servers. Compared to traditional database "programming" it is much easier to do arbitrary processing on unstructured data (data that is not already in a database). With servers already cheap, Hadoop simplifies brute force programming to take advantage of cheap parallel computing. The big commercial forces in Hadoop so far have been Cloudera (a Palo Alto startup with some of the Hadoop creators, aiming to be the RedHat of Hadoop) and Yahoo who has driven much of the ongoing core development (although Carol Bartz seems interested in spinning the group out if the market is hot enough). IBM dabbles in Hadoop and has it as an adornment in their much bigger database and analytics business. EMC is interesting because they aren't a big database company; they are much more a big-data database company if you get the nuance, and the addition of Hadoop is more significant in that context. Like Cloudera and others, EMC is going to have their own Hadoop distribution (a specific version of the OpenSource code with specific testing, integration and potential for support). Where EMC expands the Hadoop horizon is by offering a variation of the GreenPlum appliance that can do Hadoop processing, probably as a front-end processing step to be followed by more conventional analysis on the Greenplum appliance.

The Facebook Cloud

I had a chance to catch up with our old friend Jay Parikh who was running the local Akamai engineering team when we first met, and now is one of the senior engineering dudes at Facebook. In addition to all the social and privacy facets of Facebook, it's a really interesting and challenging engineering task as well. Like Google, Yahoo and Microsoft, Facebook runs a huge web property and supports hundreds of millions of users. But compared to the others Facebook is a much more intertwined application (I can "friend" any of the other half-billion users and then be connected to his activity) and it is a more extendable (and extended) application. This means that Facebook has had to lead the charge in some system design and implementation dimensions like shared persistent data management, and also means that Facebook has a more structured and variable internal computer fabric because different parts of the application require very different forms of server — disk/diskless, normal memory/very large memory ...). Of course Facebook is also a learn-a-you-go experience since the drive to change is high and because they're clearly exploring new territory every day (Jay said that FB does multiple significant application updates every day), and as with the Internet, you don't really know what is going to happen for sure until you try it. Managing a slowly changing application with 500M users would be challenging enough, I can only imagine what it's like if you're re-engineering on the fly every day. How exciting! Like Google the parking is tight but if you can find a spot the food is also good and free J . pchristy@irg-intl.com

Velocity 2010 & the grim future of Web applications

The 3rd O'Reilly Velocity Conference in Santa Clara was the source of the most remarkable tidbit of all: Web apps don't work and never can. Velocity started as a meeting that focused on site operations but has become the premier Web system performance conference as well, and in the later role is doing a great job refocusing the application community on the real performance targets (not the 8 or 11 second "rules" that Zona created in the dot com period). To this point, Google's Urs Hölzle gave a paradoxical keynote talking about the current 4.5 second typical page load times that Google measures today and contrasting them with the ultimate goal of 100ms Urs promised to outline how a factor of 10 could be reached. But as the talk progressed, he only showed how with hard work we might get an improvement of perhaps 3 (1.5 seconds). Along the way he showed that the network is rarely the issue any more (except on mobile), so more bandwidth won't make a big difference here (yes, even Google's 1Gbit to the home). For years Google has been beating the drum for Web-based applications and argued that PC's and Windows are obsolete and doomed. And yet here is one of Google's best technical people presenting a quite compelling case why it won't happen that way, and why the issues are fundamental: the speed of light, the limitations of TCP/IP (it's a miracle any of this stuff still works at all if you remember it was designed for a network with 56Kbit/sec BACKBONE links), the limitations of the construction of the Internet as a network of networks, and the practical architecture of the browser as well. This doesn't mean we can't build excellent Internet connected applications. It does mean that the fundamental issues of distributed systems won't go away and that a lot of attention must be paid to the part that runs on the client.

Huge Data Centers

One of the speakers at the TechEd Analyst Summit was Dileep Blandankar, the Chief Hardware Architect of Microsoft's large web properties, and as it turns out, an old friend from our mutual DEC days and his long stint at Intel. Like Google, Microsoft has paid a lot of attention to the building of huge data centers. Like Google (and unlike Facebook apparently) Microsoft has used a container approach. The data center is constructed from a set of container-sized modules (literally shipping containers). The containers are fitted out and tested at a factory and then shipped to the data center where they are positioned and power, cooling water and the network plugged in. Whereas Google has increasingly prescriptive about the design, Microsoft chooses to push more of the engineering work down on the supplier (companies like HP or Dell for example). Like Google, Microsoft has a very informed view of what the component technology should be (both emphasize power efficient servers for example with relatively low power CPU's). Microsoft gives the vendor a spec that has key parameters such as the total power budget and how they will measure performance and challenges the vendor to optimize that while solving all the attendant design issues like cooling. Microsoft showed off their forward looking design which is a smaller module and (more importantly) designed to operate with ambient air cooling (without refrigeration) when possible. Toward this goal the module incorporates what I grew up calling a "swamp cooler" — the kind of evaporative cooler often used in desert locations for economical cooling of homes based on the temperature drop achieve by the evaporation of water. The unit on the show floor had a garden hose (literally) that got a water supply somewhere up on the ceiling of the exhibit floor. I asked Dileep if dust was a problem and he said in general not because they can measure the pressure drop across the filters and require service as needed. He said a bigger environmental problem (for example) was sulfur particulates that turned to sulfuric acid when exposed to water — spraying servers with sulfuric acid is still a very bad idea J.

Cisco's CSR-3 — When is a new router an "Internet breakthrough?"

Cisco warned the media about an "Internet breakthrough" March announcement and when they showed up announced the CSR-3 high-end router with tripled throughput. Earlier Google had been talking about implementing Gbit connectivity to the home on a limited trial basis. Journalists seemed unclear whether these were comparable announcements. Without getting into marketing correctness, it's worth spending a few minutes on the role of "faster" in very fast routers. If the fastest router is fast enough for all applications (e.g., can handle all the traffic when sitting somehow in the "middle") then everything is fine, but this isn't the case. If the fastest router can't handle all the traffic, then you build out a system of routers to do the task, and then, depending on how many you need (how much it lacks in performance), things go south fast. You waste capacity distributing traffic among the group, and as importantly you waste expensive high-speed links on the internal communication (and then of course for reliability you may well replicate the whole things). So a 3X router can make a pretty profound impact on these demanding circumstances. Is this an "Internet breakthrough?" Perhaps to a router engineer or service provider it really is.

Google Wave

The announcement at Google I/O that made the biggest waves was Wave — a provocatively interesting collaboration tool. Wave consists of a Cloud component — the distributed infrastructure that enables a worldwide set of collaborators to share a structured document (the "Wave") and a client component that actually provides the collaboration tool. It's really important to understand that Wave is not a demonstration why the Cloud obviates the need for a PC. The Wave client is implemented as a "browser" application (part of the Windows/HTML5  debate discussed briefly in the last issue). Wave wouldn't be interesting without the part that runs on your laptop. The most interesting criticism of Wave has come from Ray Ozzie. His comments at the Churchill Club were very thoughtful — if anyone appreciates a clever collaboration application it's Ray since he's been building them since college. His point is for most purposes Wave is overkill and much more complex than needed, comparing Wave to Live Mesh and its precursors Groove and Lotus Notes. We run our business on Groove, and Ray is right — for 99% of what we do Groove is sufficient.  In Live Mesh, Ray has pared down the complexity of Groove enormously because it assumes that the heavy lifting is done by Cloud services (permitting the client footprint to be smaller). To implement Microsoft's desktop/phone/Cloud integrated vision Live Mesh is much more suitable than Wave, no doubt. And there is every reason to believe that the Cloud part of Wave could be implemented in Azure and the client component in Windows far easier than in HTML5. This does not distract from the fact that Wave is one of the best examples of a "killer" Cloud app — something that will deliver real value (Ozzie's question is the breadth of the use cases compared to those that Groove or Live Mesh enable) and that absolutely could not have been developed without the integrating Cloud collaboration services.

Flashes into the Google Kimono

Google is very cryptic about everything (excessively if you ask us) creating an environment not unlike celebrity watching where a table napkin used by Julia Roberts would be a revered souvenir.   There have been a couple of Google disclosures in the last few months that are really interesting. First, in April, Google held a meeting to tell IT leaders about their power and carbon practices. Google has achieved amazing power utilization efficiency (the percentage of power that actually goes to running the IT) and continues to improve with fascinating innovation. One source of power inefficiency is battery backup, since many data center designs create battery voltage and then reconvert to data center voltage. The Google design puts a little backup battery on each server as a smart (from a systems point of view) but intuitively unexpected solution. Google authored a book (The Data Center as a Computer) that by their direction is available as a free PDF for download. It provides more insights into the design and operation of large data centers. If you want one of Larry Page's napkins you'll have to look on eBay.

Best Joke at Structure 09

In his fireside chat, Marc Benioff kept using the term aZUNE to refer to Microsoft's Cloud Platform. You have to wonder if the naming mavens thought of this when they picked Azure? The second best joke was when the Google guys were pointing at interesting things Google had published and kept saying "if you ‘Bing' for it you will find it." There were lots of Cloud puns but generally you "had to be there." BTW, Azure was delightfully well represented at the meeting and on the side reported that they are on track for announcing pricing this summer and a commercial release by the end of year (what they said at the STB analyst meeting as well). Remember that Azure isn't competing with Amazon (selling raw VM's). The MS goal is to provide the preferred Cloud solution for their 3MM developers and 400,000 business partners. So far they are on the path of under committing and over delivering, and letting the market tell them what they should focus on next.

Microsoft and Google Square Off

Google's I/O developer conference last week was much better done and much more interesting than a year ago, starting with the fact that they had abundant registration capacity (last year they had to let everyone into the keynotes and morning sessions because registration was so slow). Google is much more articulate about how their "HTML 5" Web/browser architecture now comes much closer to what they derisively call "native applications" so they don't have to use the devil word "Windows." Microsoft is visibly not yet on the HTML 5 bandwagon, but on the other hand, Microsoft in IE 8 has really toed the standards line, even to the degree of making problems for earlier MS HTML variants. So you can't really complain about Microsoft's behavior, nor is it reasonable to assume that IE will support these additional features for some time to come, making it hard to imagine "killer" applications that can't run on IE given the market share. So how this sorts out is anything but clear with the exception of the mobile space where Microsoft's impact is much more limited. Even there it's a complicated story. Google wanted to build a first class Gmail client for the iPhone, but that is explicitly prohibited by the iPhone SDK license, since it creates a directly competitive product. But the iPhone browser has enough capability that Google accomplished their goal with browser code which of course can't be restricted on an application specific basis by Apple. And just to keep everything interesting, this week Microsoft formally announced "Bing" — the new and more competitive search service. Some industry observers accuse of Microsoft of changing the definitions, but that is only fair since HTML 5 isn't really just browser code. Game certainly on, one more time.