Dark clouds

I heard Phil Windley and Scott Lemon interview Andre’ M. DiMino of the Shadowserver foundation on Phil’s Technometria podcast at IT Conversations. Andre’ describes the organizations efforts to track bots and botnets across the internet.  These are computers, servers numbering in the 1000s and personal computers numbering in the neighborhood of 750,000 at this writing, that are compromised with nefarious software, aka malware, that can be tasked to do the bidding of the botnet commander.  Botnets are well explained at Shadowserver.

Make no mistake, this is an example of utility computing.  Except the owners of these computers have been duped into letting their machines be conscripted to do the deeds of the commanders.  These deeds are often illegal, like SPAMming.  From previous posts, you learned I am no fan of the label Cloud Computing.  If we did apply the cloud metaphor, this use would be the dark side.

Also make no mistake, the people who create these bot networks are clever and resourceful.  And while I have argued that abundant computing is almost free, it is really free for these people.  Isn’t it interesting that the dark side provides early adopters and innovative exploits of this new resource?

Economics of SPAM

Phil Windley also mentioned a fascinating article by BBC, Study Shows How Spammers Cash In on SPAM.  The article reports a study conducted by a group of researchers at the University of California, San Diego, led by Stefan Savage.  They hijacked a subset of a bot network (yes, there is no honor…), inserted their harmless SPAM directing customers to a fake pharmacy site appearing to sell a herbal supplement for libido enhancement, and counted.  Here is the summary:

• 75,869 computers were hijacked
• over the course of 26 days, 350 million SPAM emails were relayed
• 28 user clicks resulted, that would have been sales had the harmless website processed them (they didn’t) for a total of $2,732 or an average order of $98.

The paper describing the study is here.

28 sales per 350 million emails is a yield of 0.000008% or 1 per 12.5 million emails. $105 revenue per day. Use of 75,869 computers for $2732, or the use of more than 27 computers per $1 revenue. Remember the researchers only hijacked a subset of the bot network, which they estimated to be 1.5% of the total.  The researchers estimate the full network would yield $7000 revenue per day.

Pause and reflect on how much free resource the botnet commanders exploited for modest gain.

Post office sees less mail volume, in part due to the internet

Also this week, the Postal Service reports a reduction in mail volume, in part to what Postmaster General John E. Potter explains as,  “a revolution in the way people communicate has structurally changed the way America uses the mail,”  and offered as part of the rationale for reducing some mail delivery from 6 days to 5 days per week.

This news arriving the same week as the SPAM economics above, prompted me to do some back-of-the-envelope calculations comparing the economics of a direct mail campaign versus a botnet SPAM relay campaign.  Granted, I am not including the case of a legitimate (!?) bulk email campaign.

Turns out I can’t responsibly make the comparison.  Sure, I ran the calculations but there is no rational market for a 350 million piece direct mail campaign.  Some things do jump out.

Asymmetric Costs

The cost of the message payload in the case of direct mail, the printed envelope and contents, has approached some low-cost asymptote from years of cost pressures, but is still probably in the neighborhood of $0.22.  If 350 million pieces were rational to send, the payload cost would be in the tens of millions.  The weight at 0.5 ounces per piece would imply well over 5000 tons.  Of course, the cost of the payload in the case of email is nil.

Also jumping out is the cost of transport, effectively zero in the case of SPAM.  According to the calculator at the postal service website, we can expect another $0.22 for postage per piece of direct mail.  Again, tens of millions in expense.

These wildly different costs drive wildly different behaviors.  Free or almost free means you can afford to oversend, without any regard to being selective about it.

Not so great versus terrible response rates

The direct mailers tell us to expect a 2.15% response rate for a well-designed campaign.  That’s not so great, but the business case works and lots of companies and products depend on it. I suspect the 2.15% rate holds with the typical direct mail campaign in the range of 10s of thousands of pieces, not 100s of millions.

The 0.000008% response rate reported by UCSD study is terrible, something like 250,000x worse than direct mail. The business case works too, although most classical market analysis would classify it as a niche. I like niches.  Chris Anderson would call it the far end of the long tail.

I must point out, that as much as I hear people protesting SPAM, the reason we have it is that it works.  There are a few people out there who will buy in response to it.

Implications for using the Cloud

Developing and prospering from niches is not new. Realizing the benefits of utility computing is not as far off as you may think.  Clever people exploiting the dark side of the Cloud have been doing it for years.  Check your inbox.

Time for the rest of us to step up and participate. db

Point is, utility computing (the preferred, more descriptive label than cloud computing in this case) solved a juicy new problem, not some warmed over set of requirements well-served by Times’ current systems.

When you read Clayton Christensen’s Innovator’s Solution, you’ll find fascinating models which describe the utility of a product and they may help us select the right problems to solve with cloud computing.  Christensen expertly develops the hypothesis that any new product introduced that fits on the current trajectory – the continuum of functionality vs. utility – is subsumed by the incumbent suppliers that inhabit the curve.  Christensen goes on to assert that innovators that enter a market at some point on the curve or an expected point on its trajectory, will get crushed by the incumbents whose products already inhabit the curve.  His lessons?  Incremental and expected improvements revert to the benefit of the incumbents.  And if you want a distinctive, defensible position in a market, find an off-trajectory position that the incumbents cannot or will not attempt to serve.

Christensen’s model is extensible beyond a simple product.  If we consider “product” to be a “solution” consisting of a set of processes, people and technology, the model still holds.  Introduce a problem / potential solution into an organization that falls near the [improving] trajectory of existing processes, people and technologies within the organization, and the existing organization will handle it as it always does.  That’s inertia, and it resists disruption (Christensen’s term), the derivative of true innovation.

So what does this have to do with cloud computing?

Returning to our lessons above, cloud or utility computing applied to solutions we may reasonably have achieved with incumbent processes, people or technologies will likely not be innovative, disruptive or frankly very interesting.  They may indeed be cheaper, and the wheels of the competitive markets will turn over the next years to find some new equilibrium in a cost-driven model.  This is the evolutionary progression in computing of the last 40 years as we traverse from service bureau computing to corporate mainframe to departmental mini to personal computer to client server, and so on.

The spackling over of all the computing stuff we have now with the hyped “Cloud Computing” label is happening with abandon.  Good news in that awareness and buzz is high; bad news in that the inevitable post-hype backlash is coming.

What to do?

Choose wisely in selecting your problems and how you frame them to achieve the breakthrough advantages of utility computing.

I have no first-hand knowledge of the NY Times, but I imagine their IT and finance processes and people are top shelf.  Go to any well-run company’s IT shop and ask for 100 servers, or go to the ‘New Applications’ window and try signing up a project like Gottfrid’s.  You’ve chosen to play on the trajectory of the IT Infrastructure group, the new applications group, or insert some label from the current org chart here group.  Good luck.

Instead, pick a problem not served by some application already in the data center, one considered impossible by the professionals or better yet one they cannot or will solve for you.  Even better, try a business model with the most tricked-out computing requirements without owning any servers.  Pull out your credit card and get going.

We are awash with a unprecedented abundance of Free* computing capacity.  The equivalent of millions of servers, hundreds of millions of PCs and who knows how many mobile phone handsets, referring to the unused capacity on machines bought, paid for and running.  See “It’s Not a Cloud.  It’s a Mine.“  In addition, we have thousands of Free Applications (FreeApps) available via Web 2.0.  We have hardly begun to realize the potential wealth — scientific, social, educational and monetary — that could be unlocked by virtue of exploiting this capacity.

Evidence of the Theme, present today:

We have rich 3d simulated space in which to:

• Collaboratively develop insight, knowledge and overcome our limited human ability to perceive scale.  Example:  Second Life Drexel Island with avatar-sized molecules rezzed to view, interact with, and experience how they dock with proteins, etc. Free.
• Collaboratively develop new insights by rezzing data in novel structures in which to interact, view and experience them.  Example:  Second Life Data Visualization;  Green Phosphor Data Visualization; and Sun’s Project Wonderland.  Free and almost Free.
• Modify physics engine to better understand interactions of materials on at nano-scale, for example.
• Modify physics engine to simulate a Mars expedition, for example.  No reason we couldn’t create a Mars mission simulation and let the 5 to 8 y.o. kids of today (Club Penguin and Webkinz digital natives) play with it, creating an impulse to the right educational track and to create an abundant pool of qualified crew candidates 20+ years from now.  Free to use.
• Eliminate geographic and language barriers for those who may have reason to come together to solve a problem, learn a concept, evaluate a medical outcome.  Free.
• Overcome physical and social barriers.  Simple example:  allow a handicapped person to experience and express some human interaction such as dance, not otherwise available.  Free.
• Amazon with EC2 and Google with App Engine, attempting to monetize directly and indirectly (new business models, perceived stock value, etc.) spare computing capacity through Utility Computing. Almost free.
• Paraphrasing Clay Shirky from Here Comes Everybody:  It is now cheaper to just try something on the web than it is to do the analysis as to whether or not it will work.  Almost free.
• Seti@Home achieved 528 TeraFLOPs via 334,155 active computers in 210 countries, as of August 2008.

Missing Pieces:

Meanwhile, I note the elements missing in order to take advantage of this great abundance of our time:

• Educated and informed practitioners who can identify the problems to solve, formulate them in a manner which allows Utility Computing and Free Apps to solve them
• Informed business leaders who understand the power of the abundant and free computing that can be brought to bear on big problems, and can most importantly frame the value propositions such that we can attract and fund talent (not so free) to solve them
• Leadership.

* I am taking license to define “Free” as any cost less than a Starbuck’s latte grande, on a per month, week or day basis (haven’t decided).

Mining for Silicon Gold

In my last post, I asked you to switch the metaphor for abundant free (at the margin) computing from clouds and Cloud Computing to mines and miners. What you find in the mine — the water, oil or computing — is valuable when you use it.  Seems obvious for water and oil.  Computing is valuable too, but takes a little more thought to get a handle on it.

Let’s consider the provider side of cloud computing, and turn to the really big mines. Google, Amazon and Facebook, for example, operate sites that are run by thousands of computers.  Business Week reports estimates that Google probably has in excess of 500,000 servers, Facebook going from 10,000 to 50,000 servers, and so on.  I speculate Amazon employs some number of servers between Facebook’s and Google’s.  These are big numbers, and while servers are cheaper than ever to buy, 100,000 of them represent a significant capital commitment.

Why so many?  So you and I get a great experience.  Take Amazon for example.  Amazon is one of the best experiences on the web, and almost all those pages are assembled just for you on the fly.  All done for the mission of creating a compelling experience in which to buy what Amazon sells.  It takes considerable horsepower to put it all together just for us, and considerable scale to deliver it to thousands, perhaps hundreds of thousands, simultaneously.  The scale has to deliver the good experience on peak shopping days like the day after Thanksgiving, or the days leading up to Christmas.  A poor experience, a slow session, or an unavailable connection directly leads to missed sales and loss of revenue for Amazon.  So Amazon has excellent, quantifiable business reasons to invest capital in scaling its computing resources.

But what of other days, like April 15th when you and I are standing in line at the post office to file our tax returns, not shopping at Amazon?  That pre-Christmas shopping capacity is idling.  The gap, the difference between the computing capacity present and paid for, and the amount actually being used by Amazon at any given time, is the free computing to which I refer.  Invoking our metaphor again, Amazon is sitting on a huge mine.

Let’s review the direct reasons why a mine owner like Amazon, might pursue becoming a cloud computing supplier.  Managers have a duty to increase enterprise market value (EMV).  I use the EMV framework so often used in calculating value of a corporation and its stock, because it is an excellent mechanism for translating seemingly arcane issues into something that executives, employees and stockholders care about.

I include a couple of levers which are not applicable for a cloud computing supplier – because I want to keep the framework intact so that we can return to it again in the future.  In any case, see Supercharging Supply Chains by Tyndall, Gopal, Partsch and Kamauff for an excellent, practical overview of EMV.

Profitable revenue growth is straightforward – selling something you have already paid for.   Fixed capital efficiency is likewise easy:  if you have $10 million or a $100 million in servers, it makes good sense to use them for all they’re worth.  If you don’t, you’re better off leaving the server money in a bank earning interest.

Tax minimization is a little more nuanced.  First, taxes take cash out of the free cash flow stream, which is a critical contributor to EMV.  A corporation like Amazon, Google, and so on, will have multiple data centers and operations in multiple locations.  Computing the likes of which we are discussing here, can be shifted reasonably from one data center to another, subject to some limits like latency (which is one of the reasons to have multiple data centers in the first place).  Some operating costs can therefore be shifted.  In a like manner, personnel with their costs to manage the computing resources can be shifted.  All other things being equal or normalized (talent, energy, risks, etc.), a company is usually better off to spend the money – that is to take the costs – in the region with the highest effective tax rates.  Why?  Greater deductible costs result in a lower tax bill, which is cash lost permanently from free cash flow, and hence detrimental to EMV.

As much as I appreciate the classical EMV levers as business drivers, I like the indirect drivers for a credible cloud computing supplier even more.  Selling, or even giving away, computing capacity makes them better!

Think about it.  Glass houses let outsiders see your stuff (silicon dioxide, the substrate for processors, is glass BTW, making ‘glass house’ a rich double entendre as we speak of Google, Amazon and similar).

Opening up one’s servers to the public requires the traditional support processes such as billing, customer service, training, communications, etc.  These are not unique to cloud computing, and a company has a reasonably good chance of being good [or bad] at these processes without regard to computing.

There are a number of processes that are distinctive to computing including design, operations, risk management, testing, performance management, governance, training, communications, help desk and security.  These are critical IT processes that challenge most companies internally.

To expose these processes to customers and the public requires solid grasp of the operations, confidence in the management processes, and commitment to make everything better.  In short, these companies have to do these things so well, that they are confident to enter competition – and that competition will make them better.

I want to stress a point which is even sharper for Cloud Computing.  The internet is largely enabled by open source software and technologies, with the ubiquitous, free and open source LAMP stack as the poster child.  These are wrought by legions of very smart, very talented developers motivated for many reasons.  Mostly volunteers, they are not employed or paid to make the software and the tech of the internet better, which they do regularly with passion.  They care about building and maintaining cool tech, and earning recognition for their personal genius.  They care about reputation and they care about each other.  They do not suffer ‘lame’ software lightly, nor will they accept marketing hype at face value.  They are fiercely independent, not quiet and not passive.  At this early stage of cloud computing, these people are the Consumers and if a company wants to engage them, the company has to ‘have game’ sufficient to earn their participation and respect.

Fortunately, at least three of the largest mines for this kind of computing, Amazon, Google and Microsoft, have the people and skills to engage successfully with the cloud computing consumers to which I refer.  I am sure there are others, I simply haven’t looked.  All are publicly traded companies, which brings an additional driver of excellence – market perception.
Computing is a core competence of these companies.  Embracing cloud computing, with the forces for excellence it brings to their internal processes and systems, makes these companies better.  — db

Cloud Computing is a whimsical term our tech culture adopted, and we could have done better.  I understand why we landed on the cloud label, as I contributed.  You see, we all put fluffy cloud images on our slides in place of the tedium and scale of the networked computer architecture, hardware and software that is the internet.  It was all too easy to answer “It happens in the cloud,” in response to too many questions.  Our fault.

Now money is flowing into Cloud Computing and expert marketing people are busy recycling a bunch of old stuff with a Cloud Computing label.  Pity, because the predictable after-the-hype hangover will disenfranchise too many people from the merit in cloud computing.  And we desperately need their creativity to mine this untapped resource.

Obscured by the Cloud label, is an incredibly important point.  We are awash with more free but unused compute capacity than we can imagine.  I am sure that the current unused capacity far exceeds the total computing capacity, used plus unused, that existed on earth during many periods in my lifetime.

Free computing.

Our computers include servers which are the workhorses of the enterprise and the internet.  Millions of servers.   Netcraft has a survey of websites that would loosely correlate with the number of web servers (remember that many small websites like this one may be hosted on a single server, while big websites may employ thousands of web servers).  There are many hundreds of millions of personal computers and workstations, networked to the servers and each other. Forrester says there will be over a billion personal computers in operation by the end of 2008.

Stay with me.  Think about a running computer.  You bought it, powered it, cooled it and tended to it.  You spent your money.  Now that computer computes whatever you load, and in so doing is consumed some amount – say, 90% or 50% or 10%.  Then you add some additional compute load such as 1%, so the total becomes 91% or 51% or 11%.  That incremental amount doesn’t cost any more than you already spent.  At the margin, it’s free.

Now the sticklers will say, “You’re wrong, it will use a little more power and a/c.”  Technically, they’re right but it doesn’t matter.  Still free.  And the reason they are negligibly right is slightly perverse:  we are so rich with computing abundance that designers add capability to throttle back computers when not fully loaded.  Can you imagine explaining to Galileo (1564 – 1642)  or Copernicus (1473 – 1543)  that modern man is so rich with marvelous computing machines that he devises clever ways not to use them?

Calling it a Cloud does not help us think about the opportunity to use our abundant, free resource.

Let’s try another metaphor:  think about your house with a big yard, and a deep hole you dig into the ground.  You spent good money on the house, land and shovel.  At the bottom of the hole, you find water, oil or free computing.  Whether you use the water, oil or computing doesn’t change what you already spent on the house, yard and shovel. You found Computing Resources.  Mine and miner illustrate much better what’s going on with Cloud Computing, than clouds, in terms of finding and exploiting computing resources. (I realize holes where water and oil are found are wells, but ‘mines’ works too for purposes here.)

Mining for Computing Gold

(Illustration was created using Powerpoint.  Noncommercial reuse with attribution permitted.)

How big are our global computing reserves?

Before I make the calculation, you must understand this estimate is prototypical and for the purposes of illustrating the abundance.

If my numbers were to hold (i.e., not pulled out of thin air), we would have Global Computing Reserves in excess of 5 million servers and 500 million PCs.

I expect to get vigorous advice about the numbers being wrong.  And just as with global petroleum reserves, the ability to estimate the reserves does not translate into the ability to produce them all.  Unlike global petroleum reserves, we are briskly creating additional global computing reserves. By the way, I did not include mobile telephone handsets, an increasingly important and powerful set of networked computing resources.  Nor did I include the graphics cards in PCs, which are powerful computers barely used except when gaming or viewing Youtube.

So my numbers may be wrong.  Bring better numbers and your own method of making the estimate.  Even if I am over by 2 orders of magnitude, the net available computers may be between 1 and 10 million.

Hey entrepreneurs (miners), free computing!

Let’s use it.  Here is where the creativity I longed for far above is so desparately needed. – db