Tuesday, August 1, 2017

Dr. StrangeCloud – Or, How I Learned to Stop Worrying and Love the Cloud

I wrote this blog a few years ago when I was still CTO at VMware. I didn't realize it was published on the web by VMware until someone sent me this link today. The last paragraph is a sales pitch for VMware products so I've deleted it below but the rest is a fairly honest assessment of the cloud and its impact on infrastructure that I still stand by.

Dr. StrangeCloud – Or, How I Learned to Stop Worrying and Love the Cloud

  • By   
By Ben Fathi, Chief Technology Officer, VMware
Marc Andreesen famously wrote “Why Software is Eating the World” in August 2011:
“More and more major businesses and industries are being run on software and delivered as online services—from movies to agriculture to national defense. Many of the winners are Silicon Valley-style entrepreneurial technology companies that are invading and overturning established industry structures. Over the next 10 years, I expect many more industries to be disrupted by software, with new world-beating Silicon Valley companies doing the disruption in more cases than not.”
A scant three years later, it’s time for us to admit that software has already eaten the world. It’s also time for us to get over it and start dealing with the consequences.
Here’s a simple challenge: Name one industry or a single aspect of our social lives that hasn’t been dramatically and irreversibly changed by the power of software. Whether entertainment or education, travel or medicine, genetics or physics, banking or shopping, driving or communicating – I challenge you to find a single activity not influenced by or totally redefined through software. Entire industries – travel agencies, bookstores, photography labs, music stores, telcos – have either disappeared or have had to reinvent themselves to survive.
Until recently, incumbents routinely enjoyed decades of prosperity in every industry – including software. But in this brave new world, whether consumer or enterprise, we are innovating at such a pace that products introduced just a few years ago are already obsolete or being disrupted. As computation and storage costs continue to decrease and network bandwidth increases – as computing finally becomes a true utility – we’ll find more and more applications that can benefit from the power of software.
We are truly starting to live in a digital universe with software at the core of everything we do.
Just like we stream or download our movies and music today rather than going to Blockbuster or Tower Records, we download most of our applications from the internet as well. When was the last time you installed software by going to Best Buy and buying a shrink-wrapped box? The new cloud-based consumption experience is more convenient. It gets rids of the inventory problem, the manufacturing problem and the supply chain problem, but it also presents new challenges. Having these services available 24×7 is no simple feat. To pull it off, we run massive data centers in the cloud, with software designed to be resilient and scalable.
I sincerely believe we are at an inflection point in the history of computing. The future will be “cloudy” – not just for consumers but for enterprises too. The massive and unmistakable move toward cloud computing further reduces the barrier to entry for startups and simplifies the consumption experience dramatically.
We’re already seeing this in the consumer space as the complexity of Windows has given way to the simplicity of Android and iOS—simple operating systems augmented by compelling cloud services, with “worry free” upgrade and maintenance.
The same benefits apply to enterprise software as well.. Many categories of enterprise software are now being delivered as cloud based services: Salesforce for CRM, LinkedIn for recruiting, Workday for HR, Office365 for productivity, etc. Even infrastructure, such as servers and storage, can now be consumed through the Internet with IaaS offerings from the various public cloud providers.
Now let’s put ourselves in our customers’ shoes for a minute: As a progressive virtual infrastructure/private cloud admin, I would like to install the latest release of vSphere once a year so I can get access to the latest innovations from VMware and its partner ecosystem. Realistically, though, upgrading to a new version is often tied to hardware refresh cycles, so I may have to wait 3-4 years for the latest innovations. I also have a lot invested in high-end storage gear and network switches, so I want to continue to utilize them. As part of the upgrade process, I will have to install the right third-party drivers and firmware on the servers and the SAN arrays and the network switches, install and configure disaster recovery solutions, etc. In essence, I become the system integrator and take on a significant amount of work for my IT organization, in the process creating a bespoke environment that is different from every other enterprise.
Now let’s switch hats for a minute and become a “cloud” customer.  I pull out my credit card, go to www.vcloudair.com (or one of our 4,000 or so service provider partners in the VMware vCloud® Air™ Network), click a few buttons, and I’m up and running my application in minutes. If I need high availability, I just check a little box that says “Make my workload resilient to two simultaneous infrastructure failures” and the cloud takes care of the rest.
You will correctly point out that I’m comparing apples to oranges here. I’ve just outsourced my IT to VMware by using their cloud. Some poor administrator is still installing software on those servers somewhere and maintaining them. True enough, but that’s our specialty and I think we’re better positioned to handle it. The key point here is that the cloud provider optimizes their capital expenditure and operational costs by drastically reducing the hardware and software configurations that he supports to provide the service.
Let’s face it. The cloud experience is a major leap ahead of any improvements we can ever make to the “shrink-wrapped” experience. Even if we ship perfect bug-free software, we are still asking the admin to do integration of all the third party components on-site and to manage the lifecycle of all these products by performing upgrades and patching. Increasingly, admins (and CIOs) today are being asked to choose between this model and the cloud model.
... [Rest of blog (VMWare-specific comments) available here] ...

Monday, July 3, 2017

Riddle Me This: Sketches from a Brief European Sojourn


"To believe something is to believe that it is true; therefore a reasonable person believes each of his beliefs to be true; yet experience has taught him to expect that some of his beliefs, he knows not which, will turn out to be false. A reasonable person believes, in short, that each of his beliefs is true and that some of them are false."
           W. V. O. Quine. Quiddities: An Intermittently Philosophical Dictionary.

"Well, we got wines from all over the world. We got, uh, English wines from France, we got Italian wines from all over Europe."
EJ Carroll. Everybody's Fine.

“No matter what people tell you, words and ideas can change the world.”
Robin Williams. 1951-2014.

We’ve been coming to Europe on a regular basis for almost forty years now and love practically everything about it: the bustling and historic cities, the cobblestoned alleys in medieval villages, the sun-drenched islands in the south and the green valleys in the north, the many languages and cultures. I stopped going to museums and tourist sites long ago, choosing instead to walk the streets, sit at sidewalk cafes, and watch people.

---

We’ve just landed in Paris, the sun is shining, and I’m jetlagged. What better time to go for a walk? As I walk through the twisty side streets of Marais, I’m surprised to note the lack of tourists. I have always been in love with Paris but had come to dread visits surrounded by throngs of tourists at every corner. This time, things are different. The airport was practically deserted; it took us less than five minutes to clear customs. There are tourists around, to be sure, but they are few and far between. Police cars and armed officers are everywhere, government buildings are barricaded, and we are asked to cross the street when we get too close for comfort. Every few minutes, police cars and armored vehicles streak past at high speed, sirens blaring. There is a tension in the air that I’ve never sensed before. I’m sad to see that terrorism has killed tourism in this beautiful city but I’m also, secretly and guiltily, enjoying the quiet streets.

I sit at a cafe in Place des Vosges to have a drink. There are almost no tourists around and the locals are sunning themselves in the park in the middle of the square. The waiter comes by, hands me a menu, and walks away. He doesn’t say anything and neither do I. A few minutes later, I look up and notice that he’s standing in a corner, grinding his teeth, making ugly faces, and flexing his muscles while looking directly at me. That’s odd! Another couple of minutes go by and I glance at him again. If anything, his gestures have become even more pronounced. He’s clearly not happy to see me there and is making it obvious.

IMG_0192a.jpg
[Paris - Source: http://benfa.smugmug.com]

It’s a strange spectacle I’ve never witnessed before. Everyone talks about rude Parisians and I’ve seen my share of them but this is not normal. I think back to what I may have done to irritate him but we have barely interacted and haven’t even spoken to each other yet. I sit there, puzzled and uncomfortable. The next time I look in his direction, he’s talking to another waiter inside the restaurant. It’s clear that he doesn’t want to serve me and is looking for a way out. This second waiter now comes to take my order. I ask for a glass of wine and try to ignore the hostile environment but it’s no good. Amazingly, the first waiter now stands in a corner, raises his fists, and grimaces as he stares angrily at me.

I’ve had enough. I stand up and pay for the wine I haven’t drunk. As I turn to walk out, I see him smiling broadly, frankly beaming, happy to be rid of me so quickly. It occurs to me that he’s probably a Le Pen supporter, angry at all the foreigners destroying his country and culture, frustrated about the terrorists killing his people. I see a parallel to the current political climate in the US and can’t honestly blame him for his actions. What I don’t quite understand is why such a person would take a job as a waiter in a touristy location where he has to interact daily with the very people he dislikes.

---

It's 5 PM in Paris, a few days later. The angry waiter is a distant memory and, thankfully, all my other interactions have been positive. I'm sitting at a cafe on a side street in Saint Germain and trying, unsuccessfully, to blend in. My tourist clothes instantly betray me as I sip a drink and watch locals chain smoke like it's 1985!

A couple of fifty-something men of color sit down at the next table. They are sharply dressed in business suits and are clearly professionals of some sort, either businessmen or lawyers. Both are in superb physical condition - not even a hint of a beer belly - yet I doubt either of them has seen the inside of a gym in the recent past, if ever. They have dozens of printed pages in front of them and scribble notes as they discuss a business deal or perhaps argue over a court case. Surprisingly, one of them orders a glass of milk and the other orders an espresso and chocolate croissant! Did I mention it’s 5 pm?!?


For the next two and a half hours, they chain smoke and talk, laugh and argue, frequently making notations on the pieces of paper. They’re still at it as I get up to leave. For all I know, they will continue the discussion over dinner. It occurs to me, as I walk away, that I've been watching a routine business meeting that is commonplace here in Europe but would be unheard of in the US. Everything about it is different from the equivalent experience in the States: the outdoor setting instead of a conference room, the glass of milk instead of beer, the two and a half hour duration, pieces of paper instead of laptops, sharp suits instead of jeans or dockers, trim physique instead of layers of flab, the relaxed banter, cigarette smoke wafting through the air.

The scene is basically repeated at another cafe the next evening. The guy at the next table looks like a Hollywood movie star: thirty-something and, again, fit as a fiddle. Both he and his girlfriend are smoking and sipping wine. Three ancient looking women sit at another table drinking Campari and Soda, talking up a storm, and, you guessed it, smoking almost non-stop! As I look around me, I see more and more fit people walking around with their kids, buying bread for dinner, or just sitting and talking unhurriedly.

I have to admit the locals are pretty damn healthy looking, by any standard of physical fitness you may wish to use. And yet they eat fatty foods, smoke like chimneys, and stay up until midnight drinking - on weeknights, no less! Most have never seen the inside of a gym and get their only daily exercise by walking. They seem healthier than their American counterparts - regardless of age.

I see the same thing as we travel through France and Italy over the next few weeks. With fewer tourists around, the European cities and countryside are actually much more enjoyable. Rome and Tuscany, perennial tourist favorites, were crowded but the villages of Umbria are almost deserted. More than anything, it’s the relaxed pace that is jarring to Americans. Even in bustling Rome, the pace is glacially slow. Dinner can’t be had in less than three hours - the waiter will make sure of that!

Assisi_0028.jpg
[Assisi - Source: http://benfa.smugmug.com]

Riddle me this: Why do Europeans get to live like this, enjoy longer relatively healthier lives, all while staying out with friends and family eating massive dinners and drinking multiple bottles of wine until midnight, taking four hour siestas every day, never getting on an exercise bike or attending a Zumba class in their lives, and smoking cigarettes?

This will seem like a cliche but there are only two main differences that I have observed: first, Europeans eat whatever they want; second, the level of stress over there is visibly lower. We talk obsessively about work-life balance, they live it. Americans work harder and longer hours, are much more careful about their diets, avoid cigarettes like the plague, spend hours a day exercising, and yet they are chronically overweight, in debt, and tired if not sick. My admittedly simplistic interpretation of the data, based on nothing but anecdotal evidence, is that the combination of hormones and genetically modified ingredients in our foods are slowly but surely making us sick while ever-increasing levels of stress conspire to make us miserable on a daily basis.

A baguette purchased in the morning in a European city goes stale if not consumed by that same evening but a loaf of bread purchased at an American supermarket has an expiration date a month hence. Why do you suppose that is? We’ve gotten so used to having chemicals added to our basic foods that we no longer even question the need for them.
I’m not naive enough to believe that all Europeans enjoy happier healthier lives and that all Americans are doomed to suffer through stressful ones. Nor am I suggesting that Europeans do everything right and we should abandon our lifestyles tomorrow. I do, however, have to wonder when we’ll wake up and get off the treadmills we’ve created for ourselves.

---
My brother-in-law travelled with us. He’s suffered from an amazing series of gastrointestinal problems over the years. He's had colitis; he's had a quarter of his intestine removed due to digestive problems; he can't eat dairy, gluten, carbs, or sweets; he is constantly going from doctor to doctor while at home in the US, can’t eat anything other than bland home cooked meals, and is constantly at the edge of another intestinal “episode”.

He had pasta, tiramisu, and bread every single day. He had fried food, he had salads, he had cheese, he had deserts. He ate things he would never dream of eating in the US - and he was fine! He ate anything and everything while he also wondered aloud why he wasn't having any intestinal problems! Same guy, one day later, random restaurant in Italy. The experience was repeated again and again every day while traveling. No intestinal problems whatsoever. The only thing that was different were the ingredients in the food.

Anecdotal evidence.


---

Then there’s this. I googled the GDP of Italy. It’s 1.8 trillion US dollars - with a population of 60 million. Think about it. That's about three Apples! Sort of puts things in perspective to think about it that way. The numbers are only slightly better for France: $2.4 trillion and 66 million people. By comparison, the US GDP is $18 trillion for a population of 320 million. In other words, the per capita GDP is almost double that of Italy and France. Our economy is clearly doing better but do we live better lives as a result? I wish we’d take a lesson from Bhutan and start measuring our Gross National Happiness instead.

Sunday, June 18, 2017

Our True Sixth Sense: Fiction

“Not everything that can be faced can be changed, but nothing can be changed that is not faced.”
James Baldwin. I am not Your Negro.

“In the 300 years of the crucifixion of Christ to the conversion of Emperor Constantine, polytheistic Roman emperors initiated no more than four general persecutions of Christians. Local administrators and governors incited some anti-Christian violence of their own. Still, if we combine all the victims of all these persecutions, it turns out that in these three centuries the polytheistic Romans killed no more than a few thousand Christians. In contrast, over the course, of the next 1,500 years, Christians slaughtered Christians by the millions, to defend slightly different interpretations of the religion of love and compassion.”
          Yuval Noah Harari. Sapiens: A Brief History of Humankind.

Hans: As Gandhi said... An eye for an eye leaves the whole world blind.
Billy: No, it doesn't. There'll be one guy left with one eye. How's the last blind guy gonna take out the eye of the last guy left? No, it doesn't. There'll be one guy left with one eye. How's the last blind guy gonna take out the eye of the last guy left?  All that guy has to do is run away and hide behind a bush. Gandhi was wrong. It’s just that no one has the guts to come out and say it.”
Christopher Walken and Sam Rockwell. Seven Psychopaths.

We humans do have a sixth sense but, despite popular belief, it’s not ESP (Extra Sensory Perception). In fact, I would argue that ESP is simply a small subset of our broader instinct. Our true sixth sense is our sense of fiction, our innate ability to weave narratives at every opportunity, our penchant for asking “what if” at every turn, our willingness to create or participate in stories that may or may not be rooted in the physical world around us. We’re the only species capable of keeping thousands of such complex scenarios in our mind, each a fiction that we personally or collectively believe in. Other species are capable of rudimentary deception (chimps can “lie” about where they hid a banana, for example) but we're the only species who absolutely wallows in fiction in practically every part of our lives – so much so that we’re surrounded by it at almost every moment, often without even consciously recognizing it.

That fiction may be Harry Potter or Game of Thrones but those are obvious examples and momentary diversions. The more common stories are the ones we live in all day and night – our common beliefs. The best analysis of this phenomenon I’ve ever seen is from Yuval Noah Harari in Sapiens: A Brief History of Humankind. We believe in soccer and basketball teams: concepts that have no basis in physical reality but are nonetheless “religiously” pursued by millions of people. Think about it: what exactly is the definition of the NFL team, The Oakland Raiders? It’s a moniker we apply to a group of men and watch every Sunday on television. The members of the team change every year, the team manager changes once every few years, the owners change once a decade or so, the team jersey and insignia are redesigned whenever ratings sag, the rules they follow are not only arbitrary but can also be changed by a committee at their whim, even the city they are supposed to represent changes on an infrequent basis. Remember The Los Angeles Raiders? Yup. Same team – for a few years. So what exactly is The Oakland Raiders or the NBA, for that matter?

What exactly do we mean when we say “I love the Oakland Raiders?” You can make the same declaration about roses, about Mt. Fuji, about cocker spaniels, and about the moon – with one big difference. The latter are all physical objects that have a fixed definition that could be recognized, if not verbalized, by even an animal. Pretty much everything else we ever talk about or believe in is made up, a fiction, a story that we tell ourselves and others. Including the Oakland Raiders. And religion and society and government and currencies and borders and corporations and brands. Everything on this new list is just a figment of our collective imagination – something that homo sapiens dreamed up. It’s amazing to think about the world around us through this lens; it frees us of many of the prejudices that we take for granted. I call us homo theatricales – we not only make up the stories, we also star in the shows.

Now let’s turn the same lens to a different scene. What does it mean for hooligans in Britain to kill each other over a Manchester United game or for terrorists from the Middle East to kill Christians for their beliefs or for a gunman to take up an AK-47 and walk into a crowded square because IRA? Every single one of those constructs, those sets of beliefs, is a figment of our imagination. If you don’t believe me, let’s look at countries. Think about the “line in the sand” that separates our modern nations. Step over this line through the accident of birth and, suddenly, villages that are separated by a mere five miles are mortal enemies. The proverbial “line in the sand” is just one example of such a fiction. Not a physical line in the sand, mind you. It's just a line somebody drew on a piece of paper about a hundred years ago, roughly speaking, for most countries. But it’s good enough for us to kill each other over.

Most of what we believe in - nations, corporations, religions, gold as a valuable metal, money as a piece of paper that has real value - these are all fictions we have created for ourselves as a species. It is what sustains us, it is what distinguishes us. It is, unfortunately, also what separates us – what turns us into competing bands.

So, I have to ask: if we all agree there is really no line in the sand separating, say, Israel and Lebanon, if we agree that such an imaginary line shouldn't mean the inhabitants on one side are “Arabs” and the guys on the other side are “Jews”, forever and ever - and, oh by the way, we just made up that line a hundred years ago; if we agree that a piece of paper, even when adorned with fancy colors and markings, a unit of currency, does not really have any intrinsic value of its own that would lead you to hand over a car or a jacket over for, that its only value is an index into a database of international monetary exchange rates - so we know that a dollar is worth 1.24762 Euros - even though both are nothing but pieces of brightly colored paper backed by a “promise”, a fiction that exists nowhere but in our collective consciousness; if we understand all this, then why are we so adamant that our side is right - when all the rules are imaginary? Why are we convinced that Catholics are right and Muslims are wrong, or vice versa? The very concepts and teachings of both religions are nothing but the collective beliefs of a group of people. Nothing more and nothing less. Why are we so adamant that North and South Korea are enemies, when neither of those countries - those ideologies - even existed a hundred years ago? They exist nowhere but on paper and in our minds.

Almost everything we get worked up about these days - nationalities, religions, sports teams, financial markets - are nothing but figments of our collective imagination. If you think of it that way, it's much easier to let go of the dogma, the irrational belief that my side is right and your side is wrong - and I don't care which side of the argument you're on, nor do I care which argument we're talking about.

If you think of it this way, furthermore, our sixth sense being our ability to spin yarns and create fictional scenarios in which we live - be they at the personal level, at the corporate level, or at the international level - then the idea of the sixth sense being ESP also starts to make sense. Extra Sensory Perception is nothing but us spinning yarns, making up stories about things, belief in an extra-sensory experience that simply does not exist. I claim every case of reported ESP is nothing but someone creating - or following - a fictional narrative. That doesn't mean they're lying. In their minds, they believe everything they are seeing. Our confirmation biases are too strong for that to not happen.

The lesson? Question your beliefs. Don’t follow them blindly. They are often nothing but fictions, stories that we have weaved for ourselves. As a species, we're damn good at doing that.

That democrats are assholes or that republicans are idiots, that moving a couple of kilometers across a border dramatically changes people’s belief systems and outlook on life, that Manchester United is better than Real Madrid because Ronaldo - these are all fictions in our heads.  I don't even know if that last sentence made any sense because I don't follow soccer. I just know Ronaldo is the name of a character in that universe, that narrative - and the other two are “teams” that may or may not be rivals.

The next time you get all worked up about something, anything, ask yourself: Could I explain this to a member of another species – any species. Ask any passing elephant or gorilla if they are citizens of Namibia or Botswana and you will see my point. Or try to give a ten dollar bill to a chimpanzee for his bananas. Good luck. We are the only species that believes these things. They are figments of our imagination - and, as such, malleable. If we only allow ourselves to be a bit more flexible on our “beliefs”.

So what does it mean to get all bent out of shape over an imaginary line in the sand? Or a religion for that matter?


These are all fictions. We are all the same. Get over yourself.

Thursday, June 8, 2017

What Really Happened with Vista: An Insider's Retrospective

“Experience is something you don't get until just after you need it.”
Steven Wright.

I enjoyed reading Terry Crowley’s thoughtful blog (What Really Happened with Vista). Terry worked in the Office organization and did a fantastic job covering the complex machinations that went into Windows Vista and the related but doomed Longhorn project - from an outsider’s point of view. He correctly identified many of the problems that dogged the project and I don’t mean to rehash any of them here. I figured it was only fair to try to offer an insider’s view of the same events. I can’t hope to be as eloquent or thorough as Terry but hope to shed some light on what went wrong. Ten years have gone by since the original release date of Windows Vista but the lessons seem more relevant now than ever.

Windows is a beast. Thousands of developers, testers, program managers, security experts, UI designers, architects, you name it. And that’s before the supporting cast of HR people, recruiters, marketing folks, salespeople, lawyers, and of course many managers, directors, and vice presidents for each of the disciplines mentioned above. The entire ensemble cast is supported by many thousands of others at partner teams (within Microsoft as well as outside) that deliver hardware underneath or solutions on top of the platform.



Organizationally, at the time, Windows was really three teams: Core, Server, and Client. The core team delivered the “plumbing”, all the core components of the operating system (the kernel itself, storage, security, networking, device drivers, the installation and upgrade model, Win32, etc) shared by all versions of Windows. The server team, in turn, concentrated on technologies needed for the server market (terminal services, clustering and high availability, enterprise management tools, etc) while the client team was responsible for technologies related to the desktop and consumer releases (web browser, media player, graphics, shell, etc). There were, of course, many reorgs but that basic structure was kept in place even as Windows grew in popularity and the teams grew in size. It would also be fair to say, culturally and organizationally speaking, that the core team was closer to the server team than it was to the client team - at least until after Vista shipped.

By the time I arrived at Microsoft, in early 1998, Windows meant Windows NT - architecturally, organizationally, and product wise. The Windows 95 code base had largely been abandoned and Windows NT had been adopted for every personality of Windows - from the laptop to the clustered Server. Two years later, the Windows 95/98 code base would be resurrected for one last release - the much maligned Windows ME - but that project was executed by a small team while the vast majority worked on the NT code base. I was lucky enough to spend a dozen years in the belly of the beast, joining during the heyday of Windows 2000 development and staying through to the completion of Windows 7.


I spent the first seven years of my tenure managing the teams responsible for storage, file systems, high availability/clustering, file level network protocols, distributed file systems, and related technologies. Later, I spent a year or two managing security for Microsoft. This included everything from security technologies in Windows to antivirus products as add-on solutions to security marketing and emergency response such as security patches. This was towards the tail end of Vista when viruses and worms were bringing Windows to its knees and when Microsoft's reputation for building secure software had taken a massive beating in the marketplace. For the last three or four years, for the duration of the Windows 7 release, I managed all core development in Windows. That meant dev ownership of pretty much all technologies running “under the hood” and used by both the client and server teams. After Vista shipped, the Windows team was organized by disciplines and a trio (Dev, Test, PM) was put in charge at every level of the org so I had two partners in crime. I managed the development teams while they managed, respectively, test and program management.
The Windows team had a history of trying massive projects that were often abandoned or repurposed after a few years. An earlier example was the ambitious Cairo project which was eventually gutted, with some pieces salvaged and shipped as part of Windows 2000. By far the biggest problem with Windows releases, in my humble opinion, was the duration of each release. On average, a release took about three years from inception to completion but only about six to nine months of that time was spent developing “new” code. The rest of the time was spent in integration, testing, alpha and beta periods - each lasting a few months. Some projects needed more than six months of core development so they proceeded in parallel and merged with the main code base when ready. This meant that the main tree was almost always in a semi-broken state as large pieces of functionality were being integrated or replaced. Much tighter controls were put in place during the Windows 7 release to ensure a constantly healthy and functioning code base but earlier releases were plagued with daily instability for months at a time.
The chaotic nature of development often resulted in teams playing schedule chicken, convincing themselves and others that their code was in better shape than other projects, that they could “polish” the few remaining pieces of work just in time, so they would be allowed to checkin their component in a half-finished state. The three year release cycle meant we rarely knew what the competitive landscape and external ecosystem would look like when we started a release. Missing a release meant cancellation (as the feature rarely made sense six years later) or, worse, banishment to Siberia - continued development on a component that was mostly ignored by the rest of the organization and doomed to eventual failure or irrelevance, but one that the team or the execs simply couldn’t bring themselves to abandon. I was personally responsible for a few such projects. Hindsight is 20/20.
Given that each team was busy pushing their own agenda and features into the release, they often skimped on integration with other components, user interface, end to end testing, and ugly and tedious issues such as upgrade, leaving these thorny issues for the endgame. That, in turn, meant some teams quickly became bottlenecks as everyone jockeyed for their help in finishing their UI or upgrade testing at the last minute.
At any given point in time, there were multiple major releases in progress as well as multiple side projects. Different teams were responsible for code bases in various states of health resulting in a model where “the rich got richer and the poor got poorer” over time. As a project neared completion, program managers would start looking at requirements for the next release and developers in “healthy” (rich) teams would start implementing new code but vast parts of the organization (the poor) were still stuck on the current release. In particular, test teams rarely freed up from a release until it shipped so new code wasn’t thoroughly tested in the beginning of a project and “unhealthy” teams always lagged behind, putting the finishing touches on the current release and falling further and further behind. These teams were also often the ones with the lowest morale and highest attrition meaning that the engineers inherited fragile code they hadn't written and hence didn't understand.

For most of the duration of Vista/Longhorn, I was responsible for the storage and file systems technologies. That meant I was involved with the WinFS effort although it was driven primarily by the SQL database team. Bill Gates was personally involved at a very detailed level and was even jokingly referred to as “the WinFS PM”: the program manager responsible for the project. Hundreds, if not thousands, of man years of engineering went into an idea whose time had simply passed: what if we combine the query capabilities of the database with the streaming capabilities and unstructured data functionality of the file system and expose it as a programming paradigm for the creation of unique new “rich” applications.
In hindsight, it’s obvious that Google handily solved this problem, providing a seamless and fast indexing experience for unstructured data. And they did so for the entire internet, not just for your local disk. And you didn’t even need to rewrite your applications to take advantage of it. When Longhorn was cancelled and Vista was hastily put together from its smoldering embers, WinFS was kicked out of the OS release. It was pursued by the SQL team for a few more years as a standalone project. By this time, Windows had a built-in indexing engine and integrated search experience - implemented purely on the side with no application changes needed. So the relevance of WinFS became even murkier but the project still carried on.
The massive security related architectural changes in Longhorn were kept as part of the Windows Vista project. We had learned a lot about security in the rapidly expanding internet universe and wanted to apply those learnings at an architectural level in the OS to improve overall security for all customers. We had no choice. Windows XP had shown that we were victims of our own success. A system that was designed for usability fell far short in terms of security when confronted with the realities of the internet age. Meanwhile, Windows XP Service Pack 2 was another release simultaneously being pursued by the Windows team and sucking resources away from Longhorn. We couldn't exactly go backwards in terms of security in our next major OS release. So it was that Vista became massively more secure than any earlier OS shipped by Microsoft, but in the process also managed to break application and device driver compatibility in an unprecedented manner for the ecosystem. Customers hated it because their apps broke and ecosystem partners hated it because they felt they didn't have enough time to update and certify their drivers and applications as Vista was rushed out the door to compete with a resurgent Apple.
In many cases, these security changes meant deep architectural changes were required to third party solutions. And most ecosystem vendors were not incented to invest heavily in their legacy apps. I personally spent many years explaining to antivirus vendors why we would no longer allow them to “patch” kernel instructions and data structures in memory, why this was a security risk, and why they needed to use approved APIs going forward, that we would no longer support their legacy apps with deep hooks in the Windows kernel - the same ones that hackers were using to attack consumer systems. Our “friends”, the antivirus vendors, turned around and sued us, claiming we were blocking their livelihood and abusing our monopoly power! With friends like that, who needs enemies? They just wanted their old solutions to keep working even if that meant reducing the security of our mutual customer - the very thing they were supposed to be improving.
There were so many seismic shifts happening in the computing industry during those years - the advent of the internet, the rise of the mobile phone, the emergence of cloud computing, the creation of new ad-supported business models, the viral growth of social media, the relentless march of Moore's law, and the popularity of open source are just a few factors that assaulted Windows from all directions. The response, not surprisingly for a wildly successful platform, was to dig its heels in and keep incrementally improving the existing system - innovator’s dilemma in a nutshell. The more code we added, the more complexity we created, the larger the team got, the bigger the ecosystem, the harder it became to leapfrog the competition.

As if the competitive forces weren't enough, this was also the time when armies of engineers and program managers spent countless hours, days, weeks, and months with representatives from the DOJ and corporate lawyers, documenting existing APIs from previous releases in order to comply with the government's antitrust rulings.

The stark reality is that, at this point in its lifecycle, it took roughly three years to get a major release of Windows out the door and that was simply too slow for the fast moving market. WinFS, Security, and Managed Code were just a few of the massive projects on the agenda for Longhorn. There were also hundreds of smaller bets. When you have a multi-thousand person organization and literally billions of customers, everyone gets a say. The same OS release that is supposed to work on the forthcoming tablet and smartphone footprint is also supposed to work on your laptop, in servers running in the data center, and in embedded devices such as NAS boxes “Powered by Windows” - not to mention on top of a hypervisor (HyperV) in the cloud. The requirements pulled the team in opposite directions as we tried to make forward progress on all segments of the market simultaneously.



It's impossible to look at Longhorn and Vista in isolation. They make sense only when viewed in conjunction with the releases right before and right after them - Windows 2000 and XP on the one hand, Windows Server 2008 and Windows 7 on the other - and with full knowledge of the broader industry in retrospect. Windows was a victim of its own success. It had penetrated many markets successfully and each of those businesses now exerted some influence on the design of the operating system pulling it in different, and often conflicting, directions. Trying to deliver on all of those disparate requirements meant not satisfying any one of them completely. An architecture that had been massively successful during the nineties became bogged down a decade later because the world around us was changing ever more rapidly while the organization struggled to keep up with it. To be clear, we saw all these trends and we worked hard to respond to them but, if I may mix my metaphors, it was hard to turn an aircraft carrier on a dime when you're two years pregnant with a three year release.
In short, what we thought we knew three or four years ago when we planned a given OS release was laughably outdated and sometimes flat out wrong when the product finally shipped. The best thing we could have done was to enable incremental and friction-free delivery of new cloud based services to an ever-simplifying device. Instead, we kept adding features to an existing client-based monolithic system that required many months of testing before each release, slowing us down just when we needed to speed up. Now imagine supporting that same OS for a dozen years or more for a population of billions of customers, millions of companies, thousands of partners, hundreds of scenarios, and dozens of form factors - and you’ll begin to have an inkling of the support and compatibility nightmare. In hindsight, Linux has been more successful in this respect. The open source community and approach to software development is undoubtedly part of the solution. An organization, sooner or later, ships its org chart as its product; the Windows organization was no different. Open source doesn't have that problem.

Add to this, if you will, internal organizational dynamics and personalities. We all had our own favorite features, our own ecosystem partners pushing us to adopt new standards, to help them certify their solutions on the platform, to add APIs for their particular scenarios. We all had ambitions for proving that our technology, our idea would win the battle…  if we could just get it into the next release of Windows and instantly pick up millions of customers. We believed it enough to fight for it in planning meetings and war rooms. We also all had managers who wanted to get promoted and increase their sphere of influence or their team size, as a proxy. Dev and test teams were often at odds, the former pushing hard to get code checked in while the latter was rewarded for finding ever more complex and esoteric test cases that had no earthly resemblance to customer environments. The internal dynamics were complex, to say the least. As if that weren't enough, at least once a year we had a massive reorg and new organizational dynamics to deal with.
None of this, by the way, should be taken as excuses or apologies. It is not intended in that sense.
Did we make mistakes? Yup, aplenty.
Did we intentionally make bad decisions? Nope, not that I can ever recall.
Was it an incredibly complex product with an amazingly huge ecosystem (the largest in the world at that time)? Yup, that it was.
Could we have done better? Yup, you bet.
Would we make different decisions today? Yup. Hindsight is 20/20. We didn't know then what we know now.
Should we look back in dismay or regret? No, I prefer looking at it as lessons learned. I'm pretty sure none of us went on to make the same set of mistakes on later projects. We learned from the experience - which means we made a whole different set of mistakes the next time. To err is human.