Sunday, June 18, 2017

Our True Sixth Sense: Fiction

“Not everything that can be faced can be changed, but nothing can be changed that is not faced.”
James Baldwin. I am not Your Negro.

“In the 300 years of the crucifixion of Christ to the conversion of Emperor Constantine, polytheistic Roman emperors initiated no more than four general persecutions of Christians. Local administrators and governors incited some anti-Christian violence of their own. Still, if we combine all the victims of all these persecutions, it turns out that in these three centuries the polytheistic Romans killed no more than a few thousand Christians. In contrast, over the course, of the next 1,500 years, Christians slaughtered Christians by the millions, to defend slightly different interpretations of the religion of love and compassion.”
          Yuval Noah Harari. Sapiens: A Brief History of Humankind.

Hans: As Gandhi said... An eye for an eye leaves the whole world blind.
Billy: No, it doesn't. There'll be one guy left with one eye. How's the last blind guy gonna take out the eye of the last guy left? No, it doesn't. There'll be one guy left with one eye. How's the last blind guy gonna take out the eye of the last guy left?  All that guy has to do is run away and hide behind a bush. Gandhi was wrong. It’s just that no one has the guts to come out and say it.”
Christopher Walken and Sam Rockwell. Seven Psychopaths.

We humans do have a sixth sense but, despite popular belief, it’s not ESP (Extra Sensory Perception). In fact, I would argue that ESP is simply a small subset of our broader instinct. Our true sixth sense is our sense of fiction, our innate ability to weave narratives at every opportunity, our penchant for asking “what if” at every turn, our willingness to create or participate in stories that may or may not be rooted in the physical world around us. We’re the only species capable of keeping thousands of such complex scenarios in our mind, each a fiction that we personally or collectively believe in. Other species are capable of rudimentary deception (chimps can “lie” about where they hid a banana, for example) but we're the only species who absolutely wallows in fiction in practically every part of our lives – so much so that we’re surrounded by it at almost every moment, often without even consciously recognizing it.

That fiction may be Harry Potter or Game of Thrones but those are obvious examples and momentary diversions. The more common stories are the ones we live in all day and night – our common beliefs. The best analysis of this phenomenon I’ve ever seen is from Yuval Noah Harari in Sapiens: A Brief History of Humankind. We believe in soccer and basketball teams: concepts that have no basis in physical reality but are nonetheless “religiously” pursued by millions of people. Think about it: what exactly is the definition of the NFL team, The Oakland Raiders? It’s a moniker we apply to a group of men and watch every Sunday on television. The members of the team change every year, the team manager changes once every few years, the owners change once a decade or so, the team jersey and insignia are redesigned whenever ratings sag, the rules they follow are not only arbitrary but can also be changed by a committee at their whim, even the city they are supposed to represent changes on an infrequent basis. Remember The Los Angeles Raiders? Yup. Same team – for a few years. So what exactly is The Oakland Raiders or the NBA, for that matter?

What exactly do we mean when we say “I love the Oakland Raiders?” You can make the same declaration about roses, about Mt. Fuji, about cocker spaniels, and about the moon – with one big difference. The latter are all physical objects that have a fixed definition that could be recognized, if not verbalized, by even an animal. Pretty much everything else we ever talk about or believe in is made up, a fiction, a story that we tell ourselves and others. Including the Oakland Raiders. And religion and society and government and currencies and borders and corporations and brands. Everything on this new list is just a figment of our collective imagination – something that homo sapiens dreamed up. It’s amazing to think about the world around us through this lens; it frees us of many of the prejudices that we take for granted. I call us homo theatricales – we not only make up the stories, we also star in the shows.

Now let’s turn the same lens to a different scene. What does it mean for hooligans in Britain to kill each other over a Manchester United game or for terrorists from the Middle East to kill Christians for their beliefs or for a gunman to take up an AK-47 and walk into a crowded square because IRA? Every single one of those constructs, those sets of beliefs, is a figment of our imagination. If you don’t believe me, let’s look at countries. Think about the “line in the sand” that separates our modern nations. Step over this line through the accident of birth and, suddenly, villages that are separated by a mere five miles are mortal enemies. The proverbial “line in the sand” is just one example of such a fiction. Not a physical line in the sand, mind you. It's just a line somebody drew on a piece of paper about a hundred years ago, roughly speaking, for most countries. But it’s good enough for us to kill each other over.

Most of what we believe in - nations, corporations, religions, gold as a valuable metal, money as a piece of paper that has real value - these are all fictions we have created for ourselves as a species. It is what sustains us, it is what distinguishes us. It is, unfortunately, also what separates us – what turns us into competing bands.

So, I have to ask: if we all agree there is really no line in the sand separating, say, Israel and Lebanon, if we agree that such an imaginary line shouldn't mean the inhabitants on one side are “Arabs” and the guys on the other side are “Jews”, forever and ever - and, oh by the way, we just made up that line a hundred years ago; if we agree that a piece of paper, even when adorned with fancy colors and markings, a unit of currency, does not really have any intrinsic value of its own that would lead you to hand over a car or a jacket over for, that its only value is an index into a database of international monetary exchange rates - so we know that a dollar is worth 1.24762 Euros - even though both are nothing but pieces of brightly colored paper backed by a “promise”, a fiction that exists nowhere but in our collective consciousness; if we understand all this, then why are we so adamant that our side is right - when all the rules are imaginary? Why are we convinced that Catholics are right and Muslims are wrong, or vice versa? The very concepts and teachings of both religions are nothing but the collective beliefs of a group of people. Nothing more and nothing less. Why are we so adamant that North and South Korea are enemies, when neither of those countries - those ideologies - even existed a hundred years ago? They exist nowhere but on paper and in our minds.

Almost everything we get worked up about these days - nationalities, religions, sports teams, financial markets - are nothing but figments of our collective imagination. If you think of it that way, it's much easier to let go of the dogma, the irrational belief that my side is right and your side is wrong - and I don't care which side of the argument you're on, nor do I care which argument we're talking about.

If you think of it this way, furthermore, our sixth sense being our ability to spin yarns and create fictional scenarios in which we live - be they at the personal level, at the corporate level, or at the international level - then the idea of the sixth sense being ESP also starts to make sense. Extra Sensory Perception is nothing but us spinning yarns, making up stories about things, belief in an extra-sensory experience that simply does not exist. I claim every case of reported ESP is nothing but someone creating - or following - a fictional narrative. That doesn't mean they're lying. In their minds, they believe everything they are seeing. Our confirmation biases are too strong for that to not happen.

The lesson? Question your beliefs. Don’t follow them blindly. They are often nothing but fictions, stories that we have weaved for ourselves. As a species, we're damn good at doing that.

That democrats are assholes or that republicans are idiots, that moving a couple of kilometers across a border dramatically changes people’s belief systems and outlook on life, that Manchester United is better than Real Madrid because Ronaldo - these are all fictions in our heads.  I don't even know if that last sentence made any sense because I don't follow soccer. I just know Ronaldo is the name of a character in that universe, that narrative - and the other two are “teams” that may or may not be rivals.

The next time you get all worked up about something, anything, ask yourself: Could I explain this to a member of another species – any species. Ask any passing elephant or gorilla if they are citizens of Namibia or Botswana and you will see my point. Or try to give a ten dollar bill to a chimpanzee for his bananas. Good luck. We are the only species that believes these things. They are figments of our imagination - and, as such, malleable. If we only allow ourselves to be a bit more flexible on our “beliefs”.

So what does it mean to get all bent out of shape over an imaginary line in the sand? Or a religion for that matter?

These are all fictions. We are all the same. Get over yourself.

Thursday, June 8, 2017

What Really Happened with Vista: An Insider's Retrospective

This is the original version of this blog. For the current updated version, please go here.

“Experience is something you don't get until just after you need it.”
Steven Wright.

I enjoyed reading Terry Crowley’s thoughtful blog (What Really Happened with Vista). Terry worked in the Office organization and did a fantastic job covering the complex machinations that went into Windows Vista and the related but doomed Longhorn project - from an outsider’s point of view. He correctly identified many of the problems that dogged the project and I don’t mean to rehash any of them here. I figured it was only fair to try to offer an insider’s view of the same events. I can’t hope to be as eloquent or thorough as Terry but hope to shed some light on what went wrong. Ten years have gone by since the original release date of Windows Vista but the lessons seem more relevant now than ever.

Windows is a beast. Thousands of developers, testers, program managers, security experts, UI designers, architects, you name it. And that’s before the supporting cast of HR people, recruiters, marketing folks, salespeople, lawyers, and of course many managers, directors, and vice presidents for each of the disciplines mentioned above. The entire ensemble cast is supported by many thousands of others at partner teams (within Microsoft as well as outside) that deliver hardware underneath or solutions on top of the platform.

Organizationally, at the time, Windows was really three teams: Core, Server, and Client. The core team delivered the “plumbing”, all the core components of the operating system (the kernel itself, storage, security, networking, device drivers, the installation and upgrade model, Win32, etc) shared by all versions of Windows. The server team, in turn, concentrated on technologies needed for the server market (terminal services, clustering and high availability, enterprise management tools, etc) while the client team was responsible for technologies related to the desktop and consumer releases (web browser, media player, graphics, shell, etc). There were, of course, many reorgs but that basic structure was kept in place even as Windows grew in popularity and the teams grew in size. It would also be fair to say, culturally and organizationally speaking, that the core team was closer to the server team than it was to the client team - at least until after Vista shipped.

By the time I arrived at Microsoft, in early 1998, Windows meant Windows NT - architecturally, organizationally, and product wise. The Windows 95 code base had largely been abandoned and Windows NT had been adopted for every personality of Windows - from the laptop to the clustered Server. Two years later, the Windows 95/98 code base would be resurrected for one last release - the much maligned Windows ME - but that project was executed by a small team while the vast majority worked on the NT code base. I was lucky enough to spend a dozen years in the belly of the beast, joining during the heyday of Windows 2000 development and staying through to the completion of Windows 7.

I spent the first seven years of my tenure managing the teams responsible for storage, file systems, high availability/clustering, file level network protocols, distributed file systems, and related technologies. Later, I spent a year or two managing security for Microsoft. This included everything from security technologies in Windows to antivirus products as add-on solutions to security marketing and emergency response such as security patches. This was towards the tail end of Vista when viruses and worms were bringing Windows to its knees and when Microsoft's reputation for building secure software had taken a massive beating in the marketplace. For the last three or four years, for the duration of the Windows 7 release, I managed all core development in Windows. That meant dev ownership of pretty much all technologies running “under the hood” and used by both the client and server teams. After Vista shipped, the Windows team was organized by disciplines and a trio (Dev, Test, PM) was put in charge at every level of the org so I had two partners in crime. I managed the development teams while they managed, respectively, test and program management.
The Windows team had a history of trying massive projects that were often abandoned or repurposed after a few years. An earlier example was the ambitious Cairo project which was eventually gutted, with some pieces salvaged and shipped as part of Windows 2000. By far the biggest problem with Windows releases, in my humble opinion, was the duration of each release. On average, a release took about three years from inception to completion but only about six to nine months of that time was spent developing “new” code. The rest of the time was spent in integration, testing, alpha and beta periods - each lasting a few months. Some projects needed more than six months of core development so they proceeded in parallel and merged with the main code base when ready. This meant that the main tree was almost always in a semi-broken state as large pieces of functionality were being integrated or replaced. Much tighter controls were put in place during the Windows 7 release to ensure a constantly healthy and functioning code base but earlier releases were plagued with daily instability for months at a time.
The chaotic nature of development often resulted in teams playing schedule chicken, convincing themselves and others that their code was in better shape than other projects, that they could “polish” the few remaining pieces of work just in time, so they would be allowed to checkin their component in a half-finished state. The three year release cycle meant we rarely knew what the competitive landscape and external ecosystem would look like when we started a release. Missing a release meant cancellation (as the feature rarely made sense six years later) or, worse, banishment to Siberia - continued development on a component that was mostly ignored by the rest of the organization and doomed to eventual failure or irrelevance, but one that the team or the execs simply couldn’t bring themselves to abandon. I was personally responsible for a few such projects. Hindsight is 20/20.
Given that each team was busy pushing their own agenda and features into the release, they often skimped on integration with other components, user interface, end to end testing, and ugly and tedious issues such as upgrade, leaving these thorny issues for the endgame. That, in turn, meant some teams quickly became bottlenecks as everyone jockeyed for their help in finishing their UI or upgrade testing at the last minute.
At any given point in time, there were multiple major releases in progress as well as multiple side projects. Different teams were responsible for code bases in various states of health resulting in a model where “the rich got richer and the poor got poorer” over time. As a project neared completion, program managers would start looking at requirements for the next release and developers in “healthy” (rich) teams would start implementing new code but vast parts of the organization (the poor) were still stuck on the current release. In particular, test teams rarely freed up from a release until it shipped so new code wasn’t thoroughly tested in the beginning of a project and “unhealthy” teams always lagged behind, putting the finishing touches on the current release and falling further and further behind. These teams were also often the ones with the lowest morale and highest attrition meaning that the engineers inherited fragile code they hadn't written and hence didn't understand.

For most of the duration of Vista/Longhorn, I was responsible for the storage and file systems technologies. That meant I was involved with the WinFS effort although it was driven primarily by the SQL database team. Bill Gates was personally involved at a very detailed level and was even jokingly referred to as “the WinFS PM”: the program manager responsible for the project. Hundreds, if not thousands, of man years of engineering went into an idea whose time had simply passed: what if we combine the query capabilities of the database with the streaming capabilities and unstructured data functionality of the file system and expose it as a programming paradigm for the creation of unique new “rich” applications.
In hindsight, it’s obvious that Google handily solved this problem, providing a seamless and fast indexing experience for unstructured data. And they did so for the entire internet, not just for your local disk. And you didn’t even need to rewrite your applications to take advantage of it. When Longhorn was cancelled and Vista was hastily put together from its smoldering embers, WinFS was kicked out of the OS release. It was pursued by the SQL team for a few more years as a standalone project. By this time, Windows had a built-in indexing engine and integrated search experience - implemented purely on the side with no application changes needed. So the relevance of WinFS became even murkier but the project still carried on.
The massive security related architectural changes in Longhorn were kept as part of the Windows Vista project. We had learned a lot about security in the rapidly expanding internet universe and wanted to apply those learnings at an architectural level in the OS to improve overall security for all customers. We had no choice. Windows XP had shown that we were victims of our own success. A system that was designed for usability fell far short in terms of security when confronted with the realities of the internet age. Meanwhile, Windows XP Service Pack 2 was another release simultaneously being pursued by the Windows team and sucking resources away from Longhorn. We couldn't exactly go backwards in terms of security in our next major OS release. So it was that Vista became massively more secure than any earlier OS shipped by Microsoft, but in the process also managed to break application and device driver compatibility in an unprecedented manner for the ecosystem. Customers hated it because their apps broke and ecosystem partners hated it because they felt they didn't have enough time to update and certify their drivers and applications as Vista was rushed out the door to compete with a resurgent Apple.
In many cases, these security changes meant deep architectural changes were required to third party solutions. And most ecosystem vendors were not incented to invest heavily in their legacy apps. I personally spent many years explaining to antivirus vendors why we would no longer allow them to “patch” kernel instructions and data structures in memory, why this was a security risk, and why they needed to use approved APIs going forward, that we would no longer support their legacy apps with deep hooks in the Windows kernel - the same ones that hackers were using to attack consumer systems. Our “friends”, the antivirus vendors, turned around and sued us, claiming we were blocking their livelihood and abusing our monopoly power! With friends like that, who needs enemies? They just wanted their old solutions to keep working even if that meant reducing the security of our mutual customer - the very thing they were supposed to be improving.
There were so many seismic shifts happening in the computing industry during those years - the advent of the internet, the rise of the mobile phone, the emergence of cloud computing, the creation of new ad-supported business models, the viral growth of social media, the relentless march of Moore's law, and the popularity of open source are just a few factors that assaulted Windows from all directions. The response, not surprisingly for a wildly successful platform, was to dig its heels in and keep incrementally improving the existing system - innovator’s dilemma in a nutshell. The more code we added, the more complexity we created, the larger the team got, the bigger the ecosystem, the harder it became to leapfrog the competition.

As if the competitive forces weren't enough, this was also the time when armies of engineers and program managers spent countless hours, days, weeks, and months with representatives from the DOJ and corporate lawyers, documenting existing APIs from previous releases in order to comply with the government's antitrust rulings.

The stark reality is that, at this point in its lifecycle, it took roughly three years to get a major release of Windows out the door and that was simply too slow for the fast moving market. WinFS, Security, and Managed Code were just a few of the massive projects on the agenda for Longhorn. There were also hundreds of smaller bets. When you have a multi-thousand person organization and literally billions of customers, everyone gets a say. The same OS release that is supposed to work on the forthcoming tablet and smartphone footprint is also supposed to work on your laptop, in servers running in the data center, and in embedded devices such as NAS boxes “Powered by Windows” - not to mention on top of a hypervisor (HyperV) in the cloud. The requirements pulled the team in opposite directions as we tried to make forward progress on all segments of the market simultaneously.

It's impossible to look at Longhorn and Vista in isolation. They make sense only when viewed in conjunction with the releases right before and right after them - Windows 2000 and XP on the one hand, Windows Server 2008 and Windows 7 on the other - and with full knowledge of the broader industry in retrospect. Windows was a victim of its own success. It had penetrated many markets successfully and each of those businesses now exerted some influence on the design of the operating system pulling it in different, and often conflicting, directions. Trying to deliver on all of those disparate requirements meant not satisfying any one of them completely. An architecture that had been massively successful during the nineties became bogged down a decade later because the world around us was changing ever more rapidly while the organization struggled to keep up with it. To be clear, we saw all these trends and we worked hard to respond to them but, if I may mix my metaphors, it was hard to turn an aircraft carrier on a dime when you're two years pregnant with a three year release.
In short, what we thought we knew three or four years ago when we planned a given OS release was laughably outdated and sometimes flat out wrong when the product finally shipped. The best thing we could have done was to enable incremental and friction-free delivery of new cloud based services to an ever-simplifying device. Instead, we kept adding features to an existing client-based monolithic system that required many months of testing before each release, slowing us down just when we needed to speed up. Now imagine supporting that same OS for a dozen years or more for a population of billions of customers, millions of companies, thousands of partners, hundreds of scenarios, and dozens of form factors - and you’ll begin to have an inkling of the support and compatibility nightmare. In hindsight, Linux has been more successful in this respect. The open source community and approach to software development is undoubtedly part of the solution. An organization, sooner or later, ships its org chart as its product; the Windows organization was no different. Open source doesn't have that problem.

Add to this, if you will, internal organizational dynamics and personalities. We all had our own favorite features, our own ecosystem partners pushing us to adopt new standards, to help them certify their solutions on the platform, to add APIs for their particular scenarios. We all had ambitions for proving that our technology, our idea would win the battle…  if we could just get it into the next release of Windows and instantly pick up millions of customers. We believed it enough to fight for it in planning meetings and war rooms. We also all had managers who wanted to get promoted and increase their sphere of influence or their team size, as a proxy. Dev and test teams were often at odds, the former pushing hard to get code checked in while the latter was rewarded for finding ever more complex and esoteric test cases that had no earthly resemblance to customer environments. The internal dynamics were complex, to say the least. As if that weren't enough, at least once a year we had a massive reorg and new organizational dynamics to deal with.
None of this, by the way, should be taken as excuses or apologies. It is not intended in that sense.
Did we make mistakes? Yup, aplenty.
Did we intentionally make bad decisions? Nope, not that I can ever recall.
Was it an incredibly complex product with an amazingly huge ecosystem (the largest in the world at that time)? Yup, that it was.
Could we have done better? Yup, you bet.
Would we make different decisions today? Yup. Hindsight is 20/20. We didn't know then what we know now.
Should we look back in dismay or regret? No, I prefer looking at it as lessons learned. I'm pretty sure none of us went on to make the same set of mistakes on later projects. We learned from the experience - which means we made a whole different set of mistakes the next time. To err is human.