Sunday, June 18, 2017

Our True Sixth Sense: Fiction

“Not everything that can be faced can be changed, but nothing can be changed that is not faced.”
James Baldwin. I am not Your Negro.

“In the 300 years of the crucifixion of Christ to the conversion of Emperor Constantine, polytheistic Roman emperors initiated no more than four general persecutions of Christians. Local administrators and governors incited some anti-Christian violence of their own. Still, if we combine all the victims of all these persecutions, it turns out that in these three centuries the polytheistic Romans killed no more than a few thousand Christians. In contrast, over the course, of the next 1,500 years, Christians slaughtered Christians by the millions, to defend slightly different interpretations of the religion of love and compassion.”
          Yuval Noah Harari. Sapiens: A Brief History of Humankind.

Hans: As Gandhi said... An eye for an eye leaves the whole world blind.
Billy: No, it doesn't. There'll be one guy left with one eye. How's the last blind guy gonna take out the eye of the last guy left? No, it doesn't. There'll be one guy left with one eye. How's the last blind guy gonna take out the eye of the last guy left?  All that guy has to do is run away and hide behind a bush. Gandhi was wrong. It’s just that no one has the guts to come out and say it.”
Christopher Walken and Sam Rockwell. Seven Psychopaths.

We humans do have a sixth sense but, despite popular belief, it’s not ESP (Extra Sensory Perception). In fact, I would argue that ESP is simply a small subset of our broader instinct. Our true sixth sense is our sense of fiction, our innate ability to weave narratives at every opportunity, our penchant for asking “what if” at every turn, our willingness to create or participate in stories that may or may not be rooted in the physical world around us. We’re the only species capable of keeping thousands of such complex scenarios in our mind, each a fiction that we personally or collectively believe in. Other species are capable of rudimentary deception (chimps can “lie” about where they hid a banana, for example) but we're the only species who absolutely wallows in fiction in practically every part of our lives – so much so that we’re surrounded by it at almost every moment, often without even consciously recognizing it.

That fiction may be Harry Potter or Game of Thrones but those are obvious examples and momentary diversions. The more common stories are the ones we live in all day and night – our common beliefs. The best analysis of this phenomenon I’ve ever seen is from Yuval Noah Harari in Sapiens: A Brief History of Humankind. We believe in soccer and basketball teams: concepts that have no basis in physical reality but are nonetheless “religiously” pursued by millions of people. Think about it: what exactly is the definition of the NFL team, The Oakland Raiders? It’s a moniker we apply to a group of men and watch every Sunday on television. The members of the team change every year, the team manager changes once every few years, the owners change once a decade or so, the team jersey and insignia are redesigned whenever ratings sag, the rules they follow are not only arbitrary but can also be changed by a committee at their whim, even the city they are supposed to represent changes on an infrequent basis. Remember The Los Angeles Raiders? Yup. Same team – for a few years. So what exactly is The Oakland Raiders or the NBA, for that matter?

What exactly do we mean when we say “I love the Oakland Raiders?” You can make the same declaration about roses, about Mt. Fuji, about cocker spaniels, and about the moon – with one big difference. The latter are all physical objects that have a fixed definition that could be recognized, if not verbalized, by even an animal. Pretty much everything else we ever talk about or believe in is made up, a fiction, a story that we tell ourselves and others. Including the Oakland Raiders. And religion and society and government and currencies and borders and corporations and brands. Everything on this new list is just a figment of our collective imagination – something that homo sapiens dreamed up. It’s amazing to think about the world around us through this lens; it frees us of many of the prejudices that we take for granted. I call us homo theatricales – we not only make up the stories, we also star in the shows.

Now let’s turn the same lens to a different scene. What does it mean for hooligans in Britain to kill each other over a Manchester United game or for terrorists from the Middle East to kill Christians for their beliefs or for a gunman to take up an AK-47 and walk into a crowded square because IRA? Every single one of those constructs, those sets of beliefs, is a figment of our imagination. If you don’t believe me, let’s look at countries. Think about the “line in the sand” that separates our modern nations. Step over this line through the accident of birth and, suddenly, villages that are separated by a mere five miles are mortal enemies. The proverbial “line in the sand” is just one example of such a fiction. Not a physical line in the sand, mind you. It's just a line somebody drew on a piece of paper about a hundred years ago, roughly speaking, for most countries. But it’s good enough for us to kill each other over.

Most of what we believe in - nations, corporations, religions, gold as a valuable metal, money as a piece of paper that has real value - these are all fictions we have created for ourselves as a species. It is what sustains us, it is what distinguishes us. It is, unfortunately, also what separates us – what turns us into competing bands.

So, I have to ask: if we all agree there is really no line in the sand separating, say, Israel and Lebanon, if we agree that such an imaginary line shouldn't mean the inhabitants on one side are “Arabs” and the guys on the other side are “Jews”, forever and ever - and, oh by the way, we just made up that line a hundred years ago; if we agree that a piece of paper, even when adorned with fancy colors and markings, a unit of currency, does not really have any intrinsic value of its own that would lead you to hand over a car or a jacket over for, that its only value is an index into a database of international monetary exchange rates - so we know that a dollar is worth 1.24762 Euros - even though both are nothing but pieces of brightly colored paper backed by a “promise”, a fiction that exists nowhere but in our collective consciousness; if we understand all this, then why are we so adamant that our side is right - when all the rules are imaginary? Why are we convinced that Catholics are right and Muslims are wrong, or vice versa? The very concepts and teachings of both religions are nothing but the collective beliefs of a group of people. Nothing more and nothing less. Why are we so adamant that North and South Korea are enemies, when neither of those countries - those ideologies - even existed a hundred years ago? They exist nowhere but on paper and in our minds.

Almost everything we get worked up about these days - nationalities, religions, sports teams, financial markets - are nothing but figments of our collective imagination. If you think of it that way, it's much easier to let go of the dogma, the irrational belief that my side is right and your side is wrong - and I don't care which side of the argument you're on, nor do I care which argument we're talking about.

If you think of it this way, furthermore, our sixth sense being our ability to spin yarns and create fictional scenarios in which we live - be they at the personal level, at the corporate level, or at the international level - then the idea of the sixth sense being ESP also starts to make sense. Extra Sensory Perception is nothing but us spinning yarns, making up stories about things, belief in an extra-sensory experience that simply does not exist. I claim every case of reported ESP is nothing but someone creating - or following - a fictional narrative. That doesn't mean they're lying. In their minds, they believe everything they are seeing. Our confirmation biases are too strong for that to not happen.

The lesson? Question your beliefs. Don’t follow them blindly. They are often nothing but fictions, stories that we have weaved for ourselves. As a species, we're damn good at doing that.

That democrats are assholes or that republicans are idiots, that moving a couple of kilometers across a border dramatically changes people’s belief systems and outlook on life, that Manchester United is better than Real Madrid because Ronaldo - these are all fictions in our heads.  I don't even know if that last sentence made any sense because I don't follow soccer. I just know Ronaldo is the name of a character in that universe, that narrative - and the other two are “teams” that may or may not be rivals.

The next time you get all worked up about something, anything, ask yourself: Could I explain this to a member of another species – any species. Ask any passing elephant or gorilla if they are citizens of Namibia or Botswana and you will see my point. Or try to give a ten dollar bill to a chimpanzee for his bananas. Good luck. We are the only species that believes these things. They are figments of our imagination - and, as such, malleable. If we only allow ourselves to be a bit more flexible on our “beliefs”.

So what does it mean to get all bent out of shape over an imaginary line in the sand? Or a religion for that matter?


These are all fictions. We are all the same. Get over yourself.

Thursday, June 8, 2017

What Really Happened with Vista: An Insider's Retrospective

“Experience is something you don't get until just after you need it.”
Steven Wright.

I enjoyed reading Terry Crowley’s thoughtful blog (What Really Happened with Vista). Terry worked in the Office organization and did a fantastic job covering the complex machinations that went into Windows Vista and the related but doomed Longhorn project - from an outsider’s point of view. He correctly identified many of the problems that dogged the project and I don’t mean to rehash any of them here. I figured it was only fair to try to offer an insider’s view of the same events. I can’t hope to be as eloquent or thorough as Terry but hope to shed some light on what went wrong. Ten years have gone by since the original release date of Windows Vista but the lessons seem more relevant now than ever.

Windows is a beast. Thousands of developers, testers, program managers, security experts, UI designers, architects, you name it. And that’s before the supporting cast of HR people, recruiters, marketing folks, salespeople, lawyers, and of course many managers, directors, and vice presidents for each of the disciplines mentioned above. The entire ensemble cast is supported by many thousands of others at partner teams (within Microsoft as well as outside) that deliver hardware underneath or solutions on top of the platform.



Organizationally, at the time, Windows was really three teams: Core, Server, and Client. The core team delivered the “plumbing”, all the core components of the operating system (the kernel itself, storage, security, networking, device drivers, the installation and upgrade model, Win32, etc) shared by all versions of Windows. The server team, in turn, concentrated on technologies needed for the server market (terminal services, clustering and high availability, enterprise management tools, etc) while the client team was responsible for technologies related to the desktop and consumer releases (web browser, media player, graphics, shell, etc). There were, of course, many reorgs but that basic structure was kept in place even as Windows grew in popularity and the teams grew in size. It would also be fair to say, culturally and organizationally speaking, that the core team was closer to the server team than it was to the client team - at least until after Vista shipped.

By the time I arrived at Microsoft, in early 1998, Windows meant Windows NT - architecturally, organizationally, and product wise. The Windows 95 code base had largely been abandoned and Windows NT had been adopted for every personality of Windows - from the laptop to the clustered Server. Two years later, the Windows 95/98 code base would be resurrected for one last release - the much maligned Windows ME - but that project was executed by a small team while the vast majority worked on the NT code base. I was lucky enough to spend a dozen years in the belly of the beast, joining during the heyday of Windows 2000 development and staying through to the completion of Windows 7.


I spent the first seven years of my tenure managing the teams responsible for storage, file systems, high availability/clustering, file level network protocols, distributed file systems, and related technologies. Later, I spent a year or two managing security for Microsoft. This included everything from security technologies in Windows to antivirus products as add-on solutions to security marketing and emergency response such as security patches. This was towards the tail end of Vista when viruses and worms were bringing Windows to its knees and when Microsoft's reputation for building secure software had taken a massive beating in the marketplace. For the last three or four years, for the duration of the Windows 7 release, I managed all core development in Windows. That meant dev ownership of pretty much all technologies running “under the hood” and used by both the client and server teams. After Vista shipped, the Windows team was organized by disciplines and a trio (Dev, Test, PM) was put in charge at every level of the org so I had two partners in crime. I managed the development teams while they managed, respectively, test and program management.
The Windows team had a history of trying massive projects that were often abandoned or repurposed after a few years. An earlier example was the ambitious Cairo project which was eventually gutted, with some pieces salvaged and shipped as part of Windows 2000. By far the biggest problem with Windows releases, in my humble opinion, was the duration of each release. On average, a release took about three years from inception to completion but only about six to nine months of that time was spent developing “new” code. The rest of the time was spent in integration, testing, alpha and beta periods - each lasting a few months. Some projects needed more than six months of core development so they proceeded in parallel and merged with the main code base when ready. This meant that the main tree was almost always in a semi-broken state as large pieces of functionality were being integrated or replaced. Much tighter controls were put in place during the Windows 7 release to ensure a constantly healthy and functioning code base but earlier releases were plagued with daily instability for months at a time.
The chaotic nature of development often resulted in teams playing schedule chicken, convincing themselves and others that their code was in better shape than other projects, that they could “polish” the few remaining pieces of work just in time, so they would be allowed to checkin their component in a half-finished state. The three year release cycle meant we rarely knew what the competitive landscape and external ecosystem would look like when we started a release. Missing a release meant cancellation (as the feature rarely made sense six years later) or, worse, banishment to Siberia - continued development on a component that was mostly ignored by the rest of the organization and doomed to eventual failure or irrelevance, but one that the team or the execs simply couldn’t bring themselves to abandon. I was personally responsible for a few such projects. Hindsight is 20/20.
Given that each team was busy pushing their own agenda and features into the release, they often skimped on integration with other components, user interface, end to end testing, and ugly and tedious issues such as upgrade, leaving these thorny issues for the endgame. That, in turn, meant some teams quickly became bottlenecks as everyone jockeyed for their help in finishing their UI or upgrade testing at the last minute.
At any given point in time, there were multiple major releases in progress as well as multiple side projects. Different teams were responsible for code bases in various states of health resulting in a model where “the rich got richer and the poor got poorer” over time. As a project neared completion, program managers would start looking at requirements for the next release and developers in “healthy” (rich) teams would start implementing new code but vast parts of the organization (the poor) were still stuck on the current release. In particular, test teams rarely freed up from a release until it shipped so new code wasn’t thoroughly tested in the beginning of a project and “unhealthy” teams always lagged behind, putting the finishing touches on the current release and falling further and further behind. These teams were also often the ones with the lowest morale and highest attrition meaning that the engineers inherited fragile code they hadn't written and hence didn't understand.

For most of the duration of Vista/Longhorn, I was responsible for the storage and file systems technologies. That meant I was involved with the WinFS effort although it was driven primarily by the SQL database team. Bill Gates was personally involved at a very detailed level and was even jokingly referred to as “the WinFS PM”: the program manager responsible for the project. Hundreds, if not thousands, of man years of engineering went into an idea whose time had simply passed: what if we combine the query capabilities of the database with the streaming capabilities and unstructured data functionality of the file system and expose it as a programming paradigm for the creation of unique new “rich” applications.
In hindsight, it’s obvious that Google handily solved this problem, providing a seamless and fast indexing experience for unstructured data. And they did so for the entire internet, not just for your local disk. And you didn’t even need to rewrite your applications to take advantage of it. When Longhorn was cancelled and Vista was hastily put together from its smoldering embers, WinFS was kicked out of the OS release. It was pursued by the SQL team for a few more years as a standalone project. By this time, Windows had a built-in indexing engine and integrated search experience - implemented purely on the side with no application changes needed. So the relevance of WinFS became even murkier but the project still carried on.
The massive security related architectural changes in Longhorn were kept as part of the Windows Vista project. We had learned a lot about security in the rapidly expanding internet universe and wanted to apply those learnings at an architectural level in the OS to improve overall security for all customers. We had no choice. Windows XP had shown that we were victims of our own success. A system that was designed for usability fell far short in terms of security when confronted with the realities of the internet age. Meanwhile, Windows XP Service Pack 2 was another release simultaneously being pursued by the Windows team and sucking resources away from Longhorn. We couldn't exactly go backwards in terms of security in our next major OS release. So it was that Vista became massively more secure than any earlier OS shipped by Microsoft, but in the process also managed to break application and device driver compatibility in an unprecedented manner for the ecosystem. Customers hated it because their apps broke and ecosystem partners hated it because they felt they didn't have enough time to update and certify their drivers and applications as Vista was rushed out the door to compete with a resurgent Apple.
In many cases, these security changes meant deep architectural changes were required to third party solutions. And most ecosystem vendors were not incented to invest heavily in their legacy apps. I personally spent many years explaining to antivirus vendors why we would no longer allow them to “patch” kernel instructions and data structures in memory, why this was a security risk, and why they needed to use approved APIs going forward, that we would no longer support their legacy apps with deep hooks in the Windows kernel - the same ones that hackers were using to attack consumer systems. Our “friends”, the antivirus vendors, turned around and sued us, claiming we were blocking their livelihood and abusing our monopoly power! With friends like that, who needs enemies? They just wanted their old solutions to keep working even if that meant reducing the security of our mutual customer - the very thing they were supposed to be improving.
There were so many seismic shifts happening in the computing industry during those years - the advent of the internet, the rise of the mobile phone, the emergence of cloud computing, the creation of new ad-supported business models, the viral growth of social media, the relentless march of Moore's law, and the popularity of open source are just a few factors that assaulted Windows from all directions. The response, not surprisingly for a wildly successful platform, was to dig its heels in and keep incrementally improving the existing system - innovator’s dilemma in a nutshell. The more code we added, the more complexity we created, the larger the team got, the bigger the ecosystem, the harder it became to leapfrog the competition.

As if the competitive forces weren't enough, this was also the time when armies of engineers and program managers spent countless hours, days, weeks, and months with representatives from the DOJ and corporate lawyers, documenting existing APIs from previous releases in order to comply with the government's antitrust rulings.

The stark reality is that, at this point in its lifecycle, it took roughly three years to get a major release of Windows out the door and that was simply too slow for the fast moving market. WinFS, Security, and Managed Code were just a few of the massive projects on the agenda for Longhorn. There were also hundreds of smaller bets. When you have a multi-thousand person organization and literally billions of customers, everyone gets a say. The same OS release that is supposed to work on the forthcoming tablet and smartphone footprint is also supposed to work on your laptop, in servers running in the data center, and in embedded devices such as NAS boxes “Powered by Windows” - not to mention on top of a hypervisor (HyperV) in the cloud. The requirements pulled the team in opposite directions as we tried to make forward progress on all segments of the market simultaneously.



It's impossible to look at Longhorn and Vista in isolation. They make sense only when viewed in conjunction with the releases right before and right after them - Windows 2000 and XP on the one hand, Windows Server 2008 and Windows 7 on the other - and with full knowledge of the broader industry in retrospect. Windows was a victim of its own success. It had penetrated many markets successfully and each of those businesses now exerted some influence on the design of the operating system pulling it in different, and often conflicting, directions. Trying to deliver on all of those disparate requirements meant not satisfying any one of them completely. An architecture that had been massively successful during the nineties became bogged down a decade later because the world around us was changing ever more rapidly while the organization struggled to keep up with it. To be clear, we saw all these trends and we worked hard to respond to them but, if I may mix my metaphors, it was hard to turn an aircraft carrier on a dime when you're two years pregnant with a three year release.
In short, what we thought we knew three or four years ago when we planned a given OS release was laughably outdated and sometimes flat out wrong when the product finally shipped. The best thing we could have done was to enable incremental and friction-free delivery of new cloud based services to an ever-simplifying device. Instead, we kept adding features to an existing client-based monolithic system that required many months of testing before each release, slowing us down just when we needed to speed up. Now imagine supporting that same OS for a dozen years or more for a population of billions of customers, millions of companies, thousands of partners, hundreds of scenarios, and dozens of form factors - and you’ll begin to have an inkling of the support and compatibility nightmare. In hindsight, Linux has been more successful in this respect. The open source community and approach to software development is undoubtedly part of the solution. An organization, sooner or later, ships its org chart as its product; the Windows organization was no different. Open source doesn't have that problem.

Add to this, if you will, internal organizational dynamics and personalities. We all had our own favorite features, our own ecosystem partners pushing us to adopt new standards, to help them certify their solutions on the platform, to add APIs for their particular scenarios. We all had ambitions for proving that our technology, our idea would win the battle…  if we could just get it into the next release of Windows and instantly pick up millions of customers. We believed it enough to fight for it in planning meetings and war rooms. We also all had managers who wanted to get promoted and increase their sphere of influence or their team size, as a proxy. Dev and test teams were often at odds, the former pushing hard to get code checked in while the latter was rewarded for finding ever more complex and esoteric test cases that had no earthly resemblance to customer environments. The internal dynamics were complex, to say the least. As if that weren't enough, at least once a year we had a massive reorg and new organizational dynamics to deal with.
None of this, by the way, should be taken as excuses or apologies. It is not intended in that sense.
Did we make mistakes? Yup, aplenty.
Did we intentionally make bad decisions? Nope, not that I can ever recall.
Was it an incredibly complex product with an amazingly huge ecosystem (the largest in the world at that time)? Yup, that it was.
Could we have done better? Yup, you bet.
Would we make different decisions today? Yup. Hindsight is 20/20. We didn't know then what we know now.
Should we look back in dismay or regret? No, I prefer looking at it as lessons learned. I'm pretty sure none of us went on to make the same set of mistakes on later projects. We learned from the experience - which means we made a whole different set of mistakes the next time. To err is human.

Saturday, May 20, 2017

To Eden on Her Sixth Birthday: A Plea for Education Reform

“Now the problem with standardized tests is that it's based on the mistake that we can simply scale up the education of children like you would scale up making carburetors. And we can't, because human beings are very different from motorcars, and they have feelings about what they do and motivations in doing it.”
        Sir Ken Robinson.

“It’s okay, Uncle Ben. I'm sure I'll appreciate him when I’m grown up.”
Eden, handing me back the headphones and letting me down easy after her first experience with Bob Dylan.

“You know… it's not a good thing that hammocks don't have seat belts.”
         Eden, having just fallen out of one.

“Only by counting could humans demonstrate their independence of computers.”
Douglas Adams. Hitchhiker’s Guide to the Galaxy.
My niece, Eden, turned six recently. She came to visit us in California for her birthday. We went to Disneyland together and spent a few days at home as well. Typical spring break for a six year old living in New Jersey. Eden and I end up spending quite a bit of time together when she visits, mostly sitting in the backyard and talking. She relishes the California sunshine and warmth after a cold New Jersey winter spent cooped up in the house. And I, too, admit to being a sun worshipper, enjoying the backyard and California’s seemingly perpetual springtime.

These conversations with Eden can revolve around anything: recent events in school, what happened on a cartoon she was recently watching, the latest happenings on Wall Street, what Donald Trump said about the Middle East crisis, you name it. I kid you not. I have learned to talk to her like an adult and am constantly amazed by how much she can absorb. Her mom recently sent us a video she had taped of Eden in their living room as she pretended to deliver the evening news while sitting in a cardboard cutout TV. Her monologue included relevant commentary on the political situation in DC, the weather in LA and Jerusalem for the next week, the stock market, traffic on the George Washington bridge, and other topics she had picked up on TV. She obviously doesn't understand all the nuances but is smart enough and has learned enough to participate meaningfully in an adult conversation - or, in this case, presentation.

Eden, in my honest opinion, is brilliant. She speaks three languages fluently without ever having been “taught” them. She can play tunes on the piano without ever having learned to read notes. She has an amazing ear for music, a seemingly magical ability to someone like me who couldn't speak “piano-ese” any more than Chinese. She already reads at a third grade level and is, consequently, often bored in school given that she's still stuck in kindergarten! I’m sure the teachers struggle to keep her engaged in class.

Her high level of intelligence also means she often gets frustrated with the “childish” environment she is forced to inhabit. For the first couple of days of her visit, I couldn't get her to read even a picture book, one that should have been a piece of cake. She refused to concentrate and intentionally misread simple words. Once I caught her not even looking at the page as she recited the words, apparently from perfect memory of a previous reading. She was telling me she was bored with the exercise, that it was too simple. Then I put a much harder book in front of her, one with a hundred words per page instead of just ten. She immediately started reading with no problem whatsoever!

Every time I sit with Eden, I'm reminded of this amazing TED talk by Sir Ken Robinson on education and how we need to completely rethink our approach to it for the next century. Children need individualized attention and they need to be challenged mentally. Our model of education, however, is still rooted in rote memorization, standardized tests, and principles based on the needs of the industrial revolution and the eighteenth century. We take these brilliant minds and force them to sit through a dozen or more years of institutionalized hell called primary and secondary education, memorizing formulas and theorems so they can answer multiple-choice questions on a test before promptly forgetting them. I’m convinced kids are interested in everything that we bother to make interesting for them. If they lose interest in a subject, chances are it's not because they don't “get” it but rather that they didn't “get” an earlier more important concept, quite possibly because the teacher didn't make the topic interesting.

A scant few will get the privilege of working with amazing teachers who will challenge them while the vast majority will be marginalized by an education system that looks backwards instead of forwards. How else do you explain this (and other) popular YouTube videos showing children in Indian villages calculating large sums by mentally simulating an abacus for their proud teacher? You can see them fidget with their fingers as they recreate a mental image of an abacus. Is this really how we want to educate our children? Is this really the skill they need to practice for hours on end so they can be successful in the twenty first century? This is an extreme example but I claim most of the world’s children don't go through a much better educational system.

Here we are with “blank slates” that hunger to learn, children who have the mental capacity to pick up three completely distinct human languages in just a couple of years of ad hoc practice, who teach themselves to play the piano, take your pick of amazing skills you’ve seen children display. We take these geniuses - there is no other word for it - with the massive computing engines they carry around all day, and we sit them down and tell them to memorize formulas that they will never need - instead of helping them understand the deep principles behind those formulas, instead of teaching them to seek answers and not just memorize them.

After we’ve crammed their heads full of data for sixteen to twenty years, we tell them they’re all set for the rest of their lives and send them out into the workforce. This may have worked well when the average life expectancy was forty but it’s a recipe for disaster now that it’s eighty and creeping towards one hundred. What we learned half a century ago in school, assuming we even remember much of it, is stale by definition and no longer relevant to today’s - let alone tomorrow’s - needs. This is a problem we have to address if we're ever going to solve some of our biggest societal problems today. The longer we ignore it, the more we will create a generation who cannot compete effectively in the information age, will feel marginalized, and - in the right countries, with the right influences - will become radicalized. We need a model for continuous lifetime education, one that teaches children to think and learn for themselves in the long run - for the joy of learning, not because they need to make money next year.

But, back to Eden. She has long known how to Google things for herself, order apps on her iPad, watch videos on YouTube, play Words with Friends against the computer, and much more. Compare this to the intellectual universe available to a six year old a century ago - or even thirty years ago. There is no comparison. It is amazing how quickly our brains have stepped up to handle massively larger amount of information coming at us 24x7. We are only now starting to realize the extraordinary cognitive and pattern matching abilities of the human brain. But, still, we choose to take these amazing supercomputers while at the peak of their learning abilities and lock them into rooms for hours a day, teaching them... to hate learning! Nothing short of a revolution in how we think about education will ever fix that.

Having just returned from Disneyland where she stood, starry-eyed, taking pictures with her favorite princesses, I felt a bit subversive as I gave Eden this t-shirt as her birthday gift.



Thursday, May 18, 2017

We Didn't Know What We Didn't Know: WannaCry and the Case for SaaS

“There are known knowns. These are things we know that we know.
There are known unknowns. That is to say, there are things that we know we don't know.
But there are also unknown unknowns. There are things we don't know we don't know.”
Donald Rumsfeld. Former US Secretary of Defense.

“Hedley Lamarr: Unfortunately there is one thing standing between me and that property: The rightful owners.”
Harvey Korman. Blazing Saddles.

“Plan to be spontaneous tomorrow.”
Steven Wright.

I watched in horror last week, as did many of you I suspect, as the WannaCry ransomware crippled thousands of systems around the world and wreaked havoc in almost every country on the planet. I have, you might say, slightly more than a casual interest in the matter. For several years, I managed the team responsible for the SMB protocol, the vector used for the attack, and I was also the head of security for the company for a few years.

I didn't personally write a single of line of code in that protocol. That task was delegated to much smarter people; but I did manage the teams responsible for building, testing, maintaining, advancing, and securing it for several years. I was shocked, like everyone else, to see it at the center of an international meltdown of unprecedented proportions. Almost two hundred countries impacted? Over two hundred thousand businesses stymied? Hospitals? Emergency rooms? WTF?!?

For the uninitiated, here's a brief summary of the situation. If I understand correctly, the ransomware takes advantage of a previously undiscovered bug in the SMB file sharing protocol to take over a computer, encrypts all the files, and puts up a message telling the owner to pay up or lose their data. The NSA had known about the bug for years but didn't disclose its existence so it could be used as an espionage weapon. It was only discovered “in the wild” a few months ago when a group called the Shadow Brokers leaked a ton of NSA documents through WikiLeaks. Microsoft fixed the bug back in March but it's been present in all versions of Windows for years and many companies were caught off-guard as they didn’t realize the potential impact and didn’t deploy the patch on their systems.

This is a protocol, mind you, that was designed back in the eighties when computer networks were still few and far in between, was first shipped in 1990, was standardized as part of the CIFS protocol later in the nineties, and has been used for the past twenty five years for file sharing in every Windows and Windows-compatible product in the world.

Now you take one of these servers that, benignly, implements this file sharing protocol so you can… guess what… share files across a supposedly secure local area network, you add a pinch of magic dust and send it a really screwy malformed request, one that no sane human being would ever send in a reasonably written piece of software. This malformed request, in turn, triggers a bug in the implementation of the SMB protocol that allows the caller to gain supervisor access to the system. Game over. You can encrypt all my data behind my back and ask for ransom to release it.

Microsoft’s Brad Smith immediately blogged about the need for all parties to share this kind of vulnerability information in order to secure software. It is inconceivable to me, knowing what I know about the teams and process at Microsoft, that they would not have fixed this bug had they known about it. I am not here to apologize for Microsoft or the Windows team or the SMB protocol or the history of computer science. I’m here only to say simply that more such bugs will be found in the future, for the simple reason that “we didn’t know what we didn’t know back then” and it’s crazy to continue to depend on such software in today’s world where billions of people are connected to the internet, where nefarious actors abound, and where automated tools can be used to sniff out vulnerabilities.

We spent years designing this software. We spent years testing it. We spent years standardizing it in cross-industry committees and sharing it with partners. We spent years building a community around a protocol that is supported by millions of servers around the world. Our goals at the time were primarily interoperability, usability, and compatibility. We even spent thousands of man years fuzz testing the APIs to make sure attackers couldn't trigger vulnerabilities in the code. We used specialized tools that generated all kinds of random patterns in the arguments and we worked hard with the community of white hat security experts around the world to, responsibly, document and fix security related bugs in all our software.

But guess what. No one tried this particular random pattern of bits - except the NSA. And they chose to keep it to themselves because they felt they could use it to spy on people. That's the story as I've seen reported. Feel free to correct me if you have other data.

Note that “automated updates” (a la Windows Update) are not a solution. Unlike consumers, most organizations around the world spend months retesting Microsoft patches after they are released in order to make sure they don’t break compatibility with business critical applications, then they spend several more months rolling them out through their complicated networks of thousands of servers. The very same corporations and entities who are the slowest to adopt released security patches are the ones most in need of it, the ones that are highly regulated, fairly antiquated in their processes, and entirely unprepared to deal with a global security event of these proportions.

To me, this is the last nail in the coffin of onprem shrink wrapped software and the reason more and more services will move to a SaaS delivery model. I’ve blogged multiple times in the past about the public cloud several times (on private vs. public clouds, on the death of onprem infrastructure and its rebirth in the cloud, and on the architectural advantages of the public cloud). I hope WannaCry will serve as a wakeup call for all those continuing to depend on onprem shrink wrapped software.

Much will be written about this event and how it could have been avoided or more quickly remedied. But the real answer is much simpler than all that, so I'll spell it out. We didn't know what we didn't know back then. You are likely to continue to find more bugs - not just in SMB, but also in the millions of lines of code written in all the operating system software written over the past few decades that is running our businesses today. And the juiciest bugs will be hoarded by hackers and used to wreck even more havoc on our systems. The real problem is that this is a broken model of service delivery as it relies on local system administrators or worse, government bureaucrats, to decide when to install a patch. We’ve just seen an example of what that means in real life.

So the hackers will keep finding the bugs, knowing that inertia is in their favor. And they will hide it from others - so they can weaponize it, so they can monetize it, so they can benefit from it. Think about that. It's human nature. And we are all in denial of it. The motive - industrial, government, or criminal espionage - is almost secondary in nature.

The days are gone when it made sense to have so much device specific code running onprem. The pipes are so much fatter and faster these days that the same services can be offered much more securely from the cloud. The more code you have on your system, the more “attack surface”. The more compability you offer with legacy systems, the more successful you are as a platform with the onprem software delivery model, the longer the tail of companies that will be at risk of exposure for years to come. As an industry, we figured all this out a while ago and moved to the cloud as a much more robust and supportable service delivery model, but the rest of the world hasn't caught up with that model yet. They're still running 1990’s era software. Legacy is a bitch.

We can sit here and blame Microsoft but that would be a mistake. It's true that every one of the thousands of eyeballs that looked at that particular piece of code didn't notice that it would misbehave in a peculiar way when handed parameters that it was never designed to handle. Some smart kid somewhere figured it and it became weaponized. Trust me, there are many other such pieces of code out there. You and I and the rest of the world will pay the price for the next two decades, guaranteed. That's how long it takes to replace these systems in regulated industries. Did I mention that this particular version of the protocol was officially deprecated by Microsoft four years ago exactly because it was known to have fundamental security flaws in the design? Not that it matters. As became obvious last week, hundreds of thousands of businesses were still depending on it to run their applications.

The cloud model of service delivery, where the vast majority of the code runs in the cloud and is always up to date and the most recent version, conceptually bypasses all of these operational problems. If the code is running on our servers in the cloud, instead of on your servers onprem, it's so much easier to patch problems quickly before they become a liability. And trust us; we know how to better manage and patch and upgrade the servers running the code. Better than you, Mr. Hospital in the U.K., anyway.

Fundamentally more coherent and elegant architectural solutions have evolved over the past two decades that cleanly address most, if not all, of the security concerns we deal with every day in an enterprise context. Yet we continue to rely on twenty year old technology and complain vociferously as it fails to stand up when measured against our latest requirements and innovations. Continuing to run ancient software in today's hyper-connected world is akin to riding a horse and buggy down the freeway, complaining that it can't keep up with the neighbor’s latest Google controlled autonomous vehicle, and blaming the poor horse when its knees buckle under the pressure.

If you think your particular application isn't offered over the web a service, I urge you to do another google search. Meanwhile, depending on software designed thirty years ago, implemented twenty years ago, and deprecated ten years ago to run your business and trusting government bureaucrats to know when and how to maintain those systems is a recipe for disaster. It is naive and it is irresponsible in the world we live in.

WannaCry is just the first of many. There will be more and they will be worse. I'm sure of it.