021. Expanding Breadth versus Coherency: The EMS Project

Back to 020. Innovation versus Shipping: The Cairo Project

Through Microsoft Office, even the first versions, Microsoft sold a primitive form of email that worked for small groups of people in the same physical offices. Delivering enterprise email that worked for a company the size of Microsoft, and many times larger (though it would be years before companies would use email the way Microsoft did) was a massive undertaking. The product would become known as Microsoft Exchange and formed the cornerstone of the entire Microsoft enterprise strategy. If you’re looking for an analogy, Exchange was to Windows Server and enterprise computing as Excel was to Windows and the PC desktop. At least I think so. This is a look at the early days from the perspective of BillG strategy and management.

Nearly all of my job as TA involved email—long, detailed, memos written as email. BillG routinely emailed weekend or late-night missives prompting response chains that would go on for hours. Not missing a “thread” was part of the culture.

Microsoft did not use its own product for email. Well, it sort of did. Microsoft’s email ran on Xenix, a precursor to the Linux operating system, and typically a mail PC was issued that was a dumb terminal connected directly to Xenix computers. Mail was simple plain text with no formatting. Using attachments could be awkward. To alleviate the annoyance of using command lines for email, product team developers wrote mail programs to use in MS-DOS and later Windows that copied mail from Xenix and made it easy to use mail in character mode (WzMail) or GUI mode (WinMail). Given how much time we spent in email, there was no shortage of efforts to build mail programs as side projects. I was a hold out and continued to use my Xenix terminal for as long as they were supported. The big disadvantage to these tools was that all your mail was copied down from the Xenix server to a PC—if your PC hard drive crashed you also lost your mail. Clever people figured out all sorts of ways to avoid this failure point, only pointing out just how important email was and how much time and effort typical employees put into just keeping it working. Windows hanging in the middle of drafting a long message or reply was the sort of thing that ruined your day and happened all too frequently. Everyone had email horror stories.

The rest of the corporate world was light years behind Microsoft and almost never used email outside of the technology teams or other companies in the software and technology industries. IBM was the leader in corporate email, using an arcane mainframe system, Professional Office Systems (PROFS), made famous by Oliver North in the Iran-Contra hearings. Since nobody in the PC industry had mainframes, email was a collection of ad hoc tools and systems that worked well only for relatively small companies, like the one mail product Microsoft had called Microsoft Mail. Microsoft Mail competed with a Lotus-acquired product called cc:Mail, which dominated the email category to the degree it existed. Microsoft Mail was based first on a licensed Macintosh product and then subsequent versions were based on technology acquired from a Canadian company called Network Courier held by Consumers Software.

There were many other products. I installed and used several of them during my summer internships when simply connecting the computers together was difficult. These early products were built on basic infrastructure of sharing files over dedicated networks—a mailbox was simply a file on another computer and sending mail was for all practical purposes reading data from one file and copying it to another. I’m simplifying for effect.

Because email relied on connecting one set of mail products to another, there was a time when the big telephone companies believed they would provide email service much the same way that they provided voice connectivity, especially since mail between companies involved phone lines. This led to attempts to standardize email using Byzantine standards that only phone companies could love.

While all this was going on, one of the first substantial Windows programs, designed by Ray Ozzie at a company called Iris, owned by Lotus but kept independent in a nod to the difficulty of developing innovative products within an established leader, increasingly gained momentum. The product, Lotus Notes, was a platform for building custom applications that could run on several different operating systems as well as being a rich graphical application. It was also an email program. Notes created a new category called workgroup computing. Even though Notes competed with Microsoft, Ray Ozzie and the team received a Windows Pioneer Award in 1994 at a ceremony honoring the most significant third-party contributions to the creation of Windows.

Notes had an architectural appeal that cut right to the core of everything Microsoft valued. It ran on PC hardware. It was designed for Windows. It worked whether connected to the network or temporarily offline. Most of all, it was a platform with a database and a programming language, a flagship product for the PC era. The capabilities of Notes touched on every group in the company—effectively competing required Microsoft to marshal teams, and, more importantly, to change plans, for most every division. There’s a great lesson in big companies competing—the best way to compete is to build a product that spans business units or divisions as Lotus had managed to do. Ideally the core of the product would land squarely between two business units, creating a standoff of sorts between those groups as they decided how to compete.

Microsoft needed to compete. Microsoft, however, had nothing. That’s an overstatement in sense. We were selling Microsoft Mail, but we weren’t even using it internally. To complicate our internal mail architecture, we actually used the companion product for scheduling meetings called Schedule+ even though we did not use the mail product. In Microspeak, when scheduling a meeting people would often say “send me an S plus” or “Schedule plus me” – lingo that would stick around years after that product was no longer used.

A new team under MikeMap was created to build email for corporations with the acquisition of Network Courier in 1991. The importance of email was certain, though it would be five or even ten years until corporations saw email as a workplace standard. More important than email itself was building email to take advantage of the new Windows Server platform. Windows Server was still very new and part of the dual mission of NT in addition to Workstation. Version 1.0 (officially version 3.1) did not ship until mid-1993, and it would be 1996 before the first broadly used release was completed, NT 4.0. BillG’s strategy for a platform was to build the platform strength by also building apps and create strength in apps by making a singular bet on a new platform—deliberately creating the virtuous cycle. This was especially important in the early days of the platform when it was difficult to get existing companies to bet on something that might not be commercial for years. That was exactly the goal of the new EMS team—Enterprise Mail System (or sometimes Messaging, with an early code name Mercury, as in messaging).

No one had ever built email on Windows Server using the new architecture approach called client-server that Windows was designed to support. Significant internet standards for email did not yet exist reliably (the familiar @ sign was decades old, but routing email around, use of anything more than plain text in messages, attachments, and more were still flaky). Windows Server did not yet exist. Tom Evslin (TomEv) was brought to Microsoft via an acquisition (Solutions, Inc. which made Mail software used in Mac Office in 1990) ultimately to lead EMS engineering. He brought with him detailed knowledge and experience in email and experience with the challenges of scaling email to hundreds or thousands of people. Microsoft, with its history of personal computing, had yet to internalize how different computing and data storage would be when running on servers or meeting the needs of thousands of simultaneous users using PC servers, and how different it would be to sell and deploy mission critical data center software.

There was a lot for me to dive into as TA. I had to figure out how to frame the challenges the team faced so that BillG was effective in driving his strategy. Given how much the team needed to do—and I was empathetic—it would have been easier and more straightforward to minimize the reliance or dependency on other parts of Microsoft. As Bill reiterated on many occasions, that was not the sort of strategic approach Microsoft benefited from—in fact, Lotus Notes was already succeeding by not relying on Microsoft all that much, even emphasizing its cross-platform capabilities.

Just as word processing and spreadsheets were the anchors of the Windows ecosystem flywheel, Bill decided that email and database were the anchors of the server flywheel under development. The database efforts were spearheaded by David Vaskevitch (DavidV), a longtime enterprise advocate within Microsoft and database expert. The database under development was Microsoft’s variant of an industry standard known as SQL, a category dominated by giants like IBM and Oracle (Oracle in 1993 had $1.5 billion in revenue, growing 30 percent or more per year). SQL was designed to handle highly structured information like inventory or financial transactions, which was different than the relatively arbitrary information in email. In other words, SQL was not designed to handle email. SQL had become so dominant that there was not a data storage problem that advocates of its core technology were not working hard to solve using SQL.

The third triad of the server ecosystem was the directory—or, basically, the list of all the users and computers that were part of the network. The directory, from the earliest days of NT, was an integral part of the system known as simply as NT Directory (the name Active Directory would come later). The key competitor to Windows NT was Novell and they sold a separate product as a directory, so Microsoft’s strategy would be to integrate it with its new server product as was typically done in technology. The NT directory was, however, not designed to handle all the capabilities required for email. In practical terms, Microsoft was in the earliest days of deploying NT Directory, which was proving to be enormously complex and extremely difficult to operate.

In the ideal, EMS would be architected to use SQL to store email and the Windows NT directory to keep track of all the user email addresses. To compete with Lotus Notes, EMS should use Visual Basic to create the user interface for customer-developed applications. Discussion along these lines happened almost continuously. Every time the EMS team mentioned some product feature or engineering challenge, it seemed like they received feedback over how much easier it would be if they would just use SQL and NT Directory. When those were not the answer, then building on the enhanced capabilities of Cairo would serve as a placeholder for solving problems in the best architectural manner. The easiest way to characterize this is that every time EMS showed something that looked like a hierarchy of folders with items (mail messages), they would get feedback about how they should be representing that in SQL the way business applications routinely do for sales by region, country, district, etc. And failing that, they would then be told to “go talk to Cairo” who was building a way to view folders and files using SQL, so they were solving the problem. Predictably, every time the EMS team came back saying it would be better to implement something on their own, it was received as a justification for failing to think strategically.  It was messy and uncomfortable, and slowing progress to compete with Notes (which did not use SQL or the file system).

At some level of abstraction such a technology approach made sense. Building the product this way, however, amounted to crazy talk. That is what I heard as I practiced reconnaissance and shuttle diplomacy across teams. I found myself between two entirely different cross-team debates. Cairo debates seemed somewhat academic compared to EMS debates, which were rooted in getting a more practical product to work at scale.

For storage, EMS had no intention of using a commercial SQL database. Attempting to represent mail in such a structured manner had been tried before (Oracle was famously trying and not delivering). The allure was great—the industry rallied around SQL and was investing heavily in tools and infrastructure to standardize on SQL as data moved off mainframes. At the same time, the SQL team was lobbying hard for reusing all of this effort knowing that building for scale and reliability was a full-time job and they were certain that EMS would end up duplicating their work, while also trying to build email. In a sense they were correct in that any storage technology EMS used would also require them to build capabilities to scale to huge storage requirements across many computers, tools to manage all those computers and disks, as well as backup and restore should anything go wrong.

This dynamic happened so many times. A team building something new appeared determined to rebuild something that already existed because the requirements for what they needed could not be met by the existing product. At the same time, the existing product team was busy trying to win in their market and didn’t have time to take on the special requests from the new product team that had never shipped anything. As a result, Microsoft dev teams got very good at articulating reasons why some piece of code was not good for what they needed and at the same time product teams got very good at explaining why their schedule did not permit them to add special features. We never seemed to master having a rational discussion. Over many talks, BillG would admit to me that part of his reason for staking such extremes was his hope of making some progress. He knew realistically teams would end up somewhere less than he hoped, but if he started there they would end up with even less. Hearing this kind of bugged me because it sounded like everyone was playing a game of some sort. It did not sound like JeffH “promise and deliver” that I had been schooled in.

I wrote endless late-night emails summarizing the pros and cons of different data storage approaches for BillG, and there were many meetings and discussions that followed. This topic came up at almost every meeting where developing applications for NT was discussed. I was in a database group in graduate school and sympathetic to the realities outlined by SQL. But I was also part of a team looking beyond the 1970s SQL model to a new object-oriented data model (there’s that buzzword again) and was sympathetic to the new styles of usage. EMS opted for being in control of its own destiny, a decision that was not difficult for them to defend on technical grounds. EMS spent the better part of the next decade scaling the product, officially named Exchange, and working on scale and reliability, exactly as the SQL team predicted. Likewise, every limitation that Exchange faced when it came to building applications on top of the platform were exactly like the SQL team predicted. To the credit of the Exchange team, SQL did not end up being the right solution for email. In fact, huge email systems today were built using something called NoSQL, decidedly not SQL (my intent is not to start a database war as I am sure some will argue semantics over what constitutes SQL versus NoSQL).

The debate over the directory was quite different. In this case BillG had two groups, each on a path to ship and each deeply strategic. The sales proposition to customers could not be that there was one place to store all the employee names for when they signed on to the network to print or share files and an entirely different place for when employees used email. The entire purpose of a directory was to have a single source of truth for security and identity within an organization.

Windows NT was on a mission to ship, with the RTM months away and the beta already out there. The product was still quite early for customers and the directory worked in specific ways but was not yet all it had promised to be, as was typical with a 1.0 product of the era. Microsoft IT was in the process of architecting and deploying the directory for the company and wrote a huge white paper in collaboration with the NT team on how that would work. I showed this to BillG and we both marveled at the complexity—not in the positive way, but more like yikes. It was page after page of graphs showing servers in different countries required with backups, redundancy, and overnight processes to keep everything in sync so there was one master copy of the directory. It was a massive undertaking.

That proved to be a defining moment because deploying a directory was hugely complex and there was no way EMS could do it twice. In one of the rare times an architectural choice was pushed to a team, using the directory from NT became a requirement for EMS. Many others supported this, including the Server leadership. It was to them as natural as pushing Excel to use Windows—the directory was that core to NT Server—while sharing files and printers was the baseline scenario, it was the directory that brought deep enterprise value to customers. For the better part of the following year or more, EMS would not speak well of using the NT Directory, and conversely the NT team felt that EMS was trying to use the Directory in ways it was not designed to be used. This sounded to me a lot like getting Excel to work on Windows, and it played out exactly that way. Had EMS not used NT Directory, it is likely Directory never would have achieved critical mass as the defining app for the client-server era (and remained the cloud anchor for today’s Office 365). And conversely, had the NT team not met the needs of EMS, then the NT Directory would have likely been sidelined by the rising importance of the email directory in EMS. Forcing this issue, while it might be an exception, only proved the strength of a strategic bet when it is made and executed. Still, it was painful.

Developing applications for EMS turned out to be an endless series of missteps in strategy. Notes had risen to popularity as a platform for building applications. Customers buying Notes were building incredibly cool applications for tracking all sorts of line of business processes and corporate knowledge management, which also happened to be key buzzwords in business at the time. Everywhere our sales teams engaged, they saw customers piloting applications they had written internally or with help from the giant consulting firms. These applications were often the highlight of Chief Information Officers and used to show how cutting edge a large enterprise was. While they ran on Windows, they did not use any strategic (and new) Microsoft software except maybe Word and Excel files. They did not use SQL Server, NT Server, and Notes was a substitute for EMS. Even worse, Notes had its own programming tools so Visual Basic was not even part of these applications. It was a nightmare!

My contribution to the mess was a long memo outlining how to “package up” EMS, Visual Basic, and Microsoft Access (the database in Office) to compete with Notes. It was the kind of memo I wrote knowing that it was exceedingly tactical and would never win in a competitive review. Importantly, none of the teams thought doing a project like I described was their job. Everyone was busy with their own category. I worked super hard, creating screenshots, architecture diagrams, and more. Bill really wanted a team to run with this. No one was interested. I felt this left us totally exposed to compete with Notes, and Bill agreed.

What neither of us saw was that the market for custom applications built on top of email was not nearly as interesting to customers as simply getting email to all employees. In a sense, Notes had made the same miscalculation. To Notes, email was one kind of application to be built with Notes (that happened to be built by the Notes team). With EMS, email was the application, and the only one (until the Outlook email program came along to add integrated scheduling). The grander vision for the category fell victim to the near-term needs of customers. Microsoft’s sales teams would use the techniques I outlined in my memo over the coming years to sell against Notes, but few customers built and deployed applications that way. Email was the killer application for Windows NT Server and the Directory, at least until the internet came along. Importantly, email was the killer application that every company needed to roll out to every employee. It didn’t hurt that NT Directory was itself a killer application just for rolling out shared folders and printers. The virtuous cycle between these products was unlike anything, perhaps even greater than Excel and Windows.

Perhaps the biggest (and in many ways most painful) experience in building out EMS was Microsoft’s own transition to using the product internally. Email was the most mission critical software at Microsoft, probably even more important than any billing and accounting software. Every single employee used email. Part of building EMS was figuring out how to get it deployed to all of Microsoft, and by SteveB goal to do so before the product shipped to customers. In other words, Microsoft needed to be self-hosted on EMS before paying customers. Makes sense.

The EMS team (and of course the NT team because everything ran on NT Server) were pioneers in the idea of dogfooding products and created a whole new infrastructure where the team itself ran on the very latest builds, then a large set of partners and friends ran on slightly older but more stable servers, and then ultimately Microsoft’s IT department supported the rest of the company through a migration from the old Xenix system to EMS. The team was incredible at keeping servers running and keeping mail delivered. While there was downtime, when that happened pagers went off and people ran to the office to fix bugs. The team put these same skills to work with a small set of first external customers which received the highest level in support so they too could be part of the initial launch of the product, validating how it worked reliably and seamlessly for hundreds of thousands of mailboxes. When people ask me the true start of Microsoft’s enterprise approach to building and selling products, this dogfood process was absolutely ground zero and key to Microsoft’s long-term success.  The process was heavy-handed and brutal on the partner teams (as I would learn building Outlook) but it is not clear there was another way. Running Exchange email turned out to be one of the most complex asks of our enterprise customers and something that was by all accounts nearly impossible to get right (secure, reliable, etc.) unless you were the dogfood team. At the same time, there were no alternatives. The reality of software hardly working at all moved from the desktop to the data center. EMS dogfood before shipping the first time went on for about two years until the product finally shipped in 1996, long after I left my role as Technical Assistant.

Ultimately, BillG was entirely correct about the architecture required for email. Microsoft Exchange continued to struggle as a platform beyond email for decades. The lack of structured storage like SQL and programmable forms like those in Visual Basic made it difficult or even impossible to build additional applications. Microsoft would produce tools and languages for Exchange for the next 15 years, none of which gained critical mass or long-term adoption. On the other hand, Exchange dominated email without question—it arrived with global scale email at the right time and even transitioned that email capability to the cloud, forming the cloud portion of today’s Microsoft 365.

Where most executives in charge and accountable for results would have kept pushing knowing they were right (and he was), what Bill routinely did was make his point of view clear while giving room for teams to execute and deliver, when it was clearly the right thing to do. Cairo had a backup plan (actually, two) so it was the right call to keep pushing on innovation. EMS did not have a backup plan. If EMS did not become a product, then Lotus Notes had too much opportunity. The bet on NT Directory proved to be critical to the entire product line and server initiative. Bill had the foresight to believe and operate as though email would become the anchor of enterprise computing for knowledge work, and it certainly did.

On to 022. New Ideas and IQ for the Information Superhighway

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit hardcoresoftware.learningbyshipping.com