Nortel: Perspective Drives Feelings

A couple weeks ago, Brad Reese wrote a blog entry over at Network World that talked about the Nortel bankruptcy. In response, he got a bunch of comments from ex-Nortel employees discussing the reasons for Nortel's failure. Brad even quoted a couple of paragraphs from my previous blog posting on the subject. Last evening, Brad posted a roundup of some of the comments to his previous posting (including my quotes again).

What strikes me about the comments posted to Brad's original blog entry, as well as the comments I have received on my posting are that a lot of how you feel about this whole thing depends on your history with the company. There is a lot of "he said, she said" going on between old-line Nortel people and folks that came to Nortel through the Bay Networks acquisition.

If you were an old-line Nortel guy, then it's clear that the Bay Networks acquisition was the start of the screw-up. If you were a Bay Networks guy, then you probably feel that Nortel was clueless and didn't learn anything from the acquisition. Almost by definition, we're conditioned to think that the "home team" is better than the "visiting team," even when we're all supposed to be on the same team.

Fortunately for me, I was neither a career Nortel or Bay employee. I had only been at Bay for year before Nortel bought Bay. Because I was a relative newcomer to both companies, I think I have a better perspective on both the strengths and weaknesses than many other people. In this case, I think my perspective provides a distance necessary to make some objective judgements about the situation. That said, I'm just one person with one view. Everybody is entitled to their own.

So, did Bay screw up Nortel, or did Nortel screw up Bay? Short answer: Yes. The longer answer is more complicated.

When I got acquired into Bay, the company was pretty screwed up. Not many people know this, but when Bay was created from the merger of Synoptics and Wellfleet in the mid 1990s, the combined entity was actually larger than Cisco in terms of revenue. The company had a great set of Ethernet hubs, was leading in terms of the new technology of Ethernet switching, and a long-established line of excellent routers. In short, Bay took two great companies and jammed them together in what should have been an awesome company. Why did it work out that Cisco is thriving and Bay got bought out and Nortel then filed Chapter 11?

The problem, again from my perspective arriving at the company years later, was that the companies merged rather than one acquiring another. In talking with people who lived through it on both sides, this seems to be the root of all the other issues. The leadership of the companies went out of their way to try to make people feel that neither company had "conquered" the other. The problem with that was that it tied the decision-making process in knots. The leadership set up such a system of power sharing that nobody had any power to get anything done without another group being able to torpedo things. So, everybody started torpedoing everything in retaliation to being torpedoed themselves. At a macro level, almost all new product development ground to a halt because nobody could get enough critical mass to move the ball forward. Projects would start and then get repeatedly canceled.

This problem particularly devastated the Wellfleet product lines. Synoptics was very much bought into the idea that Asynchronous Transfer Mode (ATM) was THE NEXT BIG THING. ATM, it was said, would obsolete Ethernet and obsolete routing. While Synoptics wasn't the only company to believe this, they made big bets in this direction. Anybody advancing a project plan that focused on advancing the company's Ethernet switching (in which, again, Synoptics was leading) or routing product lines was often doomed to failure as budget was redirected to work on ATM-related projects. I talked with multiple Wellfleet engineers who had worked on five or six projects since being part of Bay, all of which had fallen victim to the budget redirection axe and been canceled in favor of "ATM will make that obsolete" thinking. But it wasn't just Wellfleet people. I met many Synoptics engineers trying to push forward Ethernet switching projects that suffered the same thing.

So, when I got to Bay, you had a bunch of fairly talented, but bitter people, all mad at each other, and almost no new product development coming to fruition. The sales force was desperate for something, anything new that it could sell.

During that same period, Cisco took a more pragmatic approach. It placed bets on each of these technologies, building what it could and signing OEM relationships to fill in the gaps. The Cisco marketing organization provided the "marketecture" that made everything look seamless and integrated, at least in the Powerpoint slide decks that were presented to customers in the Executive Briefing Centers ("...and of course all these various products run IOS..."). Then, Cisco watched the market and doubled-down on the market segments that were really growing. While Bay dallied with the notion that ATM was the one network to rule them all ("...and in the darkness bind them..."), Cisco quickly figured out that ATM wasn't selling and Ethernet switching and IP routing were. Appropriate projects were created to drive those product lines forward. Investments were made and profits ensued.

Now, Nortel was equally screwed up, but in a different way. Nortel was a great example of a company that did really well in the 1980s but didn't react to the way the world was moving in the 1990s. The company built great phone systems, both for private businesses (Meridian) as well as telcos (DMS). And when I say that they built them, they really did. They developed their own CPUs, wrote their own operating systems, manufactured all the boards and sheet metal and whatnot. This was old-line, vertically integrated manufacturing at its best. And it carried all the costs associated with such.

Further, Nortel really understood the voice market of the 1980s when things were still dominated by the larger telcos, many still nationalized around the world. These customers moved slowly. Products were installed and then expected to run for a decade or more, with nothing but backward-compatible upgrades provided during that time. Margins were very high. Everything was wonderfully over-engineered to deliver exceptional quality. The sales process was optimized to call on either large telcos (Bell Canada or the US Baby Bells) or work through oodles of small resellers peddling PBX systems for SMBs. This latter channel was highly optimized for installation and maintenance by unskilled labor. The rate of change was slow. In short, you built a highly engineered product, expected to be indestructable for a decade or more, and you leveraged your relationships to incrementally sell more upgrades into accounts that you won. You charged an arm and a leg for what you delivered, and it was very profitable. In that world, Nortel was very good.

Unfortunately, in the mid-1990s, the world changed. Nortel was woefully unprepared for the rise of datacom (not telecom), the Internet, and run-and-gun IT. In this environment, Nortel's slow-moving, vertically integrated development model fell apart. Just as the "Bell heads" lost out to the "Net heads" in the larger networking game, Nortel found itself on the diminishing side of the same war at the supplier level. In short, the world was going packet, and the packet world consumed and threw away more technology at a more rapid pace than Nortel had ever seen before.

In the mid-1990's, Nortel formed some relationships with Cabletron Systems in order to address packet-oriented product requirements. But this was not enough. John Roth, Nortel's CEO, knew that he needed a massive change that the corporate DNA level. Perhaps it was possible for a "Bell head" to morph into a "Net head," thought Roth, but it's obviously going to take longer than we have time for. Roth's decision to buy Bay was predicated on belief that Bay would infuse Nortel with a lot of good "Net head" DNA quickly. In this, I think Roth made a reasonable decision. Bay was still the #2 company in data, and could be afforded by Nortel at the time. You play the cards you're dealt, and in this respect Roth made a good move.

The problems came in trying to put it all together. Without an overarching vision of exactly how this merger of telecom and datacom companies was supposed to function, the whole thing got executed in a patchwork of small pieces. Roth knew he needed to shift the company, but couldn't articulate in detail to exactly where he was shifting. So everybody made their own best guess and the merged Nortel was the result.

Increasingly, the corporate marketing got more and more bland as the company tried to avoid taking a stand on anything to avoid angering a constituency. You couldn't tell new Meridian PBX customers that were expecting decade-long support that VoIP was the future and they had just bought an Edsel. You couldn't tell your channel partners that they had to understand both voice and data or they would be kicked out of the program. Etc. And so rather than anybody actually changing anything, Nortel operated like a slightly less disfunctional version of Bay, with everybody doing their own thing, with less sniping at each other. The cash generated by the optical business allowed people to avoid making the hard decisions about where the company was headed and allowed senior management to go to customers with the idea that Nortel made anything and everything, all equally well.

So, as a result, you have post-Chapter 11 comments like these from multiple points of view:

• Nortel spin: "Bay was checkmated by Cisco, and for some strange and sad reason Nortel decided to follow Bay's losing managers and strategy, turning a once strong and proud company into an also-ran in the enterprise market." -- Really? Nortel followed Bay's strategy? All I knew was that Nortel kept trying to kill my product line. A product line that has since delivered billions in revenue to Nortel.
• Nortel spin: "With the influx of Enterprise talent that were focused on short-term relationships, eventually everything went toward that strategy and away from strong, long-term relationships. As part of short term focused strategy, they outsourced their manufacturing, development and systematically "outsourced" of all of their valuable assets." So building everything from silicon on up is a good idea in an era when contract manufacturing works so well? A valuable asset?
• Bay spin: "The Bay Networks aquisition was one of the best buys out of Roth's outrageous spending spree. The people were dynamic and products leading edge, providing a real alternative to Cisco in the enterprise space...which many customers loved." Bay's people were not dynamic and products leading edge. Certainly, there were good people and some good products, but many good people left Bay way before Nortel bought them because Bay was such a disaster, and many product lines had stagnated years earlier. Bay's people were tired and cranky, having put up with a bunch of internal crap for years, and the products were tired even if they had been market-leading when they were introduced. Things were turning around, but it wasn't all sunlight and roses.
• Bay spin: "It was always perceived that Management was tinkering in product development as well. A product development life cycle would start, and then half way through be killed, and then re-spun. There was about 3 false starts on a IPT platform before one was actually released. Because of this, Nortel trailed Cisco and Avaya, and was never able to catch up." Right, not like those Bay engineers that had six projects canceled in as many years at Bay and literally never saw their hard work ship to customers in any form. Maybe Nortel did adopt some of Bay's failing strategies.

The point here isn't to suggest that anybody's recollections are wrong. This is simply a great example of the parable of the blind men with the elephant. We all saw a slice of the whole and form our judgements about blame based on our past history and our narrow vantage point. I assure you that there is more than enough blame to go around.

In any case, thanks Brad for pulling together multiple viewpoints. While we all can disagree about the cause, the fact that Nortel is now in Chapter 11 is fairly indisputable.

The Decline and Fall of Nortel Networks

The industry is still digesting the half-surprising bankruptcy of Nortel last week. Last week, I put together my own posting working through the history from what I could remember. Today, Craig Matsumoto at Light Reading wrote a great article that chronicles the whole mess using the various Light Reading stories about Nortel from 2000 through to the present. It's very well done and provides a lot of the context and actual news as it was written that my original blog post just touched on. If you're interested in how Nortel got here, have a read.

Your Vyatta system just got cheaper

So Cnet is reporting that Intel just cut prices on some CPUs by as much as 48 percent. That means your next Vyatta system just got cheaper, or more powerful for the same cost. You pick. Welcome to the world of Moore's Law Economics™. With Vyatta, you're now riding the ecosystem cost curve, not the proprietary cost curve.

Cisco could not be reached for comment.

Creating a Session Border Controller with Vyatta

Earlier this month, Michael Picher created a nice document describing how to create a SIP Session Border Controller (SBC) using Vyatta. You can find his write-up here: Install OpenSBC on Vyatta Firewall

Comparing Bubbles

The Economist has a good article today that talks about the current recession and financial industry bubble. Specifically, the article talks about how tech firms might fare over the next year or two and how the current bubble differs markedly from the previous tech bubble.

This time things are not yet that bad—and are unlikely to become so. In spite of the string of bad news, some forecasters still expect global IT spending to grow this year [emphasis mine], at least when you allow for currency fluctuations. According to Forrester Research, a market-research firm, technology purchases will decline by 3% in 2009 when counted in dollars (see chart). But the dollar’s relative strength weighs heavily on the results of American firms by devaluing their foreign revenues. When measured in a basket of local currencies, weighted for each region’s share of the global IT market, Andrew Bartels of Forrester expects an increase of 3%.

There are many reasons why spending is more robust than during the last downturn. For a start, the IT market has become more global. Between 2003 and 2008, developed countries’ share of IT spending fell from 85% to 76%, according to the OECD’s recently published Information Technology Outlook. Demand from China and India is expected to continue to grow despite the gloomy economic outlook.

More importantly, last time around the IT industry was not the victim of an economic crisis, but its cause, says Graham Vickery, author of the OECD report. For years companies had spent far too much on technology, buying more e-commerce software than they could ever hope to use, for example. When the bubble burst they abruptly cut spending. Today IT departments are much less prone to wasting money. In fact, says Mark Raskino of Gartner, another market-research firm, most are quite lean. Further cuts in technology budgets would be difficult, he argues, since they would require many firms to reorganise themselves first. “IT is certainly not sacrosanct, but fairly low on the list of things to cut,” he says.

This is an important point. While the financial industry is reeling from the current crisis, the tech industry is tightening its belt and moving on. Obviously, the current recession will affect everybody, worldwide, but there is much reason to believe that the devastating catastrophe of 2001 and 2002 that claimed many tech companies won't occur the same way again. Will there be casualties? Yes, you bet, but not like in 2001 and 2002 where whole market segments evaporated over night (everybody say "optical networking").

The article goes on to say:

In many ways the previous IT downturn marked the industry’s coming of age. In its wake, the industry was no longer mainly about “hot” new technologies that made maximal use of Moore’s law, which holds that roughly twice as much processing power is available at a given price every 18 months. Firms have since started to opt more for good-enough “cold” wares, which save them money and allow for more flexibility: commodity hardware, open-source software such as the Linux operating system and programs accessed over the internet, or “software as a service” (SaaS).

The crisis will only speed up this shift, not least because many of the cold technologies have themselves become more mature. SaaS and other computing services supplied online, and collectively called “cloud computing”, have become better and more widespread. In November Salesforce.com, the largest SaaS firm, beat analysts’ expectations, reporting sharply growing revenues and profits.

And open-source software has long since moved beyond Linux. All kinds of enterprise software is now available in this form, which in most cases means that firms pay for maintenance services, but that the programs are free. This business model already seems to be benefiting from the downturn. Revenues at Alfresco, which makes software that helps manage web content, for instance, have tripled in the last quarter, according to Mr Asay.

In short, this article lays out a great case for everything we have been saying at Vyatta. How can firms continue to justify overspending with Cisco, giving it 64% gross margins, when other options exist that deliver better performance for less? Increasingly, they aren't trying to justify it; they're simply adopting Vyatta.

Nortel: In Rememberance

Well, as might be imagined after yesterday's semi-shocking news that Nortel was filing Chapter 11 bankruptcy, the news stories are flying today about what this all means to the telecom industry.

Craig Matsumoto at Light Reading gets the "picture is worth 1000 words" award.

Jim Duffy at Network World reports that, unsurprisingly, competitors are already trying to chew what's left of the flesh off Nortel's "I'm not dead yet" corpse.

As a former Nortel employee, I can tell you that this was both surprising and not at the same time. The following is my analysis of things, having been at the company during its heyday but now from the outside looking in.

By way of background, I was with a small startup company, Rapid City, that got purchased by Bay Networks in 1997. Bay was then purchased by Nortel in 1998. Along the way, I was promoted to VP of product management for the enterprise switching group at Nortel. If you're familiar with the 8600 L3 switching platform, that came to become the core of the business we built, though I left just after the product shipped into the market in mid-2000, right before Nortel purchased Alteon Web Systems for $7.8B and the bubble burst.

In the telecom growth spurt of the late 1990s, Nortel was one of the fastest growing companies and stocks in the world. They were very well positioned to ride the optical networking hype wave that accompanied the Internet revolution and the overall building of the current worldwide network. Unlike many companies of that era, there really was a "there there." The company had real products, smart engineers, and in many cases very good technology. What they didn't have was customers with non-hyped business models and overall good business and marketing sense. In short, the bubble caught them, hook, line, and sinker.

Unfortunately, the company got very off-balance as it grew. It was very clear when I was there that anybody who wanted a long term career in Nortel had to be associated with the carrier group, preferably an optical product line. That was where all the momentum was within the company. If you were an employee working on other things (enterprise products in my case), it was clear that you weren't going to get any of the investment, attention, or promotions. In contrast, the optical management teams were masters of the universe.

When the bubble burst in late 2000/early 2001, the company's "core businesses" were smack in the middle of the action (or sudden lack of action, really). With the chair suddenly pulled out from under it, Nortel could not retrench quickly enough. They had allowed many of their other businesses to decay as the company got tunnel vision with all-things-optical, and there was nothing to fall back on. This was not the beginning of the end, but rather the end of the beginning, however.

If the company had had good leadership at that point, it could have gone through the painful process of rebuilding itself and survived. Products and product lines come and go, but good companies survive the dips and swings in the market. Unfortunately, Nortel didn't have the top-level DNA necessary at that point to do what was required and move the company forward. Many of the people who were in senior leadership roles at that point had simply been lucky to be at the right place at the right time during the bubble and found themselves unable to cope when the formerly rising tide stopped lifting all boats and instead started to recede. All big companies suffer this same problem, but Nortel suffered particularly hard because its success in the late 1990s had been so sudden and great.

The real trouble, the beginning of the end, started when accounting problems cropped up in 2002 and the company was not able to quickly move past them. They went through a couple of CEOs and CFOs during that time, some of whom later had criminal charges filed against them. While it would be hard to say that Nortel was fully "cooking the books," somebody at Nortel was clearly "warming them in the microwave." During the same time period when Enron, Tyco, and Worldcom were exploding, it was simply unacceptable to any investor and the company's stock continued to move downward as the scandal dragged on and on.

Just when you figured it would end, that Nortel would restate their earnings, and that they could get back to making good products again, instead the company would issue another press release saying that the final restatements were being delayed and there might be more bad news coming. This continued literally for years and was a constant drag on the company.

The constant bad news turned into a general malaise during this period. Instead of the vibrant company Nortel had been in the late 1990s, it became a dead man walking. The bad news affected customer confidence in the company, which led to even lower sales than they would have had in the post-bubble period, which led to layoffs and corporate debt to manage the finances, all of which in turn led to employee dissatisfaction, and attrition of good people.

So, here we are in 2009 and the chickens finally came home to roost. Just yesterday, Nortel filed for Chapter 11 bankruptcy protection in the US (and the equivalent in Canada). The luster came off Nortel back in 2001 and it's been a slow decline ever since. With the overall worldwide macro economy now faltering, Nortel simply couldn't balance the books any longer. Revenues were declining, the debt load was high, and there was very little that they could sell off that would raise cash, in spite of trying.

To be fair to Nortel, this really isn't the end. Chapter 11 bankruptcy simply protects a company from its creditors while it restructures itself under the watchful eye of the court to try to find a way to dig itself out of a very big hole. Some companies have been through Chapter 11 multiple times and are still operating (Continental Airlines springs to mind). Obviously, Nortel has a fairly traumatic future in front of it, but there is a chance that it will come back from this. If so, my own feeling is that it will be years before it finds it stride again.

If there's a lesson in this sad story for people, it's that good companies get through bad news fast. While every company has ups and downs, the good ones know how to manage through periods of bad news and do their best to get things behind them quickly. Nortel, in contrast, managed itself slowly into mediocrity.

Was everything at Nortel bad? No, certainly not. Nortel had many bright spots over the years, and I met some very talented people that I respect greatly from among the oldline (non-Bay Networks) employees. In short, there was real talent there. If there was a failure at Nortel, it was the leadership who didn't know how to handle the transition.

At this point, let me give some props to John Chambers over at Cisco. While Vyatta and Cisco obviously have big differences of opinion about how the networking market will evolve in the future, I'll say that the Cisco management team, led by Chambers, didn't fall into the same pot-holes as did Nortel following the bubble. Fundamentally, Cisco was better positioned than Nortel when the bubble burst. It was not as dependent on the optical networking product lines (because it was behind, a Nortel employee might say) and it was on the right side of the VoIP transition, with no legacy business to defend and migrate (Meridian). Still, Cisco out-executed Nortel through the post-bubble era. Cisco infiltrated Nortel's customer base, picked off key Nortel employees, and grew revenue even as Nortel declined. When the market gave them lemons, Cisco made lemonade. In my opinion, the credit for this rests with squarely with Chambers and his management team.

One question the networking pundits are starting to ask is, "Who's next?" Alcatel-Lucent has been troubled in recent years and has been laying off employees. Cisco is certainly not invincible and many of the top management team who steered the company through the post-tech-bubble period have left in recent years. Could another mega-networking company become a victim? Quite possibly, says I.

The current global recession will test everybody, I think, and we'll see who is strong enough to make it. Certainly, at Vyatta we think we'll continue to grow. While recessions aren't fun, it sometimes takes a large shock to the system to get people to reevaluate their behavior. In the same way that Cisco was on the right side of the transition to VoIP, Vyatta believes we're on the right side of the coming commoditization of networking.

While the future is never certain, Vyatta won't be filing Chapter 11 ourselves this week.

Nice Vyatta BGP tutorial

There is a nice tutorial on how to setup BGP on Vyatta over at Debuntu.org: How-To: Setting up BGP on Vyatta.

Battle of the Brands

Matt Asay wrote a nice little blog post about Vyatta last week, Vyatta: Beating Cisco with open networks. The only bummer with the post was that it arrived on January 2 and many people probably didn't see it because of the holiday season. Matt's post references a TechTarget article by Pam Derringer and pulls a quote from analyst Tony Iams from Pam's article:

Tony Iams, an analyst with Rye Brook, N.Y.-based Ideas International, said the networking market continues to grow, but Vyatta faces a challenge because Cisco has such a strong brand name, and companies are reluctant to gamble with networks because they are so critical to operations.

"Open source can be disruptive with a cheaper product," Iams said. "But Vyatta has to overcome a lot of skepticism and reassure customers that they aren't sacrificing quality."

My reaction to Tony's statement is, yes, of course. You aren't telling me anything really new. Customers always generally favor tried and true products over new and untested products. Right up until the time that they don't and the new becomes the tried and true product and the old product goes the way of the dinosaur. This is simply the way that new products and ideas come to market and it happens over and over again.

And this quote, almost verbatim, is always trotted out by analysts when they are asked to comment on a new company. It's an old saw in the analyst business, as safe and solid as motherhood and apple pie. You can't go wrong with it, because it's so uncontroversial as to be almost content-free. I don't fault Tony for using it, any other analyst would have said the same thing, but it's simply FUD without any hard evidence.

To prove the point, let's take this same quote, run in through the Wayback Machine, and see what it sounds like in other contexts:

John Doe, an analyst with New York-based, Big Analysis, Inc., said the computer market continues to grow, but Microsoft faces a challenge because IBM has such a strong brand name, and companies are reluctant to gamble with computers and operating systems because they are so critical to operations.

"Windows can be disruptive with a cheaper product," Doe said. "But Microsoft has to overcome a lot of skepticism and reassure customers that they aren't sacrificing quality."

How about this one:

Jane Doe, an analyst with Conneticut-based, Amazing Insight, said the PBX market continues to grow, but Cisco faces a challenge because Lucent has such a strong brand name, and companies are reluctant to gamble with voice networks because they are so critical to operations.

"VoIP can be disruptive with a cheaper product," Doe said. "But Cisco has to overcome a lot of skepticism and reassure customers that they aren't sacrificing quality."

Or maybe this one:

Fred Doe, an analyst with San Jose-based, Total Future Clarity, said the Unix market continues to grow, but Red Hat faces a challenge because Sun has such a strong brand name, and companies are reluctant to gamble with servers because they are so critical to operations.

"Linux can be disruptive with a cheaper product," Doe said. "But Red Hat has to overcome a lot of skepticism and reassure customers that they aren't sacrificing quality."

See what I mean? This quote appears all the time, for all new companies. It's true, but it doesn't provide any insight.

Matt's posting concludes but giving a few ways that Vyatta can help build it's brand, and they're all good suggestions that we're already working on. If you're a Vyatta customer, you also know that what the Vyatta brand means to you and how conscious we are about making sure our customers are satisfied with what we deliver. In fact, four out of five customers say that we provide better service than the big name networking companies. And that fifth customer, the odd-man out? He says we provide about the same level of service.

So, are you sacrificing anything going with Vyatta? Our current customers say no. If anything, you're getting more. Does Vyatta have to convince even more people that we can do the job? Yes. Should you buy open source out of a sense of charity, based purely on religious commitment to open source ideals, eschewing pragmatism and business sense? Of course not. Vyatta has to compete and demonstrate that it can meet real business needs. Fortunately, we are. And you in today's environment you need to ask yourself whether you'd rather be sending your money to a company with one of the highest gross profit margins in the business or to a new company that can deliver what you need for a lot less.