Friday, June 22, 2007

Stalinist Central Planning Doesn't Work

I was reading through some blogs today and Dana Blankenhorn's posting at ZDnet, The telecom revolution will not be televised, caught my eye. In it, Dana describes becoming "ticked off" at Maria Bartiromo on CNBC as she described the NxtComm show, saying how "'the industry’s heavyweights' were there to 'decide on the future of telecommunications.'"

A few months ago, I was being interviewed by a reporter about Vyatta and why I thought we would be successful. I told the reporter very directly that I believed Vyatta would succeed because (paraphrasing) "...the world has conclusively proven that Stalinist central planning doesn't work." At this point, Tom McCafferty, eavesdropping in the next cube over from me in the Vyatta office, started laughing hysterically.

After the call, Tom stuck his head over the cube wall. "Did I just hear you say that Cisco and Juniper are Stalinists," Tom asked?

"Well, sort of," I replied. "No, I didn't mean that they are communists or have killed millions of people, but only that the future is not served up as the direct result of a grand plan hatched by the strategy office of the market incumbent. I was trying to communicate the fact that companies with large amounts of market power often want to strong-arm their customers into doing what is best for the company, not what is best for the users."

"I was trying to make the point that open-source has a huge community component to finding its direction that proprietary systems just don't," I said. "Therefore it's more responsive to users. Because the technology is open, anybody with a good idea can change it or extend it, unleashing all sorts of creativity that never gets off the ground."

Tom nodded his head and laughed. "I still thinks its hilarious that you just called Cisco Stalinst," he said, sitting back down.

The reality is, it's the same idea that Dana picked up on in his blog. Describe it how you want---"bottom up innovation," "The telecomm revolution will not be televised," or "Stalinist central planning doesn't work"---it's all the same idea. Everybody is finally asking themselves, "By what right do 'the industry’s heavyweights' get to 'decide on the future of telecommunications?'" Is that something users even want? Is that good for anybody but the "industry heavyweights?"

While many people in proprietary software PR assignments like to suggest that somehow open source software is "communist" in nature because the code is open and given away, I contend that in many ways it's very capitalist (Eric Raymond has described it as something entirely different, a "gift culture," in the Cathedral and the Bazaar). The open nature allows many more people to shape the future, both users themselves as well as other vendors. The fact that there is more competition keeps everybody honest and prevents anybody from strong-arming anybody else. Companies that provide value get paid. Those that don't disappear. Don't like the future somebody else proposed? Create a new one. If people like yours better, you win. Don't add value and you can't make a buck, then you die and people pick up where you left off, without orphaned customers or the like. Whatever happens, the "industry heavyweights" aren't allowed to "decide on the future of telecommunications" without everybody else having a say-so in the matter.

So, I'll repeat it again: Stalinist central planning doesn't work. And Dana is right that the telecom revolution won't be televised. It might be carried over a VoIP call through an Asterisk PBX running over a Vyatta network, however.

Monday, June 18, 2007

Nice Vyatta review

Nick Davey reviewed some of the new features in Vyatta Community 2. One thing I like about Nick is that he's very straightforward and honest in his reviews. The Vyatta system isn't perfect (yet), and Nick tells you what he likes and doesn't like. Most of the things in his wish list are features we have in various stages of planning an execution.

And don't worry, Nick, no need to apologize.

Sun on software patents

I have posted some thoughts about patents before in this blog. I just stumbled across a posting by Sun's CTO, Greg Papadopoulos (yes, I checked the spelling of that 3 times), discussing software patents. Greg makes the point that in today's world of software, copyright is probably a more powerful instrument for software authors than patents. Further, copyrighted code that is freely distributed is better at serving the public interest in terms of teaching others about your invention than is a patent. As the holder of multiple patents, I'll agree with him wholeheartedly. The fact is, nobody but lawyers read patents unless there is lots of litigation on the table. I can barely understand my own patents. Code is soooo much better at communicating how things really should work.

In his posting, Greg, of course, pats Sun on the back for some of the things they have been doing with ZFS, OpenSolaris, and Java, and rightly so, I believe. The fact is, Sun has been embracing open source quite a lot. My favorite quote:

In case you haven't noticed, driving to a state of complete FOSS is exactly what we are doing at Sun. With some narrow exceptions, all of our software efforts are or will be under a FOSS license, and we will actively build and participate in communities around these code bases, and work as transparently as we possibly can.

In case you didn't notice, we added some Sun hardware to our Vyatta Ready hardware list. In fact, Sun called us to do the certification. That's quite embracing, I believe.

Friday, June 15, 2007


In previous blog postings, I have written about the myth that networking systems must rely on special-purpose ASICs or network processors, and I have pondered what will be achievable as MIPS become more and more free. Today, ZDNet posted a story discussing Intel's stated direction to deliver processors with 80 or more cores. While not here today, this represents a huge jump in commodity computing performance. What's interesting is that the article suggests an idea that I have had for a while: what if instead of having 80 x86 cores, you instead had 50 x86 cores along with 30 specialty cores? Some of those specialty cores could perform crypto operations, perhaps, or rapid pattern matching for anti-virus, anti-spam, or intrusion detection and prevention functions. Today, there are already x86-compatible processors from VIA Technologies that contain hardware crypto acceleration operations.

As you can see, the possibilities are quite limitless for commodity technology. Thankfully, Vyatta will be delivering software that will help take advantage of these technology advances to run your networks faster, for less money, and with the freedom of open-source.

Monday, June 04, 2007

Vyatta CAREs

In our last installment, I talked about receiving horrible customer service from a 1-800-florist. I said that I'd look at Vyatta's customer service process and discuss how my flower-buying experience could have been so much better had I been working with Vyatta and not the company I originally chose to buy from. Unfortunately, Vyatta just doesn't do flowers and so I didn't have that option. If we did, I'm sure I would have been satisfied.

Inside Vyatta, we have a slide that goes in our internal presentations that talks about our primary corporate mission. We believe that Vyatta's success is directly tied to the size and health of the community that surrounds us. Our mission is therefore to "Help users around the world develop a passion for Vyatta." Nope, it isn't to grow our revenues by a given percentage, get the company profitable, or IPO the company. We believe those things will all come in time if we're successful in executing to our primary mission.

Yes, I know that you're probably jaded and you have sat through corporate feel-good presentations that profess some of the same thoughts. In our case, though, I think we really mean it. The reason is that Vyatta fundamentally exists as an open-source company. We don't have a lot of proprietary technology that we hold hostage and only allow our customers to access by paying us money. We try to make money by developing quality code and then providing our users a set of value-added services that they really want to pay for.

I'll tell you that coming from a long history of proprietary networking companies, this inversion of your business model is quite sobering. It forces you to evaluate your real value to customers and puts things into stark perspective. Whereas things like customer service are a cost-center at other companies (something to be minimized), they are part of Vyatta's fundamental product offerings (revenue to be maximized).

I should note that Vyatta subscriptions offer more than just technical support. Vyatta subscriptions include:

  • New features and functions through the life of the subscription
  • Bug fixes
  • Software downloads and automated updates (ISO and update repository)
  • Case Management (add/edit/review)
  • Subscription Knowledge Base
  • Research Tools and Info
  • Professional Technical Assistance Center (TAC) Operations

Vyatta also offers a set of consulting services that can help you get Vyatta deployed in your network and configured just the way you want it.

To deliver all the above, we rely on building a first-rate customer service organization. We have a set of values that we demand of that organization (and the whole company, really) that ensure you have the best experience possible. First, let's recognize that no matter how perfect a company is, there are always going to be times when things go wrong. If you're an Internet florist, then sometimes the flowers are going to arrive wilted and looking sickly. If you're creating open-source networking software, there are sometimes going to be bugs or installation problems. Assuming that everybody is striving to keep those problems to a minimum, the real winners are the companies that know how to make things right when they inevitably go wrong.

To help Vyatta deliver repeatable, excellent customer service experiences, we came up with a simple acronym, CARE, that describes the behaviors we want Vyatta employees to use when dealing with anybody. Here's the explanation:

We truly appreciate the opportunity to serve our customers and partners. We let them know that each and every time we interact with them through timely, courteous and professional communications. We respect their technical knowledge and adjust our procedures accordingly. We strive to make their day better.
We insure that the customer or partner has our name, phone number, email and office hours. We respond promptly when they contact us and we proactively communicate updates. We escalate willingly when they request it. We proactively bring in additional resources as necessary.
We clearly communicate what we will do and when we will do it. Then, we do what we said we would do, when we said we would do it. We fully and completely meet our commitments on-time, each and every time.
We take ownership of the issue and we act. We never pass the buck. We never say “that’s not my department” or “you need to call…” When we need to make a quick decision, we do the right thing and sort out the details later.

Now, things like being Courteous may sound trite. You may think to yourself, "Well, of course being courteous is part of customer service. What is the alternative, being discourteous?" While it's obvious that being polite is part of the goal, being truly courteous goes beyond good manners. It also means things like, "We respect their technical knowledge and adjust our procedures accordingly." This was born out of an irritation that just about everybody at Vyatta has had with other technology customer service departments.

For example, whenever I call my broadband access provider with a problem, the level 1 technician always asks me to reboot my computer. Being a technical guy who builds IP networking equipment for a living, by the time I call tech support, I already know most of the problem, and rebooting my computer isn't part of the problem. I have found myself more than once begging to be put through to the level 3 tech support person in order to help the company help themselves, because something in its network is wonky and I know it but they have yet to figure it out. All of us wish they would annotate our customer service records with a little flag that means "this guy knows what he's talking about, so dispense with the 'reboot your PC' thing and patch them directly to level 3 support whenever he calls." At Vyatta, we strive to work with you at the level you're at. If you're a networking novice, we'll talk to you in terms you can understand. If you've built world-class networks, we'll treat you like the expert you are.

Next, we're accessible. If you want to escalate to somebody, we're more than willing to do that, all the way to the CEO if necessary. We honestly hope it never comes to that (if it does, then something else is very wrong), but if you want that, we're willing to do it. We'll make sure you have all our phone numbers and email addresses. My direct voice number is +1-650-350-3148. My email is dave at Vyatta.

Being reliable seems like table-stakes, but it's amazing how many companies aren't. They say they'll call and they don't. They say something will get done, but then you find yourself calling back again and again, re-explaining yourself to a new person who has never heard of you or your problem, and the cycle repeats. Vyatta wants to break that cycle.

The biggest one for me is empowerment. Many organizations use outsourced customer service departments to keep costs low (because customer service is a cost center, not a product). While many people will bemoan the low skill levels of outsourced call center operators, I think the biggest problem is that they are given no empowerment to do the right thing for the customer. Typically, the organizations that run these outsourced operations have very strict "scripts" that are developed in conjunction with the companies for which they work. While it's fine to try to use scripts to increase efficiency and repeatability of standard solutions, you also need to give people the latitude to go "off script" when required. Sometimes a given problem just doesn't fit the script or the computer system and the right answer isn't to tell the customer that you can't help them or to call another department. Customer service reps need to be able to work with customers to come to quick resolution and then work offline to make the right thing happen without putting the customer in the middle of it.

Now, I should also state that putting principles like this to work requires that you have a staff that is mature and responsible. All Vyatta's customer service techs are experienced building and supporting large networks. They get it. We don't stick people in tech support as the job for newbies that nobody else wants. Our technicians aren't wet-behind-the-ears recent-graduates with entry-level positions, hoping they can make it out of the bull-pen to that other job they really want. Our people love delivering great support to you. Because we hire mature professionals, we can give them full discretion on how to handle cases.

So far, these principles are serving us quite well. We have been receiving rave reviews from our customers. A small set of comments we have received to date includes:

“The incredible level of their support – even their first level of support, compared to ANY other place I’ve dealt with … it’s just outstanding.”
“As I've said in previous mails, I'm so impressed with your support and how committed your people are in solving the problem rather than closing the case.”
“[Your] expertise and willingness to help me solve my problem was excellent. Vyatta provided excellent technical assistance in my time of crisis. Thanks, Vyatta for an excellent product.”
“[You] have been excellent about all the support we have received, and [you do] a great job in taking ownership of all cases/tickets.”

So, let's take a quick second to examine how these principles could have saved me as a 1-800-florist customer.

First, let's notice that I wasn't angry when I initially called. In spite of the problem with the product (partially dead flowers), I just wanted a credit and I would have probably used the florist again in the future. Instead, I'm blogging about my bad experience for the whole Internet as a case study in bad service and I'll go out of my way to use somebody else.

It was clear that when I called the first customer service rep, Bernadette, she had been given guidelines that said that if anybody isn't satisfied with his order, offer to resend it first. If that doesn't work, promise him a partial credit. That's fine insofar as it goes, but in this case I had already made things right by ordering from another florist. I didn't want them to send yet more flowers (which they incidentally did anyway). A full refund, while not the most desirable for the company, should have been an easy thing to get. In fact, they had promised it a couple of times to me in writing, on their web site and in the email confirming my order. The fact that Bernadette wasn't empowered to give me that is poor planning on the part of 1-800-florist.

After I felt lied to and called the second customer service rep, Bob, he should have made it right on the spot. He told me that he might not be able to cancel the redelivery. Even if that was true, he should have said that they'd be redelivering it anyway as a free gift to me, even as he gave me 100% credit for my order. The fact that 1-800-florist can or can't cancel a redelivery order at a certain point is its problem, not mine. If he couldn't cancel it, he should have made it appear that it was a bonus gift and not a problem.

The fatal mistake Bob made was asking me to call him back after the first customer service rep had gotten out of my record to allow him to work. The fact that his computer system doesn't allow two customer service people to work in a record at the same time isn't my problem. If need be, he should have copied all my information onto a pad of paper and then gone back later that day and done whatever computer system gymnastics were required to give me what I wanted. Asking a customer to call back later means that you're forcing the problem back on the customer and not taking responsibility for it.

Finally, I asked to speak with Bob's supervisor. The supervisor had the nerve to decline to speak with me. That's simply inexcusable. While it may be inconvenient to take the call, as a first-level customer service supervisor, that's one of the most critical parts of your job. Finally, she acquiesced to at least calling me back, but Bob wouldn't give out her contact info so that I could contact her if she failed to do so. In effect, the company was saying, "trust us, in spite of what you have just seen of our poor customer service, we'll really make it right." That's too much to ask in an escalation situation, because in that case you know right then that trust is eroding quickly. Ideally, with a large customer service department, a supervisor is always on hand to speak with a customer, but I wouldn't have expected a director or somebody higher up the chain to have been accessible on a moment's notice. What really got me angry, however, was that the system was rigged to deny me the right of appeal to a higher power. I sometimes understand that low-level folks are battling computer systems that won't cooperate and process manuals that don't give them latitude and I typically try to cut them some slack, but I have no sympathy for a system that tries to cut me off and retain the power in the relationship.

Whew! This has been a long one, but hopefully it helped you understand the philosophy that we have at Vyatta. We're committed to delivering great customer support as a part of every interaction we have with you, whether you're a paying subscription customer or just a prospective customer asking us about our product. We want you to be satisfied. We want to "Help users around the world develop a passion for Vyatta." We think CARE helps us achieve that. From the feedback we're receiving, it's working.