Microsoft's Bing is experimenting on you

Introduction and zero blame policy

Want to build a service that keeps adding new features and fixing bugs and generally getting better? The biggest problem might be your most experienced people, who know all the ways things can go wrong.

Whether you call it devops or continuous deployment or just cloud services, the way Google and Netflix and Amazon keep experimenting and changing and fixing and updating their systems sounds almost too good to be true. "People say 'it's not possible – and I know because I've done this all my life'," admits Bing architect Craig Miller.

Bing architect Craig Miller

Proof from rivals

Ironically, it was Google that pushed Bing to learn to update so fast in spite of those attitudes. "We wanted to do it daily but at that point we shipped monthly. Everyone said 'you can't do that'. But we had the benefit of competitors saying they were doing some of this, so we had at least some proof points that say it's possible."

Bing had the advantage that it had already built a system where it could run experiments with the service and try things out. "We already had the experimentation infrastructure – we've got a very sophisticated a/b testing system we can run. We run thousands of experiments a day." In fact, Miller says, "we're experimenting constantly on you."

The idea is to get an idea out on Bing as quickly as possible to see if it's any good – if not, another one will take its place

Some of those are big changes – Miller was the person who first used a video loop for the Bing image of the day. Others are much smaller. "On the results page we might make one 2 pixel change and measure that. With the level of experimentation we have, if I change a single pixel, you would see an effect. If I reduce engagement with that change, I'll see that signal," he explains. There are some things that always make searchers less happy, the worst one being pushing things down the page so people have to scroll. "Scrolling is verboten," he notes.

Not all of the changes will work out, and Miller is fine with that. "Ideas are not all good. When I have the velocity, I can try it out for a couple of days and discover that didn't work and it was a stupid idea."

$New ideas at Bing like this metronome or the built-in guitar tuner will get tested out with a fraction of users$

Zero blame policy

For one thing, the same system that lets Bing try new things several times a day lets them change things back quickly. And whether it's a bug or a bad idea, there isn't going to be any finger pointing. "Zero blame," emphasises Miller. "It has to be a learning culture, a culture that says 'we're going to learn from this'."

In fact, he's happy if most of the ideas don't work out. "How often do we succeed? How often do we ship a feature we have put in front of you in a test? 10% of the time." Think that sounds low? "Every company in the world would be ecstatic if they knew 10% of everything they're going to do will be successful," Miller claims.

The real advantage of all those experiments and updates is how quickly Bing can make changes. Miller notes: "Ideas are cheap, but if it's expensive to implement them then you don't do it. You have to have the ideas, you have to have the quality and you have to have the ability to test them and experiment. A lot of things come through and if they fail that's okay because you've got a culture of learning fast.

"If we have a lot of ideas and deliver them faster than our competition, we will win. At the end of the day, that's all it is. If we can deliver faster than the competition, we're all good."

Talking not searching

Particularly because we all use mobile devices more and more, Bing succeeding isn't just about more people running searches and seeing ads, Miller points out. After all, relying on adverts and effectively selling user information to the advertisers brings up a lot of questions about privacy. Miller briefly notes: "We know more about you than you can imagine."

Knowledge graph

And Microsoft doesn't have to concentrate on that for Bing to be successful because, as he points out: "Cortana is built in Bing, and the Knowledge Graph that Cortana is based on continues to grow." Cortana is less about finding things than about getting things done. "Cortana has the desire to be your personal assistant; helping you schedule meetings and connect with friends," enthuses Miller. "Really, the scope is unbounded."

Conversation as a platform

He notes that messaging is gaining new popularity on mobile devices and desktops alike, whether it's Slack or Snapchat. "Think about conversation as a platform – there are things we want to work on to make sure that's a platform we can contribute to. Imagine a conversation where you're talking to a friend and she knows what you like, where you go, where you live. I type in a restaurant name and Cortana can say 'I know that place; here's its menu, here are its hours – do you want to call them?' There's a lot more we could do there. What about parking?"

As the Bing Listens ideas site explains, you might see different Bing Rewards offers compared to your friends

You can see some of that in Cortana, and also in the Bing app on iOS. "It pivots search so that instead of showing you a URL, it shows you an entity. We understand this is a restaurant, so you might want to book a table. And it's personalised because it's on your device – we know what apps you have that can take care of an action like booking the table or mapping the route.

"If you search for movies or trailers, we know where you can watch it; we can tell you it's on Netflix or you can rent it on your Amazon account. We have all those actions tied in to search."

That's a direction Miller says is particularly interesting for Bing. "It's a fascinating piece of our new generation in mobile experiences, and that's where we need to be stronger."