Testy A/B w Facebooku

Jared Spool, bardzo znany popularyzator użyteczności, autor podcastu SpoolCast, w jednym z ostatnich odcinków przepytał Julie Zhou, Product Design Managera w Facebooku odnośnie procesu powstawania nowych funkcjonalności. Częścią która mnie zaciekawiła najbardziej, było ta poświęcona stosowaniu testów A/B.

Jared: (…) You mentioned that you’re using A/B testing. What’s been the history of that at the company? Is that, again, something that sort of picked up after the whole news feed thing? And how did you start? And how has it changed once you got started?

Julie: Yeah, so definitely after Newsfeed — I think prior to Newsfeed, we really weren’t doing that much with user testing at all or A/B testing. I think the group at Facebook that really spearheaded the idea of A/B testing was the growth team.

About two years back we thought, what is the number one goal for the company? For everyone, it was basically growth, right? We’re not going to get anywhere if we can’t demonstrate the value of Facebook to new users.

Because Facebook’s a social network, if you have no friends on it, the service is essentially useless. There’s no reason you would really go back. Making sure that people had their friends on Facebook was our number one priority, so we came up with the team that would be dedicated solely to this fact.

This is sort of a very broad problem, it’s very ambiguous. But the idea is, how do we get users onto Facebook? And not just get them to register, but how do we get them engaged? How do we get them to learn how to use Facebook and basically become a user that understands its value and keeps logging in every day?

The team, because it’s such a broad problem and really we had no idea — we had some guesses. We thought “Well, of course we should look at a registration flow. First we should look at our new user experience.” What this team really did was really try to understand that core question — how does a user come to Facebook and get engaged?

A lot of the ways in which they did that was through A/B testing, and so you’d have a hypothesis like “Oh, maybe it’s the registration flow, right? Maybe if we made that a lot simpler or whatever, then a lot more people would be coming.” So it would have like different versions of this registration flow, and they’d put it all out there and A/B test and see which one did the best and which one moved the needle.

Then they would say, oh well maybe it’s not… They improved the registration flow, but now maybe we need to focus on the new user experience. Or we need to focus on people inviting their friends. Or maybe we need to focus on showing them features like photos.

People have a lot of ideas for what would get a person engaged on Facebook, and the way to test these ideas was just to run a huge number of A/B tests and try to see which aspects of improvement would really get more people to be engaged in Facebook. Part of that, too, was A/B testing like, “Hey if someone hasn’t come back to Facebook for a while, maybe we’ll send them an email. What should this email say? Well maybe the email should show them stuff that’s been going on with their friends, or maybe the email should just be a simple text block that explains we miss you, come back.” And people had a lot of ideas about that.

The growth team would just say, “OK we don’t want to just debate this. Let’s just put it out there and A/B test it and try it all.” That method I think worked really, really well because I think at this point the growth team has a really good understanding of all of the data that they’ve collected and of what really gets people to join Facebook.

We’ve tried to adopt that in other parts of the features — other products as well are trying to model what the growth team did. We usually do A/B testing for smaller features and small tweaks. Generally if we do a home page redesign, we won’t have like two different versions of the home page redesign and A/B test both because that’s just really expensive to do.

But it works really well for trying out small little features and we have a little bit of a box. You know, what would be the best thing to put in this box to increase engagement? Like, those were the kind of things that we’re constantly A/B testing.