I love marketplaces and marketplace data, so a couple months ago I grabbed some Airbnb data and made a slide deck. A few people have asked me about it, so here it is along with a short summary.

My goal was to gather data around potential product strategy, focusing on the following questions.

1. You can’t book a great place if you can’t find one. How good is Airbnb’s search ranking algorithm? What are its problems (are listings returned on the high end of the price filters? are too many poorly reviewed listings being shown?), and how can we fix them?
2. A big chunk of Amazon purchases comes from its related book algorithms. How well does Airbnb’s Similar Listings model perform? Does it showcase great related properties, or are its suggestions generally irrelevant?
3. Airbnb probably wants (and needs) to evolve. Where do users find the booking experience difficult and confusing, and what kinds of new products would improve it?
4. Different people use Airbnb differently. What are the archetypical patterns of travel, and is it worth improving their experience in different ways?
5. Hotels are Airbnb’s major competitor. When do users still prefer staying in a hotel and why?

I’ll give an overview of what I found below, and check the slide deck for more detail.

Airbnb Search Quality

Let’s start with Airbnb’s Search. In order to understand the problems and benefits of their algorithm, I asked 100 Airbnb users (from a digital labor platform of my own that I created) to think about a trip they wanted to take in the next year, to search for a place to stay on Airbnb, and to rank the quality of the results.

For example, here’s one Airbnb user who described a trip she wanted to take with her husband and kids to Disneyland:

“We love going to Disneyland and would like to go back sometime this year around the beginning of the summer. Ideally we would go for a week but it depends on price. I would want to go with my husband and kids because it’s a really fun and special experience for all of us.”

She then searched for places to stay on Airbnb:

And this is how she rated the first search result:

“Fairly bad result. It looks like a nice play to stay but I don’t like that it’s actually part of another house. I would want to feel like we have total privacy instead of sharing certain areas. There are also an awful lot of rules about when and what you can do. I understand to an extent but seeing it all written out comes across as kind of negative. It’s also not really that close to Disneyland.”

Even from just this one example, we can see a few problems and potential improvements to the search algorithm:

1. Where exactly is Disneyland, and why are listings so far away from Anaheim shown? If we look at the map of search results, it’s impossible to tell how close each listing is to Anaheim’s main attraction. Given that most travel is around landmarks in new and unfamiliar locations, how much quicker would the search and booking process be if it was easy for users to figure out where attractions were located?
2. People like to use Airbnb as a home away from home, and strict rules detract from this experience. Is it worth adding features to the ranking algorithm that measure how home-like a listing may be?
3. Staying in an RV may be fairly private, but it’s not the same as getting the place to yourself. Should RVs and other such listing types be penalized in searches for an entire property?

Here’s a numeric summary (and check out the deck for more user feedback). Around 2/3 of search results were judged good, while 1/3 were judged pretty bad:

Search results lower on the page are indeed judged worse (so the ranker does seem to be pulling out a signal):

And these were the main problems users had:

For example, the top three issues were:

1. Too expensive. Sometimes search results are on the high end of price range filters (though worth it), and other times they’re simply overpriced for what you get. Airbnb already has a price prediction algorithm; would it be worth using it to differentiate between the two?
2. Unappealing style. Listings often don’t have the style users are looking for. A listing, for example, might not be hip and modern enough, or a male user might encounter listings with pink wallpaper and flowery decorations.
3. Bad location. Results are often too far away from tourist attractions (e.g., listings 30 minutes away from the Las Vegas Strip), or in bad neighborhoods.

(Note: this sample of raters is, of course, not necessarily representative of Airbnb’s actual user base. But I think the problems they highlight are interesting and useful to understand anyways.)

Airbnb Similar Listings

I ran a similar study on Airbnb’s Similar Listings algorithm. Overall numbers are close to those in Search: roughly 2/3 of Similar Listings are good results, and 1/3 are bad.

However, one big problem that showed up on Similar Listings was the problem of inconsistent listing types. For example, users often found great search results where they would get the entire home to themselves, but all the Similar Listings displayed would be for shared, communal properties, which they obviously wouldn’t want to book)

Deterrents to Booking

What blockers prevented raters from wanting to book a place, and what information would have made the search and booking process much smoother?

One blocker was neighborhood discovery. Travelers often don’t know much about where they’re traveling, so it’s unclear which neighborhoods are convenient, or whether a great-looking listing is actually in a safe location.

For example, here’s what one rater traveling to Belfast wrote.

“It looks great, and I’d definitely want to contact the host so long as I do some research and remain as enthusiastic. The price is reasonable and the apartment looks great. The main thing I would want to check is the area it’s in, Lisburn. I don’t know how convenient that is compared to the rest of Belfast and the Titanic quarter. If it was convenient, I would probably go for this.”

Could Airbnb provide more information about neighborhoods within their site, or better guide users to where they’d want to stay?

Another feature that would have helped ease worries about listings, or which would have made listings look even better (or appropriately worse), was Helpful Reviews.

Reviews on Airbnb are currently chronologically sorted, which means feedback like the following might be all you see on the first page.

But reviews are a key part of the decision-making process. Sites like Amazon and Yelp highlight their most helpful user reviews, and 80% of users say they like to read book reviews before making a purchase. (And since traveling is usually a much more expensive and extensively researched decision, reviews are probably even more important here.)

Uncensored user feedback is helpful for understanding location, host friendliness, cleanliness, and more. How much less worried would users be about booking a place, if they better understood what other travelers liked and disliked about it?

A New Kind of Search

Let’s look forward a bit. Google is continually evolving its search experience, from letting you search for images (and even reverse image search), to incorporating videos and news stories into search pages themselves, to improving the types of queries it understands.

So what could the future of Airbnb search look like?

One issue with its current state is that while it might be pretty good if you know exactly where you want to stay, it’s less useful if you don’t. (This ties back to the issue of neighborhood discovery.)

For example, one of my raters wanted to book a skiing trip in Wisconsin with his wife.

“Within the next 2 months, I’d like to go to the northern part of my state (WI) to a skiing resort. I want to go for the weekend with my husband. I want to do this because we need a getaway and it would be romantic and we would still have plenty to do.”

So he added a skiing keyword to his search…and received completely useless listings as a result.

This, for instance, was the first listing:

“Horrible result. The room is not ideal, it’s not romantic at all. The location is unappealing, it is near an urban area and there is no mention of skiing. I’m looking for a place near a skiing area, I don’t have a specific location, because I don’t know where they are, just that we have them. The price is okay, it’s just not what I want at all.”

The rest of the listings were similarly unrelated to skiing.

If we inspect the results, we see that the reason they’re returned is poor keyword matching: the word skiing appears in the description of the host (she likes to go skiing), but not in the description of the listing itself.

This problem may be fixable (we can try to machine learn which keywords should be matched where), but the question is a more general one: is it worth designing a better experience around general activity and travel search, instead of the main context being mere location?

After all, if Airbnb can make finding places to travel to easier, thereby encouraging more travel, or if it can convince users to spend their entire travel planning process on Airbnb, that could be another strong driver of growth.

Advantages of Hotels

Another path to more bookings is, of course, to increasingly convince travelers that Airbnb is a better option than staying in a hotel.

So what are the main advantages of the hotel experience right now?

1. 65% of users mentioned that Airbnb had quality control problems. Hotels are a trusted brand that provide a consistent and guaranteed level of service (you know what you’re getting into), whereas the quality of Airbnb listings can be uncertain and erratic. Your host may be unfriendly or unresponsive, the listing may be dirtier than advertised, the pictures might not be representative (or there might not be enough), and so on. (Promoting helpful reviews might be one way to help solve this problem.)
2. 45% mentioned that booking listings on Airbnb can be a hassle, especially at the last minute. There’s a long and annoying process where you have to interact with the host, your host might cancel on you at the last minute, you often have to contact multiple hosts in order to see who will respond, and it’s not even clear that a listing will be available. In contrast, hotels are a one-click process. (While Airbnb does have an Instant Booking feature with select listings, many users didn’t seem to understand it or realize it existed.)
3. 30% mentioned the awkwardness of the forced human interaction. It can be stressful and tiring to wait for a host to let you in when you arrive, the communication process during booking is again an annoyance, and users also worry about disturbing owners too much with their asks and complaints. Companies like Uber grew in large part by removing the annoying part of human interaction (calling a taxi), and making the rest (chatting with your driver) optional. Should Airbnb do the same?
4. 25% just liked the amenities of a hotel (free breakfasts, gyms, spas, room service, and so on).

Travel Patterns

Finally, let’s take a look at typical patterns of travel.

People travel for different reasons. Some vacations are centered around discovering new places, some are for romantic honeymoon getaways, and others are just to visit family during the holidays.

People also travel in different kinds of groups. Families may enjoy Airbnb more because of the cooking options and the possibility of having rooms separate from their sleeping, early-to-bed children. Coworkers traveling together may enjoy it less because it’s harder to stay near each other (and because of potential expense difficulties).

So what are the clusters of traveler types, and how big are they? I ran a few Google Consumer Surveys, and here are the results.

Can Airbnb design different experiences around these groups, and see if they use the site in different ways?

And that’s it. For people interested in more details, there’s more information in the deck here.