5 metrics to start with
After rearchitecting, Etsy started with five metrics:
- Listings in the marketplace.
- Number of sales.
- Number of people logging in.
- Number of people registering
- The rate of posting it Etsy's bugs forum.
But that was just the beginning. Today, Elliott-McCrea says, his team is adding about 300,000 new metrics a month.
The metrics are tiered. For instance, Etsy has about 20 tier 1 services with four to 20 metrics each. If an algorithm detects a variation in one of those metrics, it will trigger an immediate response. Tier 2 and tier 3 metrics are less urgent. One tier will trigger a call while another will trigger an email.
The metrics evolved, and continue to evolve, as Etsy learns, discovers and iterates.
"You have to have the culture right," Elliott-McCrea says. "First and foremost, what are we here to do? We had five metrics when we started. We were pretty clear about who we were and who we were trying to serve. It's not about doing it all upfront. You have to have a process that encourages learning. You have share your mistakes, learnings, PSAs. If you don't have a process that focuses on that, you're probably not going to get to the right metrics."
A culture of learning from mistakes
At Etsy, that learning culture is based on openly sharing mistakes to understand what went wrong, how it went wrong and what indicators might have informed the person who made the mistake that something was going wrong.
"If you get a 500 error on Etsy, you get this great graphic of a woman knitting a sweater and it has a third arm and she's looking very confused," Elliott-McCrea says. "We give a three-arm sweater award to the person that makes the most spectacular failure in any given year. It takes a fair amount of skill and talent to do that. You can't do it by accident."
Etsy calls its process a Blameless Post Mortem. The idea comes out of Just Culture, a systems design theory that originated in the medical community as a way to limit medical malpractice.
"You assume best intentions," Elliott-McCrea, who has been a recipient of the three-armed sweater award himself, says. "You assume skillful actors and you try to get to the local rationality. I made a choice. I deployed that code to the home page. I had confidence when I did it. Something happened and now it's an infinite loop and I've taken down the website. What was it about what I was thinking at the time that gave me confidence? There's no blame, no should haves, no editorial at any point. Here's what I did. Here's what I thought was going to happen."
Sign up for CIO Asia eNewsletters.