One of the most important things we do on the product team at x.ai is talk to our users. Talking to the people who are actually using our product every day gives us an inside look into the triumphs and frustrations of our customers. We learn what we’re doing well and what we need to do better. But one of the key frustrations for us is finding the right users to talk to and scheduling those meetings.
Time is our most precious commodity, so we don’t want to waste it talking to the wrong people. If we’re building a feature for people who schedule lots of meetings with people outside their organization, we don’t want to spend time interviewing folks who never do that. Unfortunately, recruiting the right users for one-on-one research is a time consuming process. That’s why we decided to build a better solution using Typeform, Zapier and Amy.
The old way
Our original system for recruiting participants for user research wasted a lot of time—it was highly manual and had a ton of bottlenecks.
Here’s what we used to do: we’d send an email blast to users with a link to a Google Form that helped us qualify them for the particular question we were asking or feature we were testing. We’d manually review the responses and then email anyone who met the criteria to confirm their participation. For those who responded, we’d send out an email using Amy or Andrew to set up a call. We’d also have to individually email everyone who didn’t meet our criteria to let them know they weren’t a fit for this round.
And, making matters worse, we’d send the meeting invite emails from just one person’s account, usually from my teammate Jason Grant, x.ai’s Senior User Experience Designer. That meant that Amy would schedule meetings based on Jason’s calendar, even though he wasn’t the only person who could conduct the user research interviews.
None of this scaled at all.
The better way
We needed to develop a system that had a faster turnaround time, increased the percentage of people who showed up to the interview (faster to contact = faster to schedule = more likely to show up to the interview or call), and would be as automated as possible. That would give us more time to conduct interviews and talk to our users.
The first step was to create a dedicated email address specifically for user research and create an x.ai account for that address. This allows us to send all user research-related emails from a single account that anyone on the Product team can access. It also means we schedule meetings from the new email’s calendar without having Amy/Andrew look at our own individual work calendars. With multiple people on the team who can conduct interviews, it’s rare that a meeting will be scheduled that no one can take, but if that happens, Amy makes it easy to reschedule.
Next, we decided to switch from Google Forms to Typeform. Typeform offers more design options, which allows us to better maintain x.ai’s branding, and makes it easier to pre-qualify participants with more powerful relational questions and variables.
With our new email address and form built, we hooked the entire thing up using Zapier. We set up a series of Zaps, which are essentially computer instructions that connect one or more applications, to connect our Typeform form to Amy.
The first Zap we use automatically sends emails based on the responses to the form. We recently ran a user research study around how people book meeting rooms at their office, so we instructed Zapier to send an email from our dedicated user research email account to anyone who answered yes to the question, “Do you use meetings rooms at your office or place of work?” and had agreed to talk further with us on the phone. The customization of the Zap allowed us to include email@example.com in the CC field of the automatic email, and in the body of the email our copy instructed Amy to find a time to set up a call with the user. Anyone who didn’t meet our criteria was ushered to the end of the form, where we let them know they weren’t a good fit for this round of research.
This new system removed a ton of the time burden from us: we no longer have to read through individual form responses, we no longer have to manually queue up emails, and we don’t have to hunt down suddenly unresponsive users. But we still had one problem: because we’re using a new dedicated email account for this, we weren’t easily seeing how many interviews had been scheduled. We needed a way to be informed when new calls were set up to be able to close the form after a certain number of calls.
Zapier also solved this problem. We used a second Zap to connect Gmail to Slack. Now, a notification is automatically sent to Slack any time a new email is sent out from our user research account. This makes it easy to track when someone fills out the form and meets our criteria, instead of us having to remember to check Typeform or the shared Gmail account. From Slack we can see how many emails we’ve sent out to set up interviews, and it makes it easy to know when to shut off our Typeform to new entries.
Our new solution removes most of the overhead from the user research recruiting process. Now, when we send an email to customers with a link to the Typeform, we start to see meeting invites appear in the User Research inbox within minutes. We can easily track how many we’ve scheduled through Slack, and turn off the Typeform when we have enough participants.
Our schedule success and attendance rates also went way up. In the past, we’d lose almost half of the original research recruits because our manual process took too long. The first round using our new process, we successfully scheduled with every qualified participant (vs 50% before), and had an 86% attendance rate for our calls.
We plan to continue refining our Zapier+Typeform+Amy workflow. For example, we may add Zapier’s Lead Score app into the mix to send us additional customer data that will help us prepare for interviews.