This post is the fourth in a series on Product Management for the AppExchange. The third installment: Planning for the Future: Roadmapping & Prioritizing post-MVP AppExchange Products can be found here.
The Importance of Customer Feedback
If the upstream function of product management is asking the correct questions to get a winning design, then the downstream is collecting customer feedback to get the answers to those questions asked. Without proper customer feedback, the only way to know if your product HASN’T achieved market fit is via lagging indicators such as MoM customer increase, conversion rate, churn rate, average revenue per user (ARPU). Worse, when you finally see an issue, it’s been months since a problem, which you don’t have root cause understanding of, is finally visible. Consider also that during this lag, you are still working down your backlog, potentially compounding the issue by continuing to develop before you know if your solution is directionally correct. Collecting quick customer feedback is your proactive way to understand:
-
Market/fit or lack thereof
-
Problem validation
-
Customer satisfaction
-
Future priorities
What if you could make feature pivots or roadmap changes early in your development cycle? The cheapest changes are those that occur pre-development. Collect feedback about vaporware! Once you have a MVP in the marketplace, subverting future feature development, if necessary, allows you to be as cost effective with investment capital as possible. This potentially shortens your negative cash flow period and accelerates your business to profitability. For any Salesforce ISV Partner business, for any SAAS business, the period between start and profitability, the cash flow trough, is the riskiest time.
There is one more tactic that product managers need to understand. You might have a good idea, you might even have a good idea that’s profitable. Without directly communicating with customers and end users of your product regularly, it’s impossible to know if you were smart or lucky. It’s a good bet that you were lucky and that luck will run out. What if you could ensure that your luck NEVER runs out? You can.
Talk to your customers every single day.
Call contacts you know using your product, sit in on support calls, get CC’d on customer cases. Do all of these things and utilize the ideas we will explore below. If you establish a continuous and statistically significant pipeline of customer feedback into your product development organization, you can be sure that if you do end up on an incorrect development path, it won’t last long. There’s a strong second reason to establish this pipeline of communication. Some parts of your product will be obvious and required (i.e. no webstore can function without a shopping cart and checkout). However, beyond those essentials, your product roadmap isn’t static. Customer needs are emergent and change as externalities change (economy, technology, etc.). By maintaining continuous communication, your customers will inform you of these changes (in the aggregate) long before you will have the information that would cause you to react otherwise. All you need to do is listen.
Feedback Channels
Collecting customer feedback can be daunting if only because there are so many ways to accomplish the task. You can use one or, better, many of the following:
-
Product Analytics
-
Survey Tools
-
Live Interviews
-
Social Listeners
-
Success and Partner Communities
It’s very easy to get lost in the uncertainty of the extensive options available today. Truth be told, there isn’t one way to find success. Having a suite of options at your fingertips allows you to adapt to the specific needs of your product/problem/customer. Ultimately, you are collecting data about the usage and satisfaction of your product. The method is just the vehicle to achieve that goal. Whichever channel(s) you do choose, make sure you can get both qualitative and quantitative results. Without that evidence and reason, you cannot make reliable and defendable product decisions.
“If you cannot measure it, you cannot improve it.” – Lord Kelvin
Product Analytics
A popular method today of collecting “feedback” is to collect event data directly from the app itself. Tools such as Mixpanel or Heap Analytics make it easy to grab clicks on UI components all the way to marking customer journeys and tying users to revenue. These tactics are very useful for understanding unplanned customer journeys or collecting insights that are only available through aggregated usage. There are a few caveats for Salesforce.com ISV Applications, however:
Client Side Libraries (Heap/Mixpanel):
- Lightning Experience Caveat: Lightning Experience does not currently (as of Winter ’17) allow for client side callouts. You can, however, collect usage data of your Lighting components but it must be stored inside a Salesforce Org first before an asynchronous callout can then be made to the analytics collection engine. This is a gross oversimplification and a Technical Architect should evaluate any solution proffered for fit within executing contexts. No claim is made by this author about performance impact or legality of such a feature.
- User Identifiable Data: for any Salesforce ISV application that will be sold via the AppExchange you may collect usage data but you may not collect any user identifiable information. Salesforce Security Review will catch, inspect and prevent any application from going live on the AppExchange which violates this tenet. Some examples of this are email addresses, names, usernames, physical addresses, phone numbers, etc.
- Salesforce Usage Metrics: Salesforce provides some usage metrics for ISV applications. This feature tracks the execution of Visualforce pages and some statistics of custom object usage data. Frankly, the feature is limited, has not been enhanced in many releases and has spotty performance. Also, the data is exported in a flat text file, requiring text data mining to extract the data. It is this author’s hope that Salesforce will expose more back end log data about ISV application usage and enhance for Lightning component usage.
Survey Tools
Feedback surveys are useful for reaching a large number of respondents relatively quickly. While they are more ideal for an in-market product, they can also be used in a limited fashion for testing the response of new features (utilizing static mockups or images). The key to effective surveying is specificity. Ask very specific questions that result in actionable answers. Loosely worded questions that allow hypothetical thought rarely end up with actionable feedback.
Example: “Would you use an enhanced shopping cart that allows you to change quantity in real time?”
Better: “Do you need to change quantity purchased of a shopping cart item while on the checkout page? Why?”
The second example allows you to:
1. Know definitively that a customer needs to use that feature or not, and
2. The qualitative reason for that need. Summed over a large population of respondents, you’ll have real data about feature NEED not hypothetical future usage.
The following is a list of survey tools. Use at your own risk :)
Interviewing Live Customers
The dreaded customer interview might be the most painful thing about product management. Assuming you can find good candidates to interview (we’ll get to that), you have to get busy people’s time who probably don’t want to speak to you. Once you actually do get someone to speak to you, everything they tell you may invalidate some or all of your product.
Sounds awesome doesn’t it? Sign me up!
All joking aside, you should be excited to have a core product belief or feature invalidated by an interview. That’s the point! The benefit of performing live interviews is that you can get a level of depth and specificity that would otherwise be impossible via another method. The feedback is so good that you should interview potential customers: during ideation, during design, during build, and after build. At every point, feeding validation and invalidation data back into the development process. Weight the effort early in the lifecycle as the cost of changing features increases dramatically at each successive step.
A Rule For Interviewing: Don’t discuss the product.
Wait whaaaaat? Don’t ask about the product? Yes. Your brain is hardwired to subvert your efforts when you start discussing the product. As soon as you bring up your product idea, your brain attempts to validate that it’s good. This colors all future evaluation, rather than real interview results. From Michael Shermer’s book, The Believing Brain:
“We form beliefs for a variety of subjective, personal, emotional and psychological reasons in the context of environments created by family, friends, colleagues, culture and society at large; after forming our beliefs we then defend, justify and rationalize them with a host of intellectual explanations…Beliefs come first, explanations for beliefs follow. I call this process belief-dependent realism, where our perceptions about reality are dependent on the beliefs that we hold about it.”
The reason that this is such a problem is that your brain will naturally attempt to get answers that it wants, not answers that might be hard to for it to hear.
I can hear your question now: “ok smart guy, what do I ask about then?”
Your product attains market fit when you solve a pressing problem that no one else solves with their product. Your interview script should be 100% targeted at finding that pressing problem!
Here are some sample interview script questions for a CPQ tool (remember from surveying, no hypotheticals or guessing):
-
What is the hardest thing about quoting a deal?
-
Why is it hard?
-
Have you attempted to find any solutions to mitigate that pain?
-
If so, have you found any that work?
-
If so, why are they helping?
Obviously, the script must be tailored to your specific problem context but that question organization is repeatedly applicable.
Another Rule for Interviews: ‘Get in the Van’
Michael Sippey, ex-VP of Product at Twitter has a better article than I could write on interviewing customers but my favorite nugget from his expertise is thus: Get your customer, your UX, Sales, Support, Product, and Engineering Leads all in a “room” (today would be virtual) and allow those experts to hear the interview responses and the customers problem pain for themselves. Once you’ve heard 30 some odd customers’ pain, you likely have a good idea of what problem is pressing your target market. Sounds like a perfect time to start building a solution that doesn’t start with guessing.
Social Tools
Facebook and Twitter can be an effective place to find conversations about your product or problem space, although they often tend to be negative. Salesforce Social Studio (formerly Radian6) can be a great help identifying and analyzing trends and sentiment about your product. Don’t forget to read this post detailing how to respond to both positive and negative feedback online, if you end up choosing to engage.
Both Facebook and Twitter have the ability to create polls to query an audience but beware as the results of these polls are not necessarily scientific, allowing for self-selection.
Salesforce Communities
Salesforce.com’s Success and Partner Communities are a great place to join discussions with end users and partners who are seeing challenges and other who are offering up solutions. It’s almost like a conference that never ends. The conversations there are ripe with ideas and potential opportunity. Go join the conversation for free!
Mechanisms of Collection
Half the battle of collecting feedback is deciding how to collect it. The other half is actually finding the people from whom to collect. You must be querying your future customers; those who have the problem you are solving strongly enough to warrant paying you. Asking a bunch of college students about a middle age adult continuing education product probably isn’t helping you learn anything useful. Worse, it can add to the noise, making decisions more difficult. There’s a hidden nugget of benefit in the effort to find customers to interview. It’s a preview of finding customers to SELL to as well.
The harder it is to find customers to interview, the harder it will be to monetize the product.
Audience Services
Paid audience service companies provide a cohort of interviewees or survey responders based on demographic selection criteria that you specify. Most also provide consulting services and survey tools as well, such as Ask Your Target Market and Survey Monkey. These people are taking surveys for money but usually are not being paid enough for them to be swayed by the amount they receive. This is an easier way to get feedback and is most suited for B2C products.
Your Sales & Support Teams
These folks are on the front lines! Ask your account managers if their customers have emailed or called about a particular feature that worked or didn’t work for them. Arrange interviews or surveys to be sent out to your target demographic. Similarly, connect with your customer support agents to find out the top 10 most commonly asked questions or reported problems.
AppExchange Listing
Your AppExchange Listing can be an excellent place to reach potential customers. From the listing you can:
-
Collect leads when customers: try the app, watch a demo, install the app
-
List and publicize contact information (make sure to monitor the accounts) for your potential customers to reach out
-
Add custom survey links to the Customer Resources section
The Webinar Snare
This is the tried and true Salesforce.com model! Have you noticed that your Account Execs and Technical Evangelists are often holding webinars with partners? Can you think of why they might do this? Customer success! The webinars that Salesforce holds are ALWAYS full of extremely useful content. They are demonstrating a hard problem solution or advanced looks at pre-released products. Without extremely compelling free content, you cannot reliably get an audience to show up but the ancillary benefit to all parties is that you get to start a conversation with the attendees. You open a channel for questions and answers. The keys:
-
Don’t pitch your product during a webinar, you are providing free value.
-
Excite and delight the attendees with awesome content.
-
Be personal with contacts as you reach out for an interview request.
-
Be respectful and don’t over-communicate. You don’t want to spam anyone.
Your Salesforce ISV Account Executive is the right person to help you set this up but you don’t have to partner with Salesforce to be effective with this model. You can run it all on your own. Just remember to hit the fine points and always deliver value.
Conferences
Now that there are constant Salesforce oriented conferences beyond Dreamforce (SE Dreamin, Tahoe Dreamin, Midwest Dreamin, etc), combine those with your app’s target industry conferences and there is giant opportunity to apply some hustle, set up a booth and start having conversations. Every single conference I attend generates tons of leads and maybe more important, information on the state of the industry and people’s opinions on the direction that our industries are moving.
Scoring Feedback
Scoring feedback allows you to utilize consistent evaluation across a population of respondents, taking human biases out of the equation when considering a feature or problem to be worked. You do not have to use the recommendations below. It’s perfectly acceptable to find your own scoring methodology. Just be sure to apply it consistently across your experiments.
Interviews
As you collect interview feedback, it’s imperative that you score the feedback for impact to help decide if the results should indicate working on the item is prudent. For each interview, assign a score for each response item:
-
Is the interviewee actively trying to solve the problem? Yes: 10; Sort of: 5, No: 0
-
Was the interviewee engaged and focused throughout the interview? Yes: 10; Sort of: 5, No: 0
-
Did the interviewee agree to a follow up meeting where you’ll present your solution. Yes: 8; Sort of: 4, No: 0
-
Did the interviewee refer others to you for interviews? Yes: 4; Sort of: 2, No: 0
-
Did the interviewee offer to pay you immediately for the solution? Yes: 3; Sort of: 1, No: 0
Scores of 25 or higher are good scores, anything under is not. Every interview or set of interviews is an opportunity to experiment. Aggregate the scores across your interview population and observe the median value. If the median score is 20-25 or higher you likely have a winner! Keep experimenting!
Surveys
For surveys, you’ll need to structure the questions in such a way where you’re getting the respondent to self score on a 1-10 basis for your problem questions. Usually, getting consistent scoring >65% is needed to say you might have something worth working. If you do find a signal with surveying, follow up with live interviews to solidify your results. Remember: be specific!
Summary
The task of product managers is to create a feedback cycle of problem hypothesis and validation with the target industry resulting in a solution product roadmap, if warranted. The best product managers understand that this loop must be repeated over and over and over again until the correct problem/solution fit is found. The faster you can work through each iteration, the better chances you have at finding that magic combination that resonates with the market.
