Tweets Offer Little Value in Understanding Customer Needs
June 4, 2012 10 Comments
https://twitter.com/rwilcox/status/203838810809835520
Social media. A rich source of insight and opportunity for companies. Why, it’s an article of faith that customers are talking…on their terms…where they want. So get out there and learn from them! See what this IBM executive says:
Companies that embrace social media as a source of insight will be rewarded. They’ll develop a deeper understanding of customer needs and will be able to attract new customers more easily. They’ll have a better shot at providing the products and services that the market wants – before the competition does.
It’s true, there are opportunities. But really, my sense is the opportunities lie on the marketing (read: sell) side of the house. Now social media occupies a large landscape. So how about narrowing focus, to Twitter. How useful are people’s tweets for customer insight? Specifically, giving companies a better handle on customers’ jobs-to-be-done?
tl;dr answer: Not so much. Lots of grousing about personal circumstances, and some silliness. But not much insight.
Saving and paying for college, in 450 tweets
Source: Mark J. Perry
To examine this question, I assumed the perspective of a financial firm trying to get a better handle on the “paying for college” jobs-to-be-done. In aspirational, emotional terms, college continues to be a top goal of parents for their children, and of teenagers as well. In economic terms, colleges have an insatiable appetite for tuition increases.
Importance for our children, increasing costs, need to save. Surely, there are some unfulfilled jobs-to-be-done out there. So I turned to Twitter to see what people were talking about. What needs were they expressing? What insight on this topic?
To see what people are saying, I ran Twitter searches on three terms:
- college savings
- “saving for college”
- “pay for college”
I collected 150 tweets each for those three Twitter search terms (you can see them in this Google spreadsheet). With that data, I looked at (i) how many contained usable insight, (ii) what were the common words, and (iii) the collective sentiment.
Now, my research here is one of…oh say…a quadrillion possible customer insight areas to explore. Conclusions I draw here are not necessarily applicable to all areas. But it’s a good start.
Mining for jobs-to-be-done
Imagine you work for a large financial services company. You know in the U.S., people have put nearly $150 billion into 529 college savings plans. It’s a potent cocktail: lots of money; aspirations for, and symbolism of, a college degree; and escalating tuition. There have got to be opportunities to improve people’s lives here!
You want jobs-to-be-done, defined as people’s expressions of things they’re trying to get done in relation to saving and paying for college. Just what are they tweeting about out there?
You collect the tweets, and then categorize each tweet according to its value in understanding jobs-to-be-done:
- No value
- Points toward a job-to-be-done (a shadow of a job)
- Direct expression of a job-to-be-done
In reviewing 450 separate tweets, here are the results:
| Twitter search term | No value | Points to a JTBD | Direct JTBD |
| college savings | 137 | 12 | 1 |
| “saving for college” | 141 | 9 | 0 |
| “pay for college” | 150 | 0 | 0 |
| Totals | 428 | 21 | 1 |
Looking at those, you begin to understand the challenge of looking at tweets as sources of customer insight.
Tweets with no value
By far, the most prevalent case was that the tweets provide no value in understanding what jobs people are trying to get done. At least, no actionable value. Below are examples of these types of tweets:
Many tweets in the “no value” category were of this type. Honest, authentic? You bet. Fuel for developing new innovations in products and services? Not so much. I’ll admit that value may be in the eye of the beholder. Perhaps someone wants to target scholarship-winning kids to buy new cars. But for the financial services firm, looking for insight from customers, these tweets don’t help.
Note that many of the tweets in this category weren’t from real people. They were marketing tweets by institutions. Which makes it hard to just dig in to those tweets. First you need to separate the manufactured marketing tweets from the honest expressions of individuals. Welcome to the jungle.
Tweets pointing to a job-to-be-done
I’ll admit, this categorization sounds funny: “pointing toward” a job. What does that mean? Look at a couple examples:
The tweets are not themselves the jobs-to-be-done. But they reflect jobs, if you read them carefully. In the first tweet, Raeleen wants to contribute to the college fund of her nephew. The larger story is that family, beyond the kid’s parents, can be part of the college savings effort. Here’s a definition of the job-to-be-done:
Situation: When I am planning my child’s college savings…
Job: I want participation from family members and friends.
Success means: Increased savings from a broad cross-section of family and friends.
The job reflected in the second tweet is one of structurally managing the college savings apart from other savings and cash expenditures.
There weren’t a lot of these, and many of them were pretty obvious jobs-to-be-done. But in hunting for the elusive job-to-be-done in the wild, their gamey, tough meat was better than nothing.
Direct expressions of jobs-to-be-done
After going through 450 tweets, I was sure I’d find at least one person tweeting a job-to-be-done. Maybe dazed by collecting and analyzing that many, I settled on the one below:
I see here an emotional job-to-be-done. Namely, the feeling of accomplishment that one feels in preparing financially for college. That’s a feeling that should be built on. I can tell you, when my wife and I put money into our kids’ 529 plans, we get that sense of accomplishment. Something that a financial institution should plug into more strongly.
So that was a nice job-to-be-done. However, as I said, slim pickings in finding jobs-to-be-done in tweets. Which should make you question how much of the presumed insight waiting to be gathered out there is actually…there.
But I did run the tweets through a couple other analyses to see what they turn up.
Sentiment analysis
For all these tweets about saving and paying for college, how was the sentiment? Does the mood tell us something about the jobs-to-be-done? Or at least provide some form of insight?
I ran them through a couple different tools: Sentiment Analyzer and Sentiment140. Sentiment Analyzer analyzed the 450 tweets I had collected. Sentiment140 ran an analysis on the most recent 19-25 tweets it could find related to a search term. While the bases of tweets differed, the two engines came up with remarkably similar results. Sentiment140 is on top, Sentiment Analyzer is on the bottom of the graphic below:
See how the college savings tweets are strongly positive. The general idea of college savings is a positive one, as college is a very strong emotional element for us. We may have attended ourselves, with memories from that period, and we want our kids to go. Then see how the mood swings to a darker one when tweets contain the phrase “pay for college”. It’s as if the more ‘transactional’ the experience becomes, the more negative the mood gets. Another difference is that tweets about “pay for college” overwhelmingly were by teenagers and young adults, expressing frustration related to affording college.
For marketing purposes at least, this sentiment analysis may have value in targeting people in different stages of the college saving spectrum.
Word trends
Another area explored is the commonality of words in the tweets used. Do they reveal latent jobs-to-be-done? I ran the tweets through Wordle:
The Wordles do reveal some interesting patterns. When people tweet about college savings, the most common words are plan, account, day, get. In this case, day is driven by the timing of when I collected the tweets. May 29th was upcoming, and it was U.S. national 529 Day (get it? 5/29). The most frequent word plan certainly makes sense, and in some ways fits the positive sentiment seen earlier. Thinking ahead about what’s needed to pay for college.
Notice the most frequent terms switch to start, money, need, parents in the “saving for college” Wordle. Let’s focus on the transition from plan seen in the college savings Wordle to start in the “saving for college” Wordle. It’s a transition from a more abstract concept to a task that one must undertake. The vibe switches to a get-things-done mentality.
Finally, note what dominates the “pay for college” Wordle: money, gonna, help, stripper. You might look at that and say, “stripper”? First, note that I converted the various stems of strip in the tweets to a single word, stripper, so as to better capture what showed up in a lot of the tweets. Essentially, a number of people (female and male) joking about becoming strippers to pay for school. I read these tweets as reflecting the steep costs of college, and teenagers/young adults not feeling like they have options to pay for it. A reflection of escalating tuition and cases where college savings were not available.
Usable insight on customer needs?
The title of this post highlights a particular aspect of insight that interests me: customer needs. On that score, I don’t find the tweets to be that valuable. As someone who is outside the financial services industry, I do find them to be gauges of what’s going on out there. An industry insider might already have a handle on what I discovered.
Here’s what makes tweets challenging to use for insight on customer needs:
- Wide range to subjects: tweets run the gamut, even for a specific search term, and their volume makes it tough to sift through to find areas you want to explore
- Odds are low that they’re speaking about core things they are looking to get done
- People tweeting are not part of the market you’re seeking: for example, although there is $150 billion saved in U.S. 529 plans, these parents weren’t tweeting about saving and paying for college
- Tweets have a one-off quality: follow-up and discussion to get deeper insights doesn’t happen
That’s my take. What do you think? Wouldn’t surprise me if others are seeing more value than I am. If you’re interested, you can check out the tweets I used for analysis on this Google spreadsheet.
I’m @bhc3 on Twitter.
About Hutch Carpenter
I'm Not Actually a Geek 
Incredibly timely post, Hutch! I was just having a conversation around Jobs a few days ago, and we started discussing how jobs can be named or summarized. My exact comment was, “You can’t describe a job in 140 characters or less!”
The problem that arises when we look at these concise statements (whether we’re trying to summarize a job that we know a lot about (pack), or we’re trying to learn about a possible job by evaluating the statement (unpack)) is that we lose all of the rich content that is useful to us as product developers.
For example, we had a clear-cut job in the building industry that could be named, “The Downsizer.” It’s nice for us to be able to talk about the job by referring to it by name, but it’s useless as a product development tool.
The rich detail packed into the “Downsizer Job” included things like the fact that the the people moving needed a family room large enough to house their existing dining table (which had become the bank account of every memory of their family growing up – every birthday, every thanksgiving). This contradictory level of detail exists in almost all of the rich stories that were used to identify and build the job to be done (among others that were equally important), but it would be impossible to summarize down to a sentence or phrase.
For that reason, it’s impossible to get down to a level where you can say things like:
– The job of the iPod is to take my music with me when I’m on the go.
– The job of Facebook is to keep me connected with my friends and family.
What is lost in these statements is all of the richness that describes what other things you considered, and what attributes of each product or solution were important to you in your specific situation that made the product the right choice.
HOWEVER
I think that Twitter can be incredibly useful to Jobs-based product development. During the early phases of JTBD research, we always immerse ourselves in whatever topic we’re trying to understand. To have a huge wealth of comments on any given industry is very convenient and always acts as a good starting point for us. It’s important to note that we don’t draw any conclusions from it, but we can at least start to understand people’s language, and the words that they use to describe the space.
Thanks Chris. Yeah, my sense is that it’s tough to get too much across in single tweet, all by itself, before you tweet about Justin Bieber or something next.
I am curious about your take on this quote, attributed to Kenichi Ohmae (McKinsey strategist who developed something called the 3C’s model):
“Personally, I would much rather talk with three homemakers for two hours each on their feelings about, say, washing machines than conduct a 1,000 person survey on the same topic. I get much better insight and perspective on what customers are really looking for.”
I know with Re-Wired, the conversation with individuals is key. I think that’s quite important.
Aside from tweets, do you think there’s any role for online discussions amongst interested customers in revealing jobs-to-be-done? Those sort of conversations strike me as good ways to get deeper. It wouldn’t have to be just peer-to-peer. Moderators get in and work the discussion as well.
But it’s asynchronous, and you won’t see facial expressions, gestures, body position, or be able to dive more deeply in real-time.
In that scenario, does social media offer value?
Good question. The only real answer I can give is: I don’t know.
As you already know from being [pretty] closely connected to Bob and I, we’re always looking for faster, cheaper, better ways to get to the JTBD for a given space/product.
We’ve been looking into different ways to use social media or moderated online discussions (e.g. Communispace.com) to get to the jobs, but haven’t experimented much yet.
There are a lot of established players in the web/social research space, and they’re getting more and more sophisticated and accurate when it comes to measuring things like the target market’s response to the brand and to new products. I’ve had quite a few conversations with these people about trying to do some JTBD research using their technology, and many of them have expressed interest.
The crux up to this point is:
A.) I can’t gamble with client money (e.g. give it a shot with a real project and potentially come up with nothing).
B.) Most of these offerings are pretty costly (upwards of $20k+ for a study + my/our time), so if I start funding a lot of tests myself I’ll blow through quite a bit of cash pretty quickly.
In addition to not having access to facial expressions, gestures, etc, the asynchronous aspect of the technology presents an interesting “mindset” issue. In the face-to-face/phone/video interviews that we conduct we use techniques that help the interviewee to “put themselves back in the time/place when they bought and consumed.” It’s nothing too tricky, just asking a lot of questions about the situation (who were you with, what time of day was it, was it sunny out?). After doing this for a minute you can “feel” the participant start to light up as their memory starts to connect all kinds of details. Once you get them there, you can usually get the whole story of shopping/purchase/consumption/value out of them.
If you don’t work through the initial questions to get them into the right mindset, you typically end up getting very short answers to your questions that lack the detail that you need.
I admittedly haven’t put a lot of active thought into it up to this point, but I haven’t figured out a way to do that in an asynchronous manner.
If anyone reading this has suggestions or wants to try anything, I’d love to talk!
p.s. I totally agree with Mr. Ohmae.
I’ve seen more value in LinkedIn forums or Quora for insights compared to Twitter. Searching terms on twitter has more often led me to interesting blog posts about my topic rather than containing valuable insights within the 140 characters.
John – I think there’s something to arenas of discourse being better for getting insight than tweets. Twitter is less discourse, more expression of a thought. I love Twitter for that purpose. But it’s not a place for gaining a lot of insight on customer needs. Actually, not convinced Facebook is either.
The problem you’re trying to solve isn’t solvable with Twitter. It’s like listening to the buzz of mosquitoes to decide whether you need a new barbeque — it’s the wrong tool for the job.
To gain real insight into a job to be done, you actually need to have an open-ended conversation with the right sorts of prospective customers, or get out there and observe behaviors. You can’t know or test for what problems people are trying to solve based on sentiment analysis or scanning for “hot issues”. Twitter is like a worldwide stream of consciousness, and so much of it is driven by news or events and reactions to those things. The likelihood that I would tweet randomly about why I’m saving for college, or who I’m saving for, or why it’s difficult, or the problems I need solved to ensure that I have enough — next to nil. It’s far more likely I’d do that in a blog, but even then, probably only if that’s an area of interest that I specialize in, or if I was reacting to my favorite 529 plan implementing changes to benefits or structure, so the sample bias is likely to destroy the insight you’d get about the overall market.
I think the best you can do from a tool like Twitter is get clues that point you in the direction of what to look for. But you’ll still have to go out and confirm your hypotheses with real prospective customers.
There are probably a few niche markets where you could collect valuable insights from Twitter. For example, if your target market was people who make use of Twitter for marketing. Their domain of interest and need is close enough to what they’re doing that you would probably see evidence of it in their tweets, and it would be sizable enough to rise above simply being noise.
There’s a reason why behavioral anthropologists go out and watch what people do. It’s far more trustworthy than what they say, and it gives a better picture of relative importance (how much time is spent doing something, and why it’s being done). I’m afraid there aren’t a lot of shortcuts out there.
I think Chris mostly has the right notions on this, although I would suggest that paying a research company to do this for you is probably the wrong approach. First hand knowledge is required, and unless they have the domain expertise and are going to be the ones building the product, putting that filter in the middle is both costly and likely to lose critical information.
Thanks Paul, great thoughts there. I like what you say about tweets potentially pointing you in the direction of what to look for. In fact, that was what I found with some of the tweets. They “pointed” toward a job-to-be-done. But they’re not a silver bullet of EUREKA! They are parts of a larger puzzle that a company would have to piece together.
I also agree that open-ended conversations are needed. I fully agree that in-person is quite valuable. But that’s probably a less frequent activity. I’m of the belief that online venues can provide valuable insight as places for discourse. Not the depth of in-person (immediate follow-up on a thought, visibility into body language), but for the cost and effort involved, valuable in their own right.
Yes, agreed. It often isn’t possible to do in person, but the closer you can simulate that experience the better. Especially if you’re talking with a user that you already know, you can get 80-90% of what you need just talking. Still, I find those puzzled face tweaks, posture changes, where they look when they’re about to do something and other visual cues add a lot to the discovery process, and people just tell you more when you’re sitting in the same room.
I find tweets are good for zeitgeist, almost no matter what you’re looking for, or to look for confirmation evidence after you think you’re on the right track. Also good for hygiene factors — people like to complain about what bugs them on Twitter. So, they’re useful, just not for figuring out the job your product wants to get hired for.
p.s. I enjoy reading your posts — you are one of a very few practitioners I read who seems to grok this pretty deeply, and not just recite the incantation “job to be done” and expect a magical poof of inspiration to appear.
Great thought provoking post. Love it. Keep up the good work Hutch
Thanks muchly Graeme.