About these ads

Filtering FriendFeed – How Crowdsourcing Can Solve This


It would be nice to have filters on FriendFeed. For instance, it would be nice to be able to hide any post containing the word “Obama” without having to hide someone’s other stuff. Or the ability to hide any entry containing the word “ubuntu”, etc.

Thomas Hawk, FriendFeed direct post, May 1, 2008

The need for filters on FriendFeed is a recurring topic. Click here to see the numerous entries that contain the words ‘friendfeed’ and ‘filters’. Louis Gray notes the need for this in a recent post.

I want present an idea for filters that has two pieces:

  • Category filters
  • Keyword filters

The two pieces are interrelated, and crowdsourcing will be used to build out the category filters.

Let’s get to it, shall we?

Category Filters

FriendFeed already has a “Feed Preferences” page for each member. Here is where you can manage your category and keyword filters. The graphic below is mockup of this:

A. Category Filters

Various categories will be displayed, along with a link to the full list of categories. In the example, above, I say that I’d like to filter out all FriendFeed entries that relate to politics.

The value of category filtering is that it prevents you from having to manage every keyword that might relate to a category. In a recent post, I noted Dave Winer’s 38 different politics-related terms. For instance, he used the terms: Hillary, HRC, Clinton, Edwards, Obama, Rove, etc. Having the ability to automatically filter those out without having to set up keyword hides over and over would be a great benefit to many members. Particularly as FriendFeed gains traction with a flood of new members.

Now how would FriendFeed know that Hillary, Obama, HRC, etc. are part of the politics category? Keep reading.

B. Keyword Filters

Members will need the ability to see what words they have hidden. They can un-hide keywords, or add new keywords to hide directly on the Feed Preferences UI.

Keyword-Based Hides

FriendFeed currently supports hiding specific entries, plus entries from specific members and services. For instance, you can hide all Twitter updates. What is lacking is the ability to filter out entries with specific terms in them.

For instance, shown below are three tweets from Dave Winer regarding politics:

What I’d like to do is apply the Hide function to anything with ‘Harold Ickes’ or ‘Henry Waxman’. This is a mock up of that screen below:

A. Full Text of Entry Displays

The full text of the entry appears. Each word of the entry includes a link. The links are easy ways for members to populate the ‘hide terms’ input box.

B. Hide Terms Input Box

Commas separate each term.

C. Categorize the Terms to Be Hidden

As the member hides the terms, they will be asked to apply a category. The most popular categories previously applied to the keywords will be displayed. Or the member can type a category into the input box, and FriendFeed will auto-suggest different categories with each character entered.

Why do this? This is the basis of the crowdsourced solution.

Let the People Decide

People will have a much better handle on the categories that apply to a keyword than will a heavy-duty algorithm. Such human filtering is the basis of tagging.

Two elements are relevant here:

  • The need to prevent bad categories being assigned to keywords
  • The motivation to do this categorizing

Use Bayesian Stats to Prevent Bad Categories

Here’s the issue you want to avoid. Some prankster assigns the football category to the term “Paris Hilton” while hiding all entries containing her name. Suddenly, members who are filtering out football entries stop seeing their Paris Hilton updates (yes I know, horrors…).

Enter Bayesian statistics. Carl Bialik, a columnist for the Wall Street Journal, has a great column on the use of Bayesian stats for online ratings. The gist of this approach is that all items in a rating system are born with identical ratings. Their ratings only change as people vote, and it takes a sufficient number of votes to really move the rating of an item. Here’s an example of this from the WSJ column:

For instance, as noted in the column, IMDB.com doesn’t use straight averages to list the top 250 movies of all time, as voted on by its users. Instead, each movie starts out with 1,300 votes and a ranking of 6.7, which is the site’s average. That helps smooth the effects of a few intense votes; it takes a lot of votes to budge the IMDB meter up or down from 6.7

That same approach would be applied inside FriendFeed. It would take a large number of people putting a keyword into the same category before the keyword actually became “part” of that category.

Once a keyword graduates to a category, any users filtering that category won’t see entries with those keywords.

Motivation

Why would anyone bother to categorize the keywords they hide? One answer – not everyone will. But there are two drivers of members doing some keyword categorization.

First, members need to recognize that they are contributing to a system from which they are likely benefiting. If you filter any category, you will be benefiting from the work of others’ who have categorized keywords.

Second, the categorization experience has to be simple and fast. You’ve got the member right there, motivated to hide a term. Make it easy for them to channel that motivation into a simple categorization. The most popular previous categories are displayed, making it easy check them. And the auto-suggest feature can be done fairly quickly. I like how Faviki is doing it:

Faviki draws from thousands of different Wikipedia entries for this list.

Final Thoughts

One thing to consider here is that every entry coming into FriendFeed would need to be filtered for keywords. Serious processing power will be required. Fortunately, the FriendFeed guys have firsthand experience with high volumes of real-time queries for keywords at Google.

With regard to this proposal, I haven’t (yet) seen anything on the market that will provide the category tags that would help filter FriendFeed. Since it’s the members who are most in tune with what they want to filter, their common sense and motivation should be leveraged.

As FriendFeed grows, imagine new members easily managing the flow of information by simply filtering the politics category rather than having to set up an extensive list of new keywords. It would make the experience that much better for everyone.

*****

See this item on FriendFeed: http://friendfeed.com/search?q=%22Filtering+FriendFeed+-+How+Crowdsourcing+Can+Solve+This%22&public=1

About these ads

About Hutch Carpenter
Senior Consultant for HYPE Innovation (hypeinnovation.com)

15 Responses to Filtering FriendFeed – How Crowdsourcing Can Solve This

  1. Karim says:

    I like it, I’m all about this too!

  2. excellent post Hutch – very thought out and something like this could be a big plus for cleaning up some of the noise.

  3. Louis Gray says:

    Great job, Hutch. You already knew that I would agree and be a big proponent. Of all the services I’ve used, I trust this team the most to get this right.

  4. Susan Beebe says:

    Excellent post on filtering!! and really great mockups and matching feature explanations! This would be awesome!! Thx!

  5. @Karim – I just signed up for Feedego. I will check it out.

  6. @Steven – thanks. I wondered if the post would read like a lite PRD, but it looks like folks read through it.

  7. @Louis – I agree. I think the FriendFeed guys have a good sense of what makes users love the service.

  8. @Susan – thanks! Graphics = fun with Paint!

  9. Robin Cannon says:

    Good outline there. Thanks in particular for the explanation on the Bayesian statistics, which covered by itself one of the major concerns I’d have for this kind of self policing.

  10. Mark Dykeman says:

    I’m a bit embarrassed to admit that the mock-ups looked so good that I was scrutinizing the FriendFeed page to try to find them. Yet another example of people scanning rather than reading. :S

  11. Pingback: Beach Bicycles » Blog Archive » Filtering FriendFeed - How Crowdsourcing Can Solve This

  12. Scott Goldie says:

    My site – http://www.friendfeedmachine.com/ – has keyword filtering for the stream view. Nothing as cool or elaborate as you propose though ;)

  13. @Robin – yeah, the Bayesian stats approach is useful for solving the problem of too few observations. You need to accumulate a bunch of observations to move the needle.

  14. @Mark – thanks Mark. Just taking Kevin Fox’s work and running through MS Paint.

  15. @Scott – your app is great for managing the FriendFeed experience. I think applying this stuff deep into FriendFeed itself is important because it will apply to everyone using the site.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 667 other followers

%d bloggers like this: