Twitter Search Changes - What Do They Mean?

As you probably recall, Twitter released the Streaming API into production on January 5, 2010. My in-depth analysis is here. Six days later, Twitter made the following announcement:

Search API: High-Volume and Repeated Queries Should Migrate to Streaming API

What does this mean? Well, if you’re a vendor of Twitter monitoring tools that depend on Twitter Search for your input data, you should be working on this now. Why?

As I note in my analysis of the streaming API, there is an extra relevance and ranking filter between the raw tweets coming into Twitter and the tweets that get indexed for Twitter Search. As a result of this extra filter, there are more tweets available to users of the Streaming API than there are to users of the Search API.

I haven’t found any details on the current relevance and ranking filters, where they are going, or how fast they’re planning to get where they’re going. But the message is clear:

“This transition begins a fundamental shift towards a high value, high result quality, lower query volume Search API.” And in a thread on the Twitter API Developers’ Google Group, the author of that announcement, John Kalucki, added, “Both Search and Streaming discard all statuses from low-quality users. Search additionally filters the remaining statuses for relevance and ranking purposes. This may be hard to see now, unless you cross-reference the Streaming results, but this divergence will soon accelerate and become more obvious.”

I’m already seeing some evidence of this on the Twitter Developers’ Google Group. Specifically, see this thread. The message to us developers is, “Get serious with the Streaming API.” But what about end users?

Certainly, if you’re using monitoring tools that depend on Twitter Search, you should be asking the vendors about this. But if you’re just using Twitter Search for your own purposes, I think this is very good news. Personally, I recommend using Advanced Twitter Search at http://search.twitter.com/advanced. Again, from the announcement:

“Shifting the heaviest users away from Search should dramatically improve the overall Search experience. Resources can be allocated to the search architecture’s strength: historical, complex and high value queries.”