The master repository is on Github at http://github.com/znmeb/TweetCSV. As far as I know, Contacts.pl will run on any modern version of Perl, but I’ve only tested it on Perl 5.10 on openSUSE Linux 11.2, and with ActiveState ActivePerl on Windows. If you have any trouble running it, please feel free to send me a tweet @znmeb.

How does it work? TweetCSV.pl makes Twitter API calls using the Perl Net::Twitter module from CPAN. You get the results in a comma-separated-value (CSV) file, which you can then open in a spreadsheet. Instructions for downloading and running the script are at http://github.com/znmeb/TweetCSV/blob/master/README.

Again, please feel free to send me a tweet if you need help getting this running. And special thanks to Marc Mims (@semifor), who has developed the Net::Twitter Perl module that interfaces with the Twitter API!

A number of speakers are from the Portland area, and I’ve met most of them. The list of Portland-based speakers includes

• Carri Bugbee
• Kim Brater
• Bryan Rhoads
• Barry Tallis

The complete list of speakers and schedule is here.

Social Fresh Portland is just the first of a long list of social media and technical conferences happening here in Portland in 2010. In the next few weeks, as the schedules start to appear on line, I’ll be posting invitations. Meanwhile, here’s a little bit about why Portland is such a great place to visit.

The thing is, I don’t actually live in Portland, but in a suburb called Aloha that, strangely enough, has nothing to do with Hawaii. And I’ve never been to Hawaii, so I can’t very well ask you to visit there, can I? So, yes, definitely visit Portland!

I’ve been here for almost 25 years. What do we have?

Water:

Two major rivers meet here, and fresh drinking water literally falls out of the sky free for the taking! If you like it salty, there’s a few bays and coves a couple of hours to the West.

Air:

We get our air mostly fresh off the ocean, or occasionally funneled through the Columbia Gorge by a high-pressure cell. In any event, we get our air before much of the US does, and we try our damnedest not to add stuff to it on its way East.

Mountains:

Yeah, there’s one not too far away that gave us a little trouble in 1980, but for the most part, they’re pretty to look at and a great place to go hiking or skiing or just hanging out in the lodge.

Parks:

There are so many, I can’t list them all, so I’ll just give you a link to my favorite. Tryon Creek State Park. If Nature holds true to form, the trilliums will be in full bloom there!

Beer:

Contrary to popular belief, you can get imported beer here. But why would you? Ours has better hops, has more alcohol, is served in pubs, restaurants, banquet halls and even movie theaters! According to Wikipedia, “In 2008, Portland had 30 microbreweries located within the city limits, more than any city in the world and greater than one-third of the state total. With 46 microbrew outlets, Portland has more breweries and brewpubs per capita than any other city in the United States. Many have won nationwide and international acclaim.”

Food and wine:

We grow it. We catch it in the ocean. We make it. We cook it. We eat it. We package it up and ship it. And we love to share it. Our food cart scene has been featured on national television and in the New York Times.

Entertainment:

New York has Greenwich Village. Washington has Georgetown. Portland has Portland! Jazz, folk, rock, symphonic, chamber, ballet, opera, and two new music ensembles. Portland has numerous theater companies and a major performing arts center. We have a listener-supported classical radio station that’s heard around the world on the Internet. Oh, yeah – if you happen to hear bagpipes, they just might be coming from a unicyclist.

Bloggers and Tweeters and Geeks, Oh! My!:

I’m a blogger. This is my blog. I had five others once. And I have a LinkedIn page. And a Facebook page. And I tweet. A lot – at last count more than any other Portlander.

Geeks: we have Linus Torvalds. Perhaps you’ve heard of Linux? He invented it. We have Ward Cunningham. Perhaps you’ve heard of the Wiki? He invented it. We have major contributors to Perl, PostgreSQL, Ruby, WordPress and other open source projects.We have Jive Software and Zapproved. We have the Silicon Florist. We have 30 Hour Day. We have Strange Love Live.

We love social media, software and (wait for it) social media software! Software is a craft here, just like belts, jewelry and beer. You can actually sit and watch us make it in coffee shops and pubs.

So if you’re looking for a great city to visit this year, we’re here! Just be careful crossing the street if you hear bagpipes.

Ruby

As you may know, Twitter was originally a Ruby on Rails application. That’s actually where I first heard of Twitter – at RubyConf 2006. Early in 2007, I joined Twitter, and my first friends and followers were people I had met at RubyConf 2006.

As you’ll see below, Twitter has now incorporated many other technologies, but they still use Rails, and Ruby. In particular, the version of Ruby they use is Ruby Enterprise Edition (REE), a version tuned for stability and scalability.

Scala

Scala “is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages, enabling Java and other programmers to be more productive. Code sizes are typically reduced by a factor of two to three when compared to an equivalent Java application.”

It’s not just Twitter that’s using Scala; Foursquare also uses it. Because of the high visibility of Twitter and Foursquare, I expect to hear a lot more about Scala in the coming months. For those of you in the Portland, Oregon area, there is now a Scala programmers’ group, @PDXScala.

Cassandra

Cassandra is one of the newer “Non-SQL” databases. It was originally developed at Facebook, and released as open source in 2008. The description on the Cassandra home page reads, “A highly scalable, eventually consistent, distributed, structured key-value store.” The key term (pun intended) here is “key-value store”. This is somewhat like what they used to call in the ancient days (1950s) “associative memory”. Rather than specify an object (the value) by its location, we give it a name (the key) and the system can find it.

Here’s an interview with Ryan King, Twitter’s Director of Storage, on why Twitter chose Cassandra.

Hadoop and Pig

Hadoop is another highly-scalable distributed tool. Hadoop primarily implements the MapReduce operation. MapReduce is a way of applying massive processing power to massive datasets. The concepts behind MapReduce originated in the early 1960s or even earlier, during the development of the Lisp programming language. An implementation of MapReduce has been patented by Google.

Pig is a “scripting language” designed to work with Hadoop. It simplifies the programming tasks of people working with large datasets.

Summary

While the technologies are interesting to technologists like me, what does such massive power give the Twitter user? And why does Twitter need it? Here’s a simple example: The exact arrival rates of tweets at Twitter aren’t widely publicized. I don’t know if this is a “trade secret” or not – I’ve seen estimates of these rates on blogs but I’m not sure that the way those estimates were obtained is technically valid.

However, there is a publicly-available subset of the full “Firehose” data stream available via the Streaming API, called  “Sample”. There’s no official documentation on what fraction of the full Firehose comes through the Sample stream. But in a sample I collected in January, I saw a peak  of 81,718 tweets in a single hour!

And what was so special about that hour? It was, to be precise, the hour between 01:00:00 and 02:00:00 UTC January 13th, 2010. The Haiti earthquake happened at 21:53:10 UTC on January 12th, 2010 – about three hours earlier. Remember – “Sample”, as the name implies, is a subset of the full tweet stream! That’s the reason Twitter needs the massive power it is getting from these cutting-edge technologies.

Update!

Todd Hoff of Highscalability.com has just published a more detailed analysis of Twitter’s use of Hadoop and Pig, including links to a presentation by Kevin Weil, Analytics Lead at Twitter. Both are highly recommended!

Books on the Technologies

Hadoop: The Definitive Guide
by Tom White

Hadoop in Action
by Chuck Lam

Pro Hadoop
by Jason Venner

Programming in Scala
by Martin Odersky

Programming Scala: Tackle Multi-Core Complexity on the Java Virtual Machine
by Venkat Subramaniam

Programming Scala
by Dean Wampler

Beginning Scala
by David Pollak