Computational Journalism Server 1.2.6 Released

 #Blog2012, Computational Journalism, Computational Journalism Server, Data Journalism, Hiearchical Document Clustering, Linux, Linux Desktops, Semantic Visualization  2 Responses »
May 162012
 

I’ve just released Computational Journalism Server 1.2.6. There are two major updates to the functionality.

  1. I’ve added a script to upgrade the server to a desktop. The “install-lxde.bash” script installs a full LXDE desktop. You’ll have the Firefox browser, the Claws email client, the AbiWord word processor, Gnumeric spreadsheet, the Leafpad text editor, the LXTerminal terminal emulator, ePDFViewer and a graphical file manager.
  2. Given a desktop install, I’ve added scripts to download and install the prototype Overview tool for semantic visualization and hierarchical clustering of large document sets. I’ve been experimenting with this for three days now and I think it belongs in every computational journalist’s tool set.

I wrote a bit about Overview two days ago. The Overview team has an ambitious road map and I’m pretty confident their approach will help working journalists make sense of the volumes of text available. The tool as documented on the Overview web site runs on Windows and Macintosh personal computers with no modifications. In addition to the ability to install Overview in a desktop-enhanced Computational Journalism Server, I’ve provided scripts to install it on any openSUSE 12.1, Fedora 17 or Ubuntu 12.02 desktop. With minor tweaks it should run on older versions or other Linux distributions.

Here’s a sneak peek at the documentation for the new features, derived from https://github.com/znmeb/Computational-Journalism-Server/blob/master/Overview/README.md

Running Overview on the Computational Journalism Server

What’s Overview?

Overview is a tool for semantic visualization and hierarchical clustering of large document sets. Jonathan Stray of the Associated Press leads the development, with funding provided by a Knight Foundation grant.

The main project page is at http://overview.ap.org/. It’s an open source project and its repositories are on Github at https://github.com/overview. And they have a Twitter account: @overviewproject.

If you want to run the prototype of Overview on a Windows or Macintosh personal computer, the instructions are here: Getting Started with the Overview Prototype. If you want to run Overview on the Computational Journalism Server or a Linux Desktop, read on!

Running the Overview Prototype on the Computational Journalism Server

  1. You’ll need to download and install the Computational Journalism Server first. I recommend doing the “install-all.bash” full install rather than just installing the base appliance.
  2. Next, you’ll need to install the LXDE desktop. As “root”, do# cd /opt/Computational-Journalism-Server
    # ./install-lxde.bash

    When the script asks if you want to trust the repository, answer “a” for “always”.

  3. After the LXDE desktop repositories, patterns and packages are installed, you’ll be sent to a YaST2 session to change the default run level. Enter “Expert Mode”. Tab to the “Set default runlevel after booting to:” field and select “5: Full multiuser with network and display manager”. Then tab to “OK” and press “Enter”.
  4. Reboot and log in on the console as the non-root useryou created when you installed the appliance. Select “LXDE” in the “Desktop” pulldown menu. You should be in the LXDE desktop.The “install-lxde.bash” script installs a full LXDE desktop. You’ll have the Firefox browser, the Claws email client, the AbiWord word processor, Gnumeric spreadsheet, the Leafpad text editor, the LXTerminal terminal emulator, ePDFViewer and a graphical file manager.
  5. Open an “LX Terminal”. The menu button is in the lower left. Start the menu and select “System Tools -> LXTerminal”.
  6. In the terminal, type$ cd /opt/Computational-Journalism-Server
    $ cp -a Overview/ ~

    This creates a copy in your home directory where you can write.

  7. Type$ cd ~/Overview
    $ ./install-overview-openSUSE.bash

    This will take quite a bit of time to download and recompile the required Ruby 1.9.3.

  8. Type$ ./test-overview.bash

    This will run all the test cases. The “caracas” example takes quite a bit of time in the Ruby NLP step, but the others run fairly quickly on my 8 GB dual-core laptop.

Running the Overview Prototype on Linux Desktop / Laptop

If you already have a Linux desktop, do the following. I’ve tested this on openSUSE 12.1, Ubuntu 12.04 “Precise Pangolin” and Fedora 17 “Beefy Miracle”. It can probably be made to work on older Fedora or Ubuntu desktops with a little tweaking, but I’m not testing it on them. It will get tested on openSUSE 12.2 when the beta comes out.

  1. Install “git”.
  2. In some convenient directory where you have write access, type$ git clone http://github.com/znmeb/Computational-Journalism-Server
    $ cd Computational-Journalism-Server/Overview
    $ ./install-overview-<DISTRO>.bash

    where <DISTRO> is Fedora, Ubuntu or openSUSE. Do not run this as “root” – use an ordinary user account!

  3. That will take some time; as noted above, one of the steps is to download and recompile Ruby 1.9.3 from source. When it’s done, type “./test-overview.bash” as above.

Computational Journalism Server 1.2.5 Released – Now With Optional Apache™ Hadoop™

 #Blog2012, Computational Journalism, Computational Journalism Server, Data Journalism, Data Journalism Developer Studio 2012LX, Data Science, Hadoop, Platform as a Service  1 Response »
May 102012
 

Version 1.2.5 Released

I’ve just pushed all the buttons to release Computational Journalism Server 1.2.5. It’s mostly bug fixes and miscellaneous cleanup changes, but there is one new major option: Apache™ Hadoop™. Right now, all that’s there is a script to download and install the latest stable Hadoop from Apache and run the single-node test script. But it should be enough for developers to start testing the R Hadoop interface routines ‘HadoopStreaming‘ and ‘hive‘. See Parallel R for some sample code using ‘HadoopStreaming’ and ‘hive’.

To install Hadoop, do the following:

  1. Log into the server as “root”.
  2. Type “cd ~/Computational-Journalism-Server/Hadoop”.
  3. Type “./install-hadoop.bash”.
  4. Type “./test-hadoop.bash”.

This should install Hadoop and run the single-user test. For more information on configuring larger-scale Hadoop clusters, see the main documentation page at http://hadoop.apache.org/common/docs/r1.0.2/. The scripts in this release came from the Single Node Setup page.

Road Map

As a few recent posts on this blog have noted, I’m planning to migrate the platform components of Computational Journalism Server to either CloudFreeStyle or OpenShift or both, to take advantage of their existing platform-level components and community support structures. I don’t have a good estimate of dates yet, but there will be at least one more release as an openSUSE appliance before there’s any OpenShift or CloudFreeStyle release. There will also be one more release of Data Journalism Developer Studio 2012LX to catch up to the openSUSE Build Service packages.

First Impressions of Ubuntu 12.04 LTS “Precise Pangolin”

 #Blog2012, Computational Journalism, Linux  7 Responses »
May 052012
 

First of all, let me put this in perspective. I’ve been using Linux on workstations and laptops since Red Hat Linux 6.2. I stayed with Red Hat all the way through Red Hat Linux 9. When Red Hat split the distribution into Red Hat Enterprise Linux and Fedora Core in 2003, I switched to Debian. I ran Debian for about six months, then switched to Gentoo Linux. In the summer of 2008, I switched to openSUSE Linux and I’ve been on openSUSE since then.

Every time one of the major community Linux distributions ships a new stable release, I try it out. So far, none of the Debian, Fedora, Ubuntu or Mint releases has come out significantly better than openSUSE, so I’ve stuck with it. And that remains true for Ubuntu 12.04 LTS “Precise Pangolin”. If that were the end of the story, I could close this blog post now. But it’s not.

If you’ve been following this blog and my Twitter stream and Github account, you’ll know that I’ve been collecting tools for computational journalism and packaging them as appliances. And I’m moving on towards a Platform as a Service. One of the requirements I’ve put on that is that the tools should be distribution-agnostic as much as possible. Up to now, everything has been on openSUSE because of the SUSE Studio appliance construction tools and to a lesser extent the openSUSE Build Service package repositories. But I’ve come to the point where I need to make things work on Fedora and Ubuntu.

So I’ve quad-booted my laptop (Windows, openSUSE, Fedora 16 and Ubuntu 12.04). And I’m trying to triple-boot my workstation with openSUSE, Fedora and Ubuntu. Which brings us to the first problem – openSUSE and Fedora installed cleanly on the workstation, but Ubuntu 12.04 didn’t. In particular, the Ubuntu desktop doesn’t even come up on a 1024×768 monitor!

I can understand Linux not coming up on a wireless card that’s relatively new. I can understand Linux having trouble with a touchpad or with audio. After all, the hardware makers design for Windows and Apple, not Linux desktops / laptops. But a 1024×768 monitor that’s run everything from Gentoo / WindowMaker to KDE 3.5 to KDE 4 to GNOME 2 and GNOME 3 and LXDE and Cinnamon on openSUSE? A 1024×768 monitor that runs Fedora 16 without any problems? That’s just plain wrong!

I did get the Ubuntu desktop working on the laptop, which is a much newer configuration. I’m not going to spend a great deal of time on how ugly the desktop actually is when it works. That’s been covered in numerous places and desktops are

  1. A matter of personal taste, and
  2. Customized to the user’s workflow.

But for someone who, like me, is used to the GNOME 2 desktop as delivered in previous versions of Ubuntu and Fedora, the openSUSE customization of GNOME 2 and the current clean implementations of GNOME 3 on openSUSE and Fedora, Ubuntu’s Unity desktop is jarring. And it’s really hard to figure out how to do things, where stuff is, and so on.

Moreover, the whole distribution is “pushy” – it’s hawking subscriptions to Ubuntu One cloud music, for example. The software installer has favorite apps, and so on. It’s like having a Kindle Fire or an iPad or visiting the Chrome Web Store or Google Play – the Ubuntu desktop is trying to sell you something every time you move your mouse. Ubuntu has turned the Linux desktop into just another media consumption device!

That’s two strikes – annoyances but not deal-breakers. But what I want to do with Fedora and Ubuntu is use them as hosts for virtual appliances, just like I use openSUSE and Windows / VirtualBox now.. In openSUSE and Fedora, I can go into the software installer and select a “pattern” and get everything I need to do that. If Ubuntu has that, it’s well hidden under the games and the productivity suites and the media apps. Sure, I can go find how to do that on Ubuntu on the web, but it seems to be going against the grain of the distribution. It only took me two minutes to find it on Fedora after almost four years of working daily on openSUSE!

I’m sure “Precise Pangolin” is a fine distribution “under the hood.” The previous long-term support version, 10.04, is an acknowledged workhorse in servers along with Debian, RHEL/CentOS/Scientific Linux and SLES. I have to test on it, and I’ll figure out how to be productive at it. But if Canonical can’t come up with a desktop built for Linux professionals like me, they’re going to lose us.

Computational Journalism Server – The Way Forward

 #Blog2012, Computational Journalism, Computational Journalism Server, Data Journalism, Linux, Platform as a Service  6 Responses »
May 022012
 

As I’ve noted here, the Computational Journalism Server “wants to be a Platform-as-a-Service (PaaS) when it grows up.” In plotting the way forward to that goal, I’ve looked at three options:

  1. Remain on openSUSE / SUSE Studio and collect other open source tools to provide the additional services that would make the current server into a true PaaS.
  2. Start with the Cloud Foundry or one of its derivatives and add the computational journalism tools.
  3. Start with the Red Hat OpenShift Origin PaaS and add the computational journalism tools.

Remain on openSUSE

If I remain on openSUSE, as noted above, I’d need to collect more tools to provide additional services. Many of these deal with the underlying infrastructure. My target infrastructure is OpenStack Essex. That’s still the target. Moreover, the overall goal is still to provide an R-language tool set for dealing with large-scale computational journalism problems using library packages in the Comprehensive R Archive Network that implement parallel, cluster and grid computing.

When I started the original Data Journalism Developer Studio project, the Platform as a Service concept was in its infancy. It has matured rapidly, though. Cloud Foundry and Red Hat OpenShift are both about a year old, and several derivatives of Cloud Foundry have already appeared.

Cloud Foundry runs on Ubuntu Linux and OpenShift on Red Hat / Fedora. There isn’t an equivalent packaged solution for openSUSE. I’d have to build that for the Computational Journalism Server to be a true PaaS. And that seems to me a diversion from the mission.

Cloud Foundry

Cloud Foundry is an open source project from VMware. Derivative projects include AppFog for PHP projects, PaaS.io for Haskell, Stackato for Perl and Python and a community fork called CloudFreeStyle. Of these, neither AppFog nor PaaS.io are relevant, since there’s little PHP or Haskell in the current or planned use cases for the Computational Journalism Server. So the options are Cloud Foundry itself, Stackato or CloudFreeStyle.

I’ve ruled out the base Cloud Foundry for two main reasons.

  1. Most of the frameworks, services and tools provided by Cloud Foundry are totally unfamiliar to me. I know enough Ruby to do simple scripting and enough Java to call class libraries, but I know virtually nothing about Ruby on Rails, and I know absolutely nothing about Spring, Scala, RabbitMQ or Eclipse. I’d have a steep learning curve on tools that aren’t relevant to the core of the Computational Journalism Server – R, SQL and NoSQL databases, Hadoop and to a lesser extent Perl and Python.
  2. For an open source project with a year’s history under its belt, the documentation is, in a word, abysmal. In particular, the tasks I need to accomplish to make Computational Journalism Server into a PaaS – primarily adding R parallel programming capabilities and application packages – are totally undocumented.

I’m a member of the CloudFreeStyle project. I joined because I wanted to learn how to do what’s in Cloud Foundry and how to enhance it. Because it’s a source-level fork of Cloud Foundry, it would be easy to add functionality and ignore the components I don’t need, at least in the beginning. The “glue logic” to talk to applications and to the cloud infrastructure is already there and should “just work”. But, like its parent, there’s little documentation and I’d have to figure things out from the source.

Finally, there’s Stackato. Stackato is a very impressive product and ActiveState’s documentation, support and tool set is world-class. I’ve been a happy ActiveState Perl Development Kit user for years. If the Computational Journalism Server was a commercial product / business venture rather than an open source project, I’d go with Stackato. But the Computational Journalism Server isn’t there yet and may never be.

OpenShift Origin

OpenShift Origin, released on April 30, 2012, is an open source PaaS construction platform from Red Hat. I’ve spent about a day and a half browsing the documentation and I’m blown away by how comprehensive it is, especially for someone like me who wants to build a tool set from the ground up. The OpenShift Origin documentation is every bit as awesome as Cloud Foundry’s is abysmal.

The demos include a number of “LAMP stack oldies but goodies” – MediaWiki, WordPress, and Drupal. There’s also an OpenShift Origin LiveCD, based on Fedora 16, that turns any 64-bit Intel / AMD workstation, laptop or virtual machine into an OpenShift PaaS. With a few additional steps you can install OpenShift Origin permanently on a real or virtual machine.

The Way Forward

At the moment, I’m keeping three options open:

  1. Remain on openSUSE,
  2. CloudFreeStyle, and
  3. OpenShift Origin.

But I suspect the strength of the documentation will pull the project towards OpenShift Origin sooner rather than later. “Watch this space,” as the saying goes.

Computational Journalism Server 1.0.0 Released

 Computational Journalism, Data Journalism, Platform as a Service  1 Response »
Apr 272012
 

Computational Journalism Server: SUSE Gallery Download Page

Computational Journalism Server: Github Project

Data Journalism Developer Studio Users Google Group

Computational Journalism Studio 1.0.0 is now available in the SUSE Studio Gallery at http://j.mp.compjournoserver. It’s not exactly a full Platform-as-a-Service yet, which is the eventual goal, but just about all the components I want in the appliance are there. I’ve put fairly detailed instructions on getting started on the SUSE Gallery download page and they’re duplicated on the Github project page. If you run into difficulties, please feel free to comment here, send me a tweet, post an issue on Github, make a comment on the appliance download page, or rant on the Google Group. No carrier pigeons, please – I’ll just cook them for dinner.

I’ve got fairly big plans for the evolution of this tool set. In rough calendar order but with no firm dates yet, they are:

  • Get the major use cases described in Parallel R working,
  • Get the R integration with the ATLAS libraries working by default,
  • Get the appliance working as an lxc Linux Containers guest,
  • Package the appliance so it can function as an OpenStack Essex compute node,
  • Port the workstation development / testing environment to Ubuntu 12.04 and Fedora 17, and
  • Port the server to CloudFreeStyle and OpenShift Platform-as-a-Service environments.