/me@(voip.)?|www./ SamLown.com

Recursive Lambdas and How to completely flatten a Hash in Ruby

Today I’ve been toiling with a complex problem in Ruby; how to completely flatten a hash of hashes and arrays into a single flat array. My aim was to be able to create a unique and platform consistent MD5 string of a complex hash. You cannot guarantee a Hash will always be ordered the same way across platforms, but a flat array can be sorted, converted to a string, and checksummed easily.

A common approach would be to monkey patch the Array and Hash methods with recursive methods that iterate through their entries, flattening along the way. Unless absolutely necessary this doesn’t sound like a good idea, even less so for a one-off problem that’s unlikely to be used elsewhere.

Having searched around for a while and only finding samples for factorial solutions and other examples of recursive lambdas I finally worked out the following solution:

flatten =
  lambda {|r|
    (recurse = lambda {|v|
      if v.is_a?(Hash)
        v.to_a.map{|v| recurse.call(v)}.flatten
      elsif v.is_a?(Array)
        v.flatten.map{|v| recurse.call(v)}
      else
        v.to_s
      end
    }).call(r)
  }
# Flatten a Hash:
flatten.call({:a => {:b => :c => 12345}}})
#      produces: ['a', 'b', 'c', '12345']

This will work in both Ruby 1.8.7 and 1.9.2. Combining two lambdas is required in Ruby 1.8 as it does not support using the variable the lambda is assigned to, unless contained inside another lambda. (Took a while to grasp that.) Ruby 1.9 does not suffer this setback so if backwards compatibility is not a problem, the following is little bit clearer:

flatten =
  lambda {|r|
    if v.is_a?(Hash)
      v.to_a.map{|v| flatten.call(v)}.flatten
    elsif v.is_a?(Array)
      v.flatten.map{|v| flatten.call(v)}
    else
      v.to_s
    end
  }

In case you’re interested, this code is used in couchrest_model to be able to include checksums in CouchDB design documents in an attempt to make checking for updates and changes in views a little bit easier.

Planetaki powered by MongoDB

Over the last year now we’ve been working on developing a new version of Planetaki and finally the fruits of our labour are in production, and wow is it fast! Its an awesome feeling to know that your ideas actually work as intended.

We started working on Planetaki way back in September 2007. At that time, there weren’t many options for alternatives to the traditional databases, and if there were, I certainly didn’t know enough about them to use them on a new project. The obvious choice was to put everything in MySQL and hope for the best. As Planetaki grew it became clear that a traditional database wasn’t going to make the mark. Relational databases are optimised for handling lots of small rows of data in tables, Planetaki caches huge amounts of data from posts all over the internet resulting in very large rows. The result is that the traditional database caching and optimisation techniques just aren’t as effective, and Planetaki was getting progressively slower and harder to manage (you might have noticed if you are a frequent user!)

Welcome MongoDB. MongoDB is a document database, as opposed to a relational database. Data is stored literally as you send it, as opposed to being stored in tables. This removes the restriction of having to fit all you data into columns allowing much greater flexibility and the ability to store more complex structures. The cost of this is a larger database, but given the ever dropping cost of storage, this is not really a problem. Some people call MongoDB a NoSQL database, personally I think this is a crap name as it says nothing about what it is, only what it is not.

Planetaki now uses a hybrid solution for storing data. The complex feeds and their entries are stored in MongoDB, and user data is stored in a traditional Postgresql database. This allows us to split the system into two parts, on one side we have the feed “reaping”, and on the other the website which mainly accesses the tables of data. To ensure we can scale, a queueing system (RabbitMQ) is used so that when a new entry is created, each user’s planet data is updated as soon as the system can allocate CPU time to do so. Eventually this will allows us to add multiple servers and support incoming update requests (coming soon!). You can see me babbling about this on the Vostok blog post: Coming up: a new version of Planetaki

I think the results speak for themselves (fortunately!) and I encourage you to check out Planetaki if you are not already a user. In the upcoming weeks I hope to talk about a bit more of the more technical features, in particular the move to AJAX.

Inheriting views in CouchRest ExtendedDocument

I recently decided to upgrade autofiscal to use the awesome CouchDB document based storage engine. The change from a traditional RDMS (ActiveRecord) to this new storage engine (CouchRest) has been pretty straight forward but there was still one tiny issue I hadn’t got round to resolving until now.

The scenario is of Sale and Purchase classes which inherit from the Invoice class. Methods and properties are all inherited correctly, but the views use internal logic based on the class name. As such, defining a view in the Invoice class will only perform lookups on documents whose couchrest-type is “Invoice”. To get around this issue we need to define the views at the moment the Invoice class is inherited:

class Invoice < CouchRest::ExtendedDocument
  def self.inherited(subclass)
    super
    subclass.class_eval do
      view_by :client
    end
  end
end

Using code like this, self.to_s when used inside the view_by will return the name of the class using the view.

Translate Columns updated

Small update regarding translate columns which I thought I’d mention. The latest version available on github now finally has some proper testing and much improved handling of model validations.

Some of the code has been re-factored and now uses the unusual alias_method_chain method to override the standard saving process in a Rails-standard way. A quick example:

def save_with_translation(perform_validation = true)
  if perform_validation && valid? || !perform_validation  
    # Do something
    save_without_translation(false)
  else
    false
  end
end

translate_columns includes something similar to the method above for saving translations, in order for it to be used before a normal save, we need to add it to the save method chain:

alias_method_chain :save, :translation

This will automatically create an alias to the old save method as save_without_translation and alias save to save_with_translation. Which is nice. Took me a while to discover this, eventually I found it on the Ruby on Rails forum which discusses how save_without_validation does its magic.

Welcome to the new SamLown.com... again

Its been a little over a two years since the last time I decided I didn’t like the way my website worked, and I’ve just gone and done it again.

One reason is that I’d fallen out with the design, the old was too “fussy” and I didn’t feel was easy to read. The new site is ultra-minimalist in contrast and allows for a few more subtle changes in the future.

The key reason for the change however is behind the scenes. Two years is a long time in computing and I’ve found myself learning lots of new and better techniques for development, including streamlining layouts with Blueprint and haml, keeping code as DRY as possible, RESTful controllers, learning how to really take advantage of jQuery, not including Javascript inside the views and controllers, using JSON whenever possible,.. the list goes on. I’ve also developed a pet hate for “admin” interfaces, those dreadfully ugly parts of a website designed to bombard the user with options and confuse as much as possible (I’m sure working with Vostok has had no small part in this.)

All this has lead me to develop my latest hobby project that I’m distributing as Open Source: Just Documents. I’ll blog about it in detail at a later date, but for the moment suffice to mention that its essentially a content management system that only understands documents. Everything is a document, a page is single document and a blog is document composed of documents. There is no “admin” section, and when logged in as an editor you can quickly edit and publish without leaving the page you’re looking at. Styling is handled through standard Ruby on Rails views, with a little addition for handling themes, so it can be easily used as a rapid development base for a new project.

I hope to work on Just Documents and this site during the next few months, and welcome anyone to leave their thoughts or try it out.

Ruby: Handling fixed string lengths and UTF-8

Until Ruby 1.9 is supported by your favorite plugins and gems, proper UTF-8 support in your ruby code is a serious challenge. If, like me, you don’t have time to wait, the following really simple code snippet should help you to at least fix the sprintf problem of fixing the lengths of strings with unicode characters.


  # UTF-8 safe operations for setting a string width
  # Aligns to left by default and pads with spaces. Options include:
  #
  #   :align => :left (default), :right
  #   :pad => ' '
  #
  def string_to_width(string, width, options = {})
    options.reverse_merge!(
      :pad => ' ', :align => :left
    )
    new_string = ""
    i = 0
    string.each_char do |c|
      break if i == width
      new_string += c
      i += 1
    end
    # add padding
    if new_string.jlength < width
      padding = (options[:pad] * (width - new_string.jlength))
      new_string = options[:align] == :left ?
        new_string + padding : padding + new_string
    end
    new_string
  end

I place that in my application_helper.rb and ensure require 'jcode' is included in the Rails environment.rb file.

Migrating to acts_as_taggable_on from acts_as_taggable_on_steroids

The Acts As Taggable On Steroids plugin for Rails has certainly been around the block a few times, I indeed have been using it in many of my projects for a few years. Now the time has come however where the lack of named_scope support is more than a minor irritation, and its time to move on: Acts As Taggable On is a fine replacement with all the named scope magic and additional context support which might be useful in some situations.

Out of the box, the new plugin is not table compatible with the old, so if you have an older project you need to upgrade without loosing the old taggings we need to perform a migration. tkwong mentioned a similar process, but there seems to be the solution wasn’t very clear to me, so I wrote my own.

Firstly, you need to grab the plugin and move the old one out the way:

script/plugin install git://github.com/mbleigh/acts-as-taggable-on.git
mv vendor/plugins/acts_as_taggable_on_steroids /tmp

Then we add a new empty migration to handle the table changes:

script/generate migration acts_as_taggable_on_migration

The contents of the migration should look like the following:

class ActsAsTaggableOnMigration < ActiveRecord::Migration
  def self.up
    remove_column :tags, :reference_count
    rename_column :taggings, :created_on, :created_at
    add_column    :taggings, :tagger_id, :integer
    add_column    :taggings, :tagger_type, :string
    add_column    :taggings, :context, :string

    remove_index  :tags, :name
    remove_index  :taggings, [:taggable_id,:taggable_type]

    add_index     :taggings, [:taggable_id, :taggable_type, :context]

    Tag.connection.execute("UPDATE taggings SET context = 'tags'")
  end

  def self.down
  end
end

I’ve been lazy and not included the reverse migration to save a bit of space. Note the last bit of manual SQL code, this will ensure all of the old tagging associations are defined in the :tags context.

All the old models using the old acts_as_taggable call can be modified to the following:

acts_as_taggable_on :tags

And thats it! You can now fully enjoy the benefits of named scopes:

Artifact.tagged_with('rosas', :on => :tags).paginate :page => 1, :per_page => 10

Fantastic!

Update: Waiting for an iMac, with style... the wave

I could help share but share this minor extension to the story as a tribute to our long suffering doorman. I’d told him the night before about the camera so he wouldn’t be alarmed, but I still found the following quite amusing.

About to turn the light off with a smile.

A quick wave.

And the door is closed.

I don’t expect there are many places in the world where this happens, but then, this is Madrid!

Waiting for an iMac, with style

Few things are as taxing as waiting for an iMac to arrive. It doesn’t help when your girlfriend drags you away for the weekend to go to a wedding… “I have friday off, so you can work from my mother’s house”, “yes, but I’m expecting the iMac on Friday… ugh”. What is a modern man to do in such situations?

Set up a video surveillance system using a web cam, a 2m USB 2.0 extension lead plugged into the home multimedia Ubuntu system, with motion detection, of course!

Thats the doorman showing in the UPS delivery guy this morning with the iMac and no doubt commenting on how much of a paranoid freak the resident is to have set up a video camera!

Now its just a question of working out how to cope with the rest of the weekend ahead knowing that the box is sitting there in the darkness just waiting to be opened, not to mention the spanish wedding.

Sadly, they left the box just outside the view of the camera, and the lighting is rather poor so I can only take solace with the fact that any suspicious activity will be immediately sent to my server.

Nokia's iPhone beater... Is that a pen?

The Register have shared the news of Nokia’s iPhone beater, here’s a picture which pretty much sums up what to expect:

Yes, that is a plastic tipped pen.

I like Nokia hardware, modest, enduring, and quite reliable, but including a pen clearly shows that they haven’t quite grasped the touch screen concept or indeed usability. No doubt the OS takes full advantage of tiny virtual keyboards and frustrating menus also seen in the Nokia N800 and most of the mobile phone range.

I think the last time I enjoyed the Symbian OS was when its great-great-granddad was around in the Psion Organiser II in the early 90s, but then, BASIC for me was its major feature :-)