The Dev Blog

A New Rails Plugin: acts_as_rated

Posted by Guy Naor Mon, 05 Feb 2007 03:20:00 GMT

I just released on RubyForge a new rails plugin for rating of any ActiveRecord model. The project page is at rubyforge.org/projects/acts-as-rated.

Though similar to other rating plugins, this one has a ton of options to customize, while still making it very easy to use. And most important for my use, can cache the statistics of the ratings (total/count/average) in the model itself or an external statistics table, eliminating the need to call sum/count/avg on the ratings table itself. . To install:

script/plugin install svn://rubyforge.org/var/svn/acts-as-rated/trunk/acts_as_rated

Usage example:

class Book < ActiveRecord::Base
  acts_as_rated
end

u = User.find_by_name "guy"
b = Book.find "Catch 22"
b.rate 5, u 
u = User.find_by_name "john"
b.rate 3, u

b.rating_average # => 4
Book.find_by_rating 2..3 # => [<Book:"Catch 22">]

b.find_rated_by User.find_by_name("guy") # => [<Book:"Catch 22">]

The plugin comes with a full set of migration methods to make it easy to add to any project, and it also has extensive testing included.

Features:

• Rate any model
• Optionally add fields to the rated objects to optimize speed
• Optionally add an external rating statistics table with a record for each rated model
• Can work with the added fields, external table or just using direct SQL count/avg calls
• Use any model as the rater (defaults to User)
• Limit the range of the ratings
• Average, total and number of ratings
• Find objects by ratings or rating ranges
• Find objects by rater
• Extensively tested

Enjoy!

Posted in acts_as_rated, Rails, Ruby | 8 comments

Adding Support for Functional Indexes in Rails

Posted by Guy Naor Mon, 29 Jan 2007 18:48:00 GMT

Rails migrations are a great tool, and one of the things I really love about rails. It made database changes phobia a thing of the past :-) But the migration support a pretty low common denominator as to what can be done without resorting to sending direct SQL commands.

One of the things I use all the time and really miss in migrations, are functional indexes. Postgres supports those, and it's a shame not to use it. For those not aware of what a functional index is, it's an index that is built by calling a function for the row values to index, insted of using the actual value itself. The simplest use I have for it is when I want to make names case-insensitive when searching. So that a:

select * where lower(name) = 'test'

The change works in such a way that if the database doesn't support functional indexes (as defined in the adapter in rails) it will fall back to generate the regular index. So the following statement in a migration:

add_index :users, :name, :functional => 'lower(name)'

Will create a regular index in MySQL, but will create a functional index in Postgres.

Here is the diff agains rails 1.2.1. The changes can also be transfered to other rails versions as they are pretty simple:

Index: activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb
===================================================================
--- activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb    (revision 28)
+++ activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb    (working copy)
@@ -186,14 +186,16 @@
       def add_index(table_name, column_name, options = {})
         column_names = Array(column_name)
         index_name   = index_name(table_name, :column => column_names)
+        functional   = nil

         if Hash === options # legacy support, since this param was a string
           index_type = options[:unique] ? "UNIQUE" : ""
           index_name = options[:name] || index_name
+          functional = options[:functional] if supports_functional_indexes?
         else
           index_type = options
         end
-        quoted_column_names = column_names.map { |e| quote_column_name(e) }.join(", ")
+        quoted_column_names = functional.nil? ? column_names.map { |e| quote_column_name(e) }.join(", ") : functional
         execute "CREATE #{index_type} INDEX #{quote_column_name(index_name)} ON #{table_name} (#{quoted_column_names})"
       end

Index: activerecord/lib/active_record/connection_adapters/abstract_adapter.rb
===================================================================
--- activerecord/lib/active_record/connection_adapters/abstract_adapter.rb      (revision 28)
+++ activerecord/lib/active_record/connection_adapters/abstract_adapter.rb      (working copy)
@@ -42,6 +42,12 @@
         false
       end

+      # Does this adapter support functional indexes? Backend specific, as the
+      # abstract adapter always returns +false+.
+      def supports_functional_indexes?
+        false
+      end
+
       # Does this adapter support using DISTINCT within COUNT?  This is +true+
       # for all adapters except sqlite.
       def supports_count_distinct?
Index: activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
===================================================================
--- activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb    (revision 28)
+++ activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb    (working copy)
@@ -111,6 +111,10 @@
         63
       end

+      def supports_functional_indexes?
+        true
+      end
+
       # QUOTING ==================================================

       def quote(value, column = nil)

Posted in Rails, Ruby, Postgres | no comments

How to get AjaxScaffold working with Rails 1.2

Posted by Guy Naor Sat, 20 Jan 2007 22:05:37 GMT

Rails 1.2 was just released, and I decided to see how much pain it will be to make the help application work on it. As the application has a very good testing coverage (Code/Test LOC: 1:3.1) it is a good candidate to try with, as I can easily test most of the application using the automated tests.

I found a few deprecated things which I changed, and then some other routing problems and changes based on the new Prototype. It took some work, but wasn't too bad. One thing to note, is that you need to update Prototype to the latest from rails:

rake rails:update:javascripts

If you don't, the JavaScripts helpers will generate code that isn't compatible with the older Prototype.

After getting all tests to pass, and all deprecations warning taken care of, it was time for testing the UI in the browser, and this is where the "fun" started.

Not a single call to the AJAX requests made it to the server. It was just swallowed by Prototype. Luckilly FireBug makes JavaScript debugging tolerable. Using some console.log() calls, and the console in FireBug, I found there is an exception raised inside the Ajax.Request object. Some more logging, and I realised where the problem comes from: rico_corner.js used by AjaxScaffold is not compatible with the latest Prototype. It extends all the Prototype objects, and while doing it, extends an array that Prototype uses to set headers, adding to it a function that throws off the Prototype code completely.

There's a simple fix described here. It's kind of a hack, but at least will get you going for now.

Hope this saves you a few hours of head banging...

Posted in Rails, Ruby | no comments

AjaxScasffold, Security and Deployment Problems in Rails

Posted by Guy Naor Wed, 17 Jan 2007 23:05:28 GMT

AjaxScaffold was already mentioned in my previous post, so no need to sing it's praise again...

While deploying the Famundo help to my staging server, it stopped working, not even leaving a single clue in the logs. For me this is always a sign that something isn't initializing correctly. So it was time for a small investigation.

After playing a bit with the code, I realized the problem is caused by init.rb in AjaxScaffold trying to copy it's files into the application main directories. The reason this is a problem is my desire to make the system as secure as possible. Part of that is not letting the user the application runs as, write access into the application directory. This prevents a bug or breakin from writing into the application directories, reducing the damage that can be caused. The user running the application has only read access to the application directories.

Time to fix AjaxScaffold. First of all, I don't think that in production mode those files need to be copied over. It's done in development mode, and then are there for production mode. I do think it's a nice thing for development mode as it allows easy upgrade to a new AjaxScaffold version. Second, an error like that shouldn't kill the application with no explanation.

So my fix just adds an if around the copy and skip it in production mode, and also surounds it with begin/rescue/end, logging the error if one happens.

I also opened a ticket in the AjaxScaffold bug database, and I'll try to find who to email this to. For now, just take this file and replace your init.rb with it, or just copy the changes.

NOTE: The edge code of AjaxScaffold plugin moved the file copy to install.rb, so you'll have to change that file instead.

# Include hook code here
require 'ajax_scaffold_plugin'

ActionController::Base.send(:include, AjaxScaffold)
ActionView::Base.send(:include, AjaxScaffold::Helper)

# copy all the files over to the main rails app, want to avoid .svn
# Do not copy in production mode!!! And catch errors and log them
if ENV['RAILS_ENV'] != 'production'
  begin
    source = File.join(directory,'/app/views/ajax_scaffold')
    dest = File.join(RAILS_ROOT, '/app/views/ajax_scaffold')
    FileUtils.mkdir(dest) unless File.exist?(dest)
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public')
    dest = RAILS_ROOT + '/public'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public/stylesheets')
    dest = RAILS_ROOT + '/public/stylesheets'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public/javascripts')
    dest = RAILS_ROOT + '/public/javascripts'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public/images')
    dest = RAILS_ROOT + '/public/images'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)
  rescue Exception => ex
    RAILS_DEFAULT_LOGGER.error "AjaxScaffold error while copying the AjaxScaffold files to the application directory. (#{ex.t_s})"
  end
end

Posted in Rails, Ruby, Linux | no comments

Ruby Duck Typing Goodness

Posted by Guy Naor Fri, 12 Jan 2007 22:48:00 GMT

Coming over from C++ to Ruby, duck typing was one of the really cool features I learned about. You don't use it every day, but when you need it, it's an amazing tool.

Case in point: writing the admin interface to the Famundo help app (you can currently see the public interface at Famundo's Help. (A new version is coming out shortly, and I intend to Open Source it soon.) I used AjaxScaffold for the interface, but wanted to also manage file uploads. I wanted to keep it the same interface as the rest of the tables, as it gives me a nice UI, sorting, searching, etc...

To make that work, I duck typed a class that uses the filesystem, but acts like an ActiveRecord class. I then pointed AjaxScaffold at it and as far as the user experience goes, it's just like managing a database table. Simple and intuitive.

I didn't duck type everything in ActiveRecord, just the stuff that AjaxScaffold needed. Of course, if the need comes, adding more methods is very easy.

Following is the class I created. The $STORAGE_DIR global points to where the storage dir is. Usually something under public so that it's easy to serve the files. But you can also use x-send-file or some other trick and put it someplace else.

class StoredFile 

  attr_accessor :size, :filename, :modified_time

  # Trick the id...
  alias_method :id, :filename 
  alias_method :id=, :filename=

  # All the methods we need to overwrite from active_record...
  def self.table_name
    'stored_files'
  end

  def self.primary_key
    'filename'
  end

  def filename_before_type_cast
    filename
  end

  def self.count(*args)
    options = args.last.is_a?(Hash) ? args.pop : {} # Taken from the rails source!
    fltr = options[:conditions] || ''
    Dir.entries($STORAGE_DIR).delete_if{|i| i[0..0] == '.' || (fltr.is_a?(Regexp) ? i !~ fltr : !i.downcase.include?(fltr.to_s))}.size
  end

  def self.find(*args)
    options = args.last.is_a?(Hash) ? args.pop : {} # Taken from the rails source!
    # See if it's a single file find, like ID in rails tables
    if args.first.is_a?(String)
      fname = args.first.gsub(/[^\w\.\-]/, '_')
      raise "File not found #{args.first}" if !File.exist?("#{$STORAGE_DIR}#{fname}")
      return StoredFile.init_from_file(fname)
    end

    fltr = options[:conditions] || ''
    flist = Dir.entries($STORAGE_DIR).delete_if{|i| i[0..0] == '.' || (fltr.is_a?(Regexp) ? i !~ fltr : !i.downcase.include?(fltr.to_s))}.collect{|f| StoredFile.init_from_file f }

    # This is now an unsorted list of files. Now we can sort and apply limits/offsets
    if options[:order]
      options[:order] =~ /^(.+) (asc|desc)?$/i
      reverse_ord = !$2.nil? && ($2.downcase == "desc")
      case $1
        when 'filename'     : flist.sort! {|a,b| a.filename <=> b.filename }
        when 'modified_time': flist.sort! {|a,b| a.modified_time <=> b.modified_time }
        when 'size'         : flist.sort! {|a,b| a.size <=> b.size }
      end

      flist.reverse! if reverse_ord
    end

    idx = (options[:offset] ? options[:offset] : 0).to_i
    len = (options[:limit]  ? options[:limit ] : flist.size).to_i

    flist[idx,len]
  end

  def errors
    []
  end

  def self.destroy fname
    FileUtils.rm_f "#{$STORAGE_DIR}#{fname}"
    fname
  end

  def destroy
    StoredFile.destroy self.id
  end

  protected

  def self.init_from_file f
    fstat = File::stat "#{$STORAGE_DIR}#{f}"
    sf = StoredFile.new
    sf.filename, sf.modified_time, sf.size = f, fstat.mtime, fstat.size
    sf
  end

end

Posted in Rails, Ruby | 2 comments

Improving Postgres LISTEN/NOTIFY Support in Ruby/Rails

Posted by Guy Naor Thu, 07 Dec 2006 12:00:00 GMT

The Postgres DB server has a very cool mechanism for passing messages between the backend and clients and between the clients, using a simple LISTEN and NOTIFY system.

The way it works is that any client can register to LISTEN to a named event, and whenever any other client (or the backend using triggers/functions) issues a NOTIFY command for the named event, all listening clients will be notified of it. You can find the full documentation here and here.

A usage scenario is a background process that need to act at specific times based on database changes. The process can sleep not consuming any CPU, until a notification arrives, it wakes up does it's thing, and go back to sleep. It can also be used to keep cache consistency or to manage locks between clients. And many other scenarios.

The ruby postgres driver (C based version) has some support for this using the get_notify method. But it has some limitations which I wanted to remove:

1. Notifications arrive at the client whenever they are sent, but to know they arrived you need to poll the library, in effect making it again a poll solution. I wanted to change it to a solution that uses no CPU while waiting, and also isn't dependant on polling intervals.

2. Currently you have to issue a query to know that the notification arrived. This makes the polling solution even less attractive as you have to generate some query to know if something arrived.

The Postgres library provides a function called PQconsumeInput that solves the problem of having to send a query to see if a notification arrives. Once executed, calling get_notify will return any waiting notifications (call it multiple times until a nil is returned, as each call returns one notification).

To completely remove the need to poll (even with PQconsumeInput you still need to poll) I added another function based on an example from the Postgres documentation. This functions will get the current connection socket, and using a select() (system one, not sql) wait for any new activity on the socket. Once a notification arrives, the select() will stop blocking, and we can call consume_input and then get_notify() to get any waiting notifications. Put it in a loop and go back to waiting after handling the incoming events, and you have a process that waits for notifications without using any CPU time while waiting.

Please note that in order to use those new functions in rails applications you have to use the raw_connection:

ActiveRecord::Base.connection.raw_connection.consume_input

Here is diff to the Ruby driver (you will need to recompile it). If you would like to also have the patched C file, let me know.

--- ruby-postgres-0.7.1/postgres.c   2003-01-05 17:38:20.000000000 -0800
+++ postgres.c  2006-12-04 13:04:55.000000000 -0800
@@ -439,6 +439,40 @@
     return ary;
 }

+
+static VALUE
+pgconn_consume_input(obj)
+    VALUE obj;
+{
+    PGconn *conn = get_pgconn(obj);
+    if (PQconsumeInput(conn) == 0)
+      rb_raise(rb_ePGError, PQerrorMessage(conn));
+    return Qnil;
+}
+
+
+static VALUE
+pgconn_wait_for_activity(obj)
+    VALUE obj;
+{
+    fd_set input_mask;
+    PGconn *conn = get_pgconn(obj);
+    int    sock  = PQsocket(conn);
+
+    if (sock < 0)
+         rb_raise(rb_ePGError, "Bad connection socket");
+
+    FD_ZERO(&input_mask);
+    FD_SET(sock, &input_mask);
+
+    /* Wait for something to happen on the socket */
+    if (select(sock + 1, &input_mask, NULL, NULL, NULL) < 0)
+        rb_raise(rb_ePGError, "Socket select() failed");
+
+    return Qnil;
+}
+
+
 static VALUE pg_escape_regex;
 static VALUE pg_escape_str;
 static ID    pg_gsub_bang_id;
@@ -1443,6 +1477,8 @@
     rb_define_method(rb_cPGconn, "getline", pgconn_getline, 0);
     rb_define_method(rb_cPGconn, "endcopy", pgconn_endcopy, 0);
     rb_define_method(rb_cPGconn, "notifies", pgconn_notifies, 0);
+    rb_define_method(rb_cPGconn, "consume_input", pgconn_consume_input, 0);
+    rb_define_method(rb_cPGconn, "wait_for_activity", pgconn_wait_for_activity, 0);

 #ifdef HAVE_PQSETCLIENTENCODING
     rb_define_method(rb_cPGconn, "client_encoding",pgconn_client_encoding, 0);

Posted in Rails, Ruby, Postgres | no comments

A World Time Server In One Line Of Rails

Posted by Guy Naor Thu, 09 Nov 2006 15:26:00 GMT

First, sorry for the delay in posting more on the XMPP/Jabber serie. I was a bit busy. So until I write the next installment, here's a small piece of rails coolness.

Want to know the exact time anywhere in the world? Including daylight saving taken into account? How about doing it in one line of rails code?

1. Create a new rails application: rails .

2. Install the tzinfo plugin: script/plugin install tzinfo_timezone

3. Edit app/controllers/application.rb

4. Add to it the following method (ok, it's 3 lines if we count the def and the end and not try to squeeze it into one with ; ):

def get_time
   render :text => TzinfoTimezone[params[:id]].utc_to_local(Time.now.getutc).strftime('%Y-%m-%d %H:%M:%S') rescue render :text => "ERROR - check your time zone", :status => 500
end

Now run it: script/server

Browse to: http://localhost:3000/application/get_time/Tokyo or http://localhost:3000/application/get_time/London. Check out the tzinfo plugin for the names of the supported time-zones. And you can add more to the mapping there.

You can improve performance a bit by turning sessions off completely. Do that either in the configuration or by adding session :off to the application controller class.

If you keep your computer clock accurate with ntp, you will get a pretty accurate time. The request is processed at a really high speed, so that shouldn't be a problem. And if the round trip to the server and back is quick, you will have a 1 second accuracy. Not bad for one line of code.

Posted in Rails, Ruby | no comments

Ruby and XMPP/Jabber Part 3: Adding html to the messages

Posted by Guy Naor Wed, 18 Oct 2006 15:02:00 GMT

If you used a Jabber client like GAIM or Trillian you know that it supports rich text messages. Those can be really nice to send some nicely formatted messages.

But I couldn't find much documentation on how to do it, outside the XEP/JEPs. To save you the pain of reading RFCs and XEPs/JEPs, I'll show you here how to send those nice rich messages.

The rich text is sent as an additional element of the message. The body still remains the same plain text body, and will be displayed by clients that do not support rich-text.

The type of markup that can be used is pretty limited - a small subset of regulat xhtml markup. The full details of the markup and how to use it are given in XEP-0071. But I'll go here over the important details.

To send the message we add a new element of type html with the namespace http://jabber.org/protocol/xhtml-im, and in it we put the html. The html should be the body part only. The following tags are supported: a, br, img, ul, li, ol, p, span. (Technically there are more tags that could match the namespace, but are not recommended.) The markup is styled with CSS using the style attribute of the tag. Supported styles are: background-color, color, font-family, font-size, font-style, font-weight, margin-left, margin-right, text-align, text-decoration.

Lets give it a try in irb (using the same login details from the previous post:

require 'xmpp4r/client'
include Jabber

# Login
jid = JID::new('test@yeush.com/Testing')
password = 'test'
cl = Client::new(jid)
cl.connect
cl.auth(password)

# Create a message
to = "test_with_me@yeush.com"
subject = "XMPP4R Rich-Text Test"
body = "Wow, I can do HTML now. But if you see this, your client doesn't support it"
m = Message::new(to, body).set_type(:normal).set_id('1').set_subject(subject)

# Create the html part
h = REXML::Element::new("html")
h.add_namespace('http://jabber.org/protocol/xhtml-im')

# The body part with the correct namespace
b = REXML::Element::new("body")
b.add_namespace('http://www.w3.org/1999/xhtml')

# The html itself
t = REXML::Text.new( "This is so <strong><span style='background: #003EFF; '><span style='font-size: large; '>COOL!!!</span></span></strong>. I can really do <strong>HTML</strong> now.", false, nil, true, nil, %r/.^/ )

# Add the html text to the body, and the body to the html element
b.add(t)
h.add(b)

# Add the html element to the message
m.add_element(h)

# Send it
cl.send m

It's pretty simple once you understand how to add the element to the message. Nothing fancy going on. And you can use other ways to build the markup. I'm using REXML as it's what XMPP4R considers the native elements. But assigning pre-formatted elements will also work.

Well, go send some nice messages, and come back for the next post about queries and callbacks. This will get us a bit deeper into the protocol, and get us ready for more interesting things.

UPDATE: A short explanation of the last parameter on the REXML::Text.new parameter is in order. As we are passing a pre-formatted text that include characters that are XHTML markup characters, we need to tell REXML to ignore those. We do that by passing a regular expression that will never match anything as the illegal parameter.

Posted in XMPP4R/Jabber, Ruby | no comments | no trackbacks

Ruby and XMPP/Jabber Part 2: Logging in and sending simple messages

Posted by Guy Naor Sat, 14 Oct 2006 15:34:00 GMT

Time to get down to some sample code!

In this part I will show how to log in to a Jabber server and send a simple message. This is the basis of everything else, and will give you something nice to start with. After all, sending messages is the most common use for Jabber.

Installation

Get the code from here, unpack it and then run ./setup.rb from the directory you open the archive to. YOu can also get the gem and install it locally. As it's pure ruby, there aren't any complications with it.

What we will need?

To start, get irb going as we'll do it interactively so we can get immediate results. You will also need a Jabber account you can login to, and another account to send messages to. You could send to the same account, actually, but it's nicer with a second one.

For this session, I will be using my account at DreamHost - they provide Jabber as one of the features of the account, which is pretty cool. YOu can also setup one of your own, but it's beyons the scope of this post.

Server: yeush.com

User: test

Password: test (don't worry, I'm not this crazy, this is a fake password...)

And we'll interact with another accounts (that has the test account as a buddy that can connect to): test_with_me@yeush.com.

Now in irb, make sure we have xmpp4r included:

irb
require 'xmpp4r/client'
include Jabber # Makes using it a bit easier as we don't need to prepend Jabber:: to everything

Logging in

To start any communication with a Jabber server, we need to first log in to the server:

# Jabber::debug = true # Uncomment this if you want to see what's being sent and received!
jid = JID::new('test@yeush.com/Testing')
password = 'test'
cl = Client::new(