The Dev Blog: Category Postgres

Adding Support for Functional Indexes in Rails

guy.naor@famundo.com (Guy Naor) — Mon, 29 Jan 2007 13:48:00 -0500

Rails migrations are a great tool, and one of the things I really love about rails. It made database changes phobia a thing of the past :-) But the migration support a pretty low common denominator as to what can be done without resorting to sending direct SQL commands.

One of the things I use all the time and really miss in migrations, are functional indexes. Postgres supports those, and it's a shame not to use it. For those not aware of what a functional index is, it's an index that is built by calling a function for the row values to index, insted of using the actual value itself. The simplest use I have for it is when I want to make names case-insensitive when searching. So that a:

select * where lower(name) = 'test'

can actually use the index efficiently.

The change works in such a way that if the database doesn't support functional indexes (as defined in the adapter in rails) it will fall back to generate the regular index. So the following statement in a migration:

add_index :users, :name, :functional => 'lower(name)'

Will create a regular index in MySQL, but will create a functional index in Postgres.

Here is the diff agains rails 1.2.1. The changes can also be transfered to other rails versions as they are pretty simple:

Index: activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb
===================================================================
--- activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb    (revision 28)
+++ activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb    (working copy)
@@ -186,14 +186,16 @@
       def add_index(table_name, column_name, options = {})
         column_names = Array(column_name)
         index_name   = index_name(table_name, :column => column_names)
+        functional   = nil

         if Hash === options # legacy support, since this param was a string
           index_type = options[:unique] ? "UNIQUE" : ""
           index_name = options[:name] || index_name
+          functional = options[:functional] if supports_functional_indexes?
         else
           index_type = options
         end
-        quoted_column_names = column_names.map { |e| quote_column_name(e) }.join(", ")
+        quoted_column_names = functional.nil? ? column_names.map { |e| quote_column_name(e) }.join(", ") : functional
         execute "CREATE #{index_type} INDEX #{quote_column_name(index_name)} ON #{table_name} (#{quoted_column_names})"
       end

Index: activerecord/lib/active_record/connection_adapters/abstract_adapter.rb
===================================================================
--- activerecord/lib/active_record/connection_adapters/abstract_adapter.rb      (revision 28)
+++ activerecord/lib/active_record/connection_adapters/abstract_adapter.rb      (working copy)
@@ -42,6 +42,12 @@
         false
       end

+      # Does this adapter support functional indexes? Backend specific, as the
+      # abstract adapter always returns +false+.
+      def supports_functional_indexes?
+        false
+      end
+
       # Does this adapter support using DISTINCT within COUNT?  This is +true+
       # for all adapters except sqlite.
       def supports_count_distinct?
Index: activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
===================================================================
--- activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb    (revision 28)
+++ activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb    (working copy)
@@ -111,6 +111,10 @@
         63
       end

+      def supports_functional_indexes?
+        true
+      end
+
       # QUOTING ==================================================

       def quote(value, column = nil)

Improving Postgres LISTEN/NOTIFY Support in Ruby/Rails

guy.naor@famundo.com (Guy Naor) — Thu, 07 Dec 2006 07:00:00 -0500

The Postgres DB server has a very cool mechanism for passing messages between the backend and clients and between the clients, using a simple LISTEN and NOTIFY system.

The way it works is that any client can register to LISTEN to a named event, and whenever any other client (or the backend using triggers/functions) issues a NOTIFY command for the named event, all listening clients will be notified of it. You can find the full documentation here and here.

A usage scenario is a background process that need to act at specific times based on database changes. The process can sleep not consuming any CPU, until a notification arrives, it wakes up does it's thing, and go back to sleep. It can also be used to keep cache consistency or to manage locks between clients. And many other scenarios.

The ruby postgres driver (C based version) has some support for this using the get_notify method. But it has some limitations which I wanted to remove:

Notifications arrive at the client whenever they are sent, but to know they arrived you need to poll the library, in effect making it again a poll solution. I wanted to change it to a solution that uses no CPU while waiting, and also isn't dependant on polling intervals.
Currently you have to issue a query to know that the notification arrived. This makes the polling solution even less attractive as you have to generate some query to know if something arrived.

The Postgres library provides a function called PQconsumeInput that solves the problem of having to send a query to see if a notification arrives. Once executed, calling get_notify will return any waiting notifications (call it multiple times until a nil is returned, as each call returns one notification).

To completely remove the need to poll (even with PQconsumeInput you still need to poll) I added another function based on an example from the Postgres documentation. This functions will get the current connection socket, and using a select() (system one, not sql) wait for any new activity on the socket. Once a notification arrives, the select() will stop blocking, and we can call consume_input and then get_notify() to get any waiting notifications. Put it in a loop and go back to waiting after handling the incoming events, and you have a process that waits for notifications without using any CPU time while waiting.

Please note that in order to use those new functions in rails applications you have to use the raw_connection:

ActiveRecord::Base.connection.raw_connection.consume_input

Here is diff to the Ruby driver (you will need to recompile it). If you would like to also have the patched C file, let me know.

--- ruby-postgres-0.7.1/postgres.c   2003-01-05 17:38:20.000000000 -0800
+++ postgres.c  2006-12-04 13:04:55.000000000 -0800
@@ -439,6 +439,40 @@
     return ary;
 }

+
+static VALUE
+pgconn_consume_input(obj)
+    VALUE obj;
+{
+    PGconn *conn = get_pgconn(obj);
+    if (PQconsumeInput(conn) == 0)
+      rb_raise(rb_ePGError, PQerrorMessage(conn));
+    return Qnil;
+}
+
+
+static VALUE
+pgconn_wait_for_activity(obj)
+    VALUE obj;
+{
+    fd_set input_mask;
+    PGconn *conn = get_pgconn(obj);
+    int    sock  = PQsocket(conn);
+
+    if (sock < 0)
+         rb_raise(rb_ePGError, "Bad connection socket");
+
+    FD_ZERO(&input_mask);
+    FD_SET(sock, &input_mask);
+
+    /* Wait for something to happen on the socket */
+    if (select(sock + 1, &input_mask, NULL, NULL, NULL) < 0)
+        rb_raise(rb_ePGError, "Socket select() failed");
+
+    return Qnil;
+}
+
+
 static VALUE pg_escape_regex;
 static VALUE pg_escape_str;
 static ID    pg_gsub_bang_id;
@@ -1443,6 +1477,8 @@
     rb_define_method(rb_cPGconn, "getline", pgconn_getline, 0);
     rb_define_method(rb_cPGconn, "endcopy", pgconn_endcopy, 0);
     rb_define_method(rb_cPGconn, "notifies", pgconn_notifies, 0);
+    rb_define_method(rb_cPGconn, "consume_input", pgconn_consume_input, 0);
+    rb_define_method(rb_cPGconn, "wait_for_activity", pgconn_wait_for_activity, 0);

 #ifdef HAVE_PQSETCLIENTENCODING
     rb_define_method(rb_cPGconn, "client_encoding",pgconn_client_encoding, 0);

Fixing Rails for Postgres Schemas

guy.naor@famundo.com (Guy Naor) — Sat, 03 Jun 2006 18:26:00 -0400

Postgres has a very powerful feature in the schemas, especially with the schema search_path. Using it I can have multiple separate families, all using the same database connection, but also fully separated (with a different user to access the schema files). Using the search path and the SESSION AUTHORIZATION it is completely transparent to rails. I'll have a later post explaining how to use this technique in a rails application.

The cool thing about the search_path is that you can have multiple instances of the same table in different schemas, and using SET SESSION AUTHORIZATION, you can see a different table each time. Rails works great with this.

But there's a small bug in the postgres driver for handling sequences. (The bug doesn't show up in development mode because the objects are reloaded all the time.) When postgres loads the sequence name used for the auto increment keys (serial type in postgres), it adds the schema to the sequence name. So instead of messages_id_seq, it will return public.messages_id_seq. If another request coming in, tries to insert a record, the call to SELECT currval() will return the wrong sequence, and if (like I do) there's user security between schemas, you get an access error and the insert fails.

The fix is very simple - just return the unqualified sequence name, as the rest of the access in rails is unqualified.

I'm posting a ticket with the fix and a test file for it. The fix to the rails edge is the following (in diff format):

Index: lib/active_record/connection_adapters/postgresql_adapter.rb
===================================================================
--- lib/active_record/connection_adapters/postgresql_adapter.rb (revision 4414)
+++ lib/active_record/connection_adapters/postgresql_adapter.rb (working copy)
@@ -296,8 +296,9 @@
               AND def.adsrc ~* 'nextval'
           end_sql
         end
-        # check for existence of . in sequence name as in public.foo_sequence.  if it does not exist, join the current namespace
-        result.last['.'] ? [result.first, result.last] : [result.first, "#{result[1]}.#{result[2]}"]
+        # check for existence of . in sequence name as in public.foo_sequence.  if it does not exist, return unqualified sequence
+        # We cannot qualify unqualified sequences, as rails doesn't qualify any table access, using the search path
+        [result.first, result.last]
       rescue
         nil
       end

And So It Starts

guy.naor@famundo.com (Guy Naor) — Sat, 28 Jan 2006 15:44:00 -0500

It wasn't too long ago that my main development language was C++ on Windows. But when I decided I want to do web development, this just plain didn't make sense. Enter Ruby and Rails!

I was on a search for a good web development environment, and looked at many of the options available, but none seemed to meet my requirements. Java was too heavy (I did do Java development in the past, but that was long ago), I didn't like the then available Python stacks and I really dislike all the .NET stuff. Call it too many years on Windows, or a dislike to proprietary systems, but I just won't touch it.

While searching, I stumbled upon RubyOnRails. I was weary of all the promises I read about it, but playing with it a bit proved me wrong. What a refreshing surprise!!!

Then came the need to do a 180 degree turn in my head. From the strict mindset of C++ to the "crazy" way Ruby and the other dynamic languages do things. I had no prior Ruby experience, so it required lots of learning. Luckily I had lots of experience with Linux and all related technologies, so this part was taken care of.

A few weeks after starting learning, I switched my dev environment from Visual Studio on Windows to Eclipse on Fedora Core 4 (past experience with RedHat made it my choice), and I can't be happier :-).

Some dev choices I have made so far:

OS - Linux FC4
IDE - Eclipse with RadRails
Source control - Subversion
Database - PostgreSQL

Come visit this blog for some development ideas and solutions related to my new development journey. And for details as I take an amazing new application online.