The Dev Blog

New Theme and Typo Version

Posted by Guy Naor Tue, 23 Jan 2007 08:21:00 GMT

Time for a new theme, and to update the blog to the latest typo.

With this new theme, I introduce the new Famundo look to the blog. This follows the transition of the help systems to the new look. The rest of the sites will follow soon.

I also upgraded to the latest stable Typo. And the latest Rails.

Enjoy!

Posted in Rails | no comments

How to get AjaxScaffold working with Rails 1.2

Posted by Guy Naor Sat, 20 Jan 2007 22:05:37 GMT

Rails 1.2 was just released, and I decided to see how much pain it will be to make the help application work on it. As the application has a very good testing coverage (Code/Test LOC: 1:3.1) it is a good candidate to try with, as I can easily test most of the application using the automated tests.

I found a few deprecated things which I changed, and then some other routing problems and changes based on the new Prototype. It took some work, but wasn't too bad. One thing to note, is that you need to update Prototype to the latest from rails:

rake rails:update:javascripts

If you don't, the JavaScripts helpers will generate code that isn't compatible with the older Prototype.

After getting all tests to pass, and all deprecations warning taken care of, it was time for testing the UI in the browser, and this is where the "fun" started.

Not a single call to the AJAX requests made it to the server. It was just swallowed by Prototype. Luckilly FireBug makes JavaScript debugging tolerable. Using some console.log() calls, and the console in FireBug, I found there is an exception raised inside the Ajax.Request object. Some more logging, and I realised where the problem comes from: rico_corner.js used by AjaxScaffold is not compatible with the latest Prototype. It extends all the Prototype objects, and while doing it, extends an array that Prototype uses to set headers, adding to it a function that throws off the Prototype code completely.

There's a simple fix described here. It's kind of a hack, but at least will get you going for now.

Hope this saves you a few hours of head banging...

Posted in Rails, Ruby | no comments

AjaxScasffold, Security and Deployment Problems in Rails

Posted by Guy Naor Wed, 17 Jan 2007 23:05:28 GMT

AjaxScaffold was already mentioned in my previous post, so no need to sing it's praise again...

While deploying the Famundo help to my staging server, it stopped working, not even leaving a single clue in the logs. For me this is always a sign that something isn't initializing correctly. So it was time for a small investigation.

After playing a bit with the code, I realized the problem is caused by init.rb in AjaxScaffold trying to copy it's files into the application main directories. The reason this is a problem is my desire to make the system as secure as possible. Part of that is not letting the user the application runs as, write access into the application directory. This prevents a bug or breakin from writing into the application directories, reducing the damage that can be caused. The user running the application has only read access to the application directories.

Time to fix AjaxScaffold. First of all, I don't think that in production mode those files need to be copied over. It's done in development mode, and then are there for production mode. I do think it's a nice thing for development mode as it allows easy upgrade to a new AjaxScaffold version. Second, an error like that shouldn't kill the application with no explanation.

So my fix just adds an if around the copy and skip it in production mode, and also surounds it with begin/rescue/end, logging the error if one happens.

I also opened a ticket in the AjaxScaffold bug database, and I'll try to find who to email this to. For now, just take this file and replace your init.rb with it, or just copy the changes.

NOTE: The edge code of AjaxScaffold plugin moved the file copy to install.rb, so you'll have to change that file instead.

# Include hook code here
require 'ajax_scaffold_plugin'

ActionController::Base.send(:include, AjaxScaffold)
ActionView::Base.send(:include, AjaxScaffold::Helper)

# copy all the files over to the main rails app, want to avoid .svn
# Do not copy in production mode!!! And catch errors and log them
if ENV['RAILS_ENV'] != 'production'
  begin
    source = File.join(directory,'/app/views/ajax_scaffold')
    dest = File.join(RAILS_ROOT, '/app/views/ajax_scaffold')
    FileUtils.mkdir(dest) unless File.exist?(dest)
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public')
    dest = RAILS_ROOT + '/public'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public/stylesheets')
    dest = RAILS_ROOT + '/public/stylesheets'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public/javascripts')
    dest = RAILS_ROOT + '/public/javascripts'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)

    source = File.join(directory,'/public/images')
    dest = RAILS_ROOT + '/public/images'
    FileUtils.cp_r(Dir.glob(source+'/*.*'), dest)
  rescue Exception => ex
    RAILS_DEFAULT_LOGGER.error "AjaxScaffold error while copying the AjaxScaffold files to the application directory. (#{ex.t_s})"
  end
end

Posted in Rails, Ruby, Linux | no comments

Ruby Duck Typing Goodness

Posted by Guy Naor Fri, 12 Jan 2007 22:48:00 GMT

Coming over from C++ to Ruby, duck typing was one of the really cool features I learned about. You don't use it every day, but when you need it, it's an amazing tool.

Case in point: writing the admin interface to the Famundo help app (you can currently see the public interface at Famundo's Help. (A new version is coming out shortly, and I intend to Open Source it soon.) I used AjaxScaffold for the interface, but wanted to also manage file uploads. I wanted to keep it the same interface as the rest of the tables, as it gives me a nice UI, sorting, searching, etc...

To make that work, I duck typed a class that uses the filesystem, but acts like an ActiveRecord class. I then pointed AjaxScaffold at it and as far as the user experience goes, it's just like managing a database table. Simple and intuitive.

I didn't duck type everything in ActiveRecord, just the stuff that AjaxScaffold needed. Of course, if the need comes, adding more methods is very easy.

Following is the class I created. The $STORAGE_DIR global points to where the storage dir is. Usually something under public so that it's easy to serve the files. But you can also use x-send-file or some other trick and put it someplace else.

class StoredFile 

  attr_accessor :size, :filename, :modified_time

  # Trick the id...
  alias_method :id, :filename 
  alias_method :id=, :filename=

  # All the methods we need to overwrite from active_record...
  def self.table_name
    'stored_files'
  end

  def self.primary_key
    'filename'
  end

  def filename_before_type_cast
    filename
  end

  def self.count(*args)
    options = args.last.is_a?(Hash) ? args.pop : {} # Taken from the rails source!
    fltr = options[:conditions] || ''
    Dir.entries($STORAGE_DIR).delete_if{|i| i[0..0] == '.' || (fltr.is_a?(Regexp) ? i !~ fltr : !i.downcase.include?(fltr.to_s))}.size
  end

  def self.find(*args)
    options = args.last.is_a?(Hash) ? args.pop : {} # Taken from the rails source!
    # See if it's a single file find, like ID in rails tables
    if args.first.is_a?(String)
      fname = args.first.gsub(/[^\w\.\-]/, '_')
      raise "File not found #{args.first}" if !File.exist?("#{$STORAGE_DIR}#{fname}")
      return StoredFile.init_from_file(fname)
    end

    fltr = options[:conditions] || ''
    flist = Dir.entries($STORAGE_DIR).delete_if{|i| i[0..0] == '.' || (fltr.is_a?(Regexp) ? i !~ fltr : !i.downcase.include?(fltr.to_s))}.collect{|f| StoredFile.init_from_file f }

    # This is now an unsorted list of files. Now we can sort and apply limits/offsets
    if options[:order]
      options[:order] =~ /^(.+) (asc|desc)?$/i
      reverse_ord = !$2.nil? && ($2.downcase == "desc")
      case $1
        when 'filename'     : flist.sort! {|a,b| a.filename <=> b.filename }
        when 'modified_time': flist.sort! {|a,b| a.modified_time <=> b.modified_time }
        when 'size'         : flist.sort! {|a,b| a.size <=> b.size }
      end

      flist.reverse! if reverse_ord
    end

    idx = (options[:offset] ? options[:offset] : 0).to_i
    len = (options[:limit]  ? options[:limit ] : flist.size).to_i

    flist[idx,len]
  end

  def errors
    []
  end

  def self.destroy fname
    FileUtils.rm_f "#{$STORAGE_DIR}#{fname}"
    fname
  end

  def destroy
    StoredFile.destroy self.id
  end

  protected

  def self.init_from_file f
    fstat = File::stat "#{$STORAGE_DIR}#{f}"
    sf = StoredFile.new
    sf.filename, sf.modified_time, sf.size = f, fstat.mtime, fstat.size
    sf
  end

end

Posted in Rails, Ruby | 2 comments

rake support for capistrano deprecated - yeah!

Posted by Guy Naor Mon, 25 Dec 2006 20:47:36 GMT

A changest committed to the rails tree deprecated running capistrano commands through rake. Now you have to run them using the cap command.

I think this is a very good decision! It confused many users before, as some remote tasks worked through rake, while others only through cap (user created cap recipies didn't work through rake unless added by the user).

Now everyone will know: rake for local, cap for remote. Clean and simple.

Posted in Rails | no comments

Improving Postgres LISTEN/NOTIFY Support in Ruby/Rails

Posted by Guy Naor Thu, 07 Dec 2006 12:00:00 GMT

The Postgres DB server has a very cool mechanism for passing messages between the backend and clients and between the clients, using a simple LISTEN and NOTIFY system.

The way it works is that any client can register to LISTEN to a named event, and whenever any other client (or the backend using triggers/functions) issues a NOTIFY command for the named event, all listening clients will be notified of it. You can find the full documentation here and here.

A usage scenario is a background process that need to act at specific times based on database changes. The process can sleep not consuming any CPU, until a notification arrives, it wakes up does it's thing, and go back to sleep. It can also be used to keep cache consistency or to manage locks between clients. And many other scenarios.

The ruby postgres driver (C based version) has some support for this using the get_notify method. But it has some limitations which I wanted to remove:

1. Notifications arrive at the client whenever they are sent, but to know they arrived you need to poll the library, in effect making it again a poll solution. I wanted to change it to a solution that uses no CPU while waiting, and also isn't dependant on polling intervals.

2. Currently you have to issue a query to know that the notification arrived. This makes the polling solution even less attractive as you have to generate some query to know if something arrived.

The Postgres library provides a function called PQconsumeInput that solves the problem of having to send a query to see if a notification arrives. Once executed, calling get_notify will return any waiting notifications (call it multiple times until a nil is returned, as each call returns one notification).

To completely remove the need to poll (even with PQconsumeInput you still need to poll) I added another function based on an example from the Postgres documentation. This functions will get the current connection socket, and using a select() (system one, not sql) wait for any new activity on the socket. Once a notification arrives, the select() will stop blocking, and we can call consume_input and then get_notify() to get any waiting notifications. Put it in a loop and go back to waiting after handling the incoming events, and you have a process that waits for notifications without using any CPU time while waiting.

Please note that in order to use those new functions in rails applications you have to use the raw_connection:

ActiveRecord::Base.connection.raw_connection.consume_input

Here is diff to the Ruby driver (you will need to recompile it). If you would like to also have the patched C file, let me know.

--- ruby-postgres-0.7.1/postgres.c   2003-01-05 17:38:20.000000000 -0800
+++ postgres.c  2006-12-04 13:04:55.000000000 -0800
@@ -439,6 +439,40 @@
     return ary;
 }

+
+static VALUE
+pgconn_consume_input(obj)
+    VALUE obj;
+{
+    PGconn *conn = get_pgconn(obj);
+    if (PQconsumeInput(conn) == 0)
+      rb_raise(rb_ePGError, PQerrorMessage(conn));
+    return Qnil;
+}
+
+
+static VALUE
+pgconn_wait_for_activity(obj)
+    VALUE obj;
+{
+    fd_set input_mask;
+    PGconn *conn = get_pgconn(obj);
+    int    sock  = PQsocket(conn);
+
+    if (sock < 0)
+         rb_raise(rb_ePGError, "Bad connection socket");
+
+    FD_ZERO(&input_mask);
+    FD_SET(sock, &input_mask);
+
+    /* Wait for something to happen on the socket */
+    if (select(sock + 1, &input_mask, NULL, NULL, NULL) < 0)
+        rb_raise(rb_ePGError, "Socket select() failed");
+
+    return Qnil;
+}
+
+
 static VALUE pg_escape_regex;
 static VALUE pg_escape_str;
 static ID    pg_gsub_bang_id;
@@ -1443,6 +1477,8 @@
     rb_define_method(rb_cPGconn, "getline", pgconn_getline, 0);
     rb_define_method(rb_cPGconn, "endcopy", pgconn_endcopy, 0);
     rb_define_method(rb_cPGconn, "notifies", pgconn_notifies, 0);
+    rb_define_method(rb_cPGconn, "consume_input", pgconn_consume_input, 0);
+    rb_define_method(rb_cPGconn, "wait_for_activity", pgconn_wait_for_activity, 0);

 #ifdef HAVE_PQSETCLIENTENCODING
     rb_define_method(rb_cPGconn, "client_encoding",pgconn_client_encoding, 0);

Posted in Rails, Ruby, Postgres | no comments

Retrieving Files With Capistrano

Posted by Guy Naor Thu, 23 Nov 2006 08:13:00 GMT

I had today an interesting problem - how do I retrieve a file from a server using capistrano. First an answer to questions I know I'm going to have to answer...

Why with capistrano? Becuase it's VERY easy to do with capistrano. We just need a small addition to it. And most important, the connections to the servers and the ability to run commands are already there in capistrano, making it easy to automate the tasks. One cap command instead of multiple shell commands.

So as an example, I want to retirieve a database dump and some files from the server, in order to test stuff on the development machines. Capistrano doesn't have a get, it only has a put. And for a good reason. First of all, this isn't a common task, and second, how do you retrieve the same file from multiple servers? What about overwriting?

So first we lay some ground rules. We will retrieve from one server only (though it's easy to make it work for multiple, but it won't be generic). To make sure it's run on one server, we'll mark the server as primary. Just like for a database. But we'll do it for an app server. We will create a local directory to save the files into. We also create a temp directory on the server to store the files we will retrieve.

In the area of deploy.rb that has all the server definition, define the main app server:

role :app, "app.test.com", :primary => true

And create a new task:

desc "Get database dump and uploaded files from the server"
task :retrieve_content, roles => :app, :only => { :primary => true } do
  cont_dest  = "tmp_content"

  # Add your rollback here...

  transaction do
    system("mkdir -p #{cont_dest}") # Create a directory for storing the files
    # Prepare the backup
    run <<-CMD
      mkdir -p ~/#{cont_dest} &&
      pg_dump -d test_production -U tester -c > ~/#{cont_dest}/backup.sql &&
      tar -C #{deploy_to}/current/public/uploads/ -czf ~/#{cont_dest}/uploads.tar.gz .
    CMD

    # Now we retrieve the files...
    execute_on_servers(options) do |servers|
      puts "    >> Retrieving remote files into #{cont_dest}"

      # Here is the important "magic": we retrieve the current SSH session
      # belonging to our server (remember, we have one server only so we use servers[0])
      # then we create on top of that session an sftp session and use get_file to retrieve the files
      self.sessions[servers[0]].sftp.connect do |tsftp|
        tsftp.get_file "#{cont_dest}/backup.sql", "#{cont_dest}/backup.sql"
        tsftp.get_file "#{cont_dest}/uploads.tar.gz", "#{cont_dest}/uploads.tar.gz"
      end
    end

    # For cleanup we remove the remote files
    delete "~/#{cont_dest}", :recursive => true
  end
end

Now that the basics are there, go and retrieve whatever you want from the server. And see how cool it is with capistrano, where I just have to type

cap retrieve_content

Posted in Rails | no comments

Visual Detour for Thanksgiving

Posted by Guy Naor Wed, 22 Nov 2006 12:08:00 GMT

Coding isn't all there is to life, and just a a few days ago I went on a hiking trip to the most amazing place, and thought it's a good idea to share some nice photos for a Thanksgiving post. After all, between coding, we also need some fresh air, nice views and good company to keep us happy!

This place is called The Wave and it's an eroded sand-stone formation located in Utah, close to the Zion and Bryce canyons. It's a short hike (about 2 miles) in a breathtaking area. But nothing prepares you to The Wave. It's just AWESOME.

So lets have the pictures do the talking.

no comments

A World Time Server In One Line Of Rails

Posted by Guy Naor Thu, 09 Nov 2006 15:26:00 GMT

First, sorry for the delay in posting more on the XMPP/Jabber serie. I was a bit busy. So until I write the next installment, here's a small piece of rails coolness.

Want to know the exact time anywhere in the world? Including daylight saving taken into account? How about doing it in one line of rails code?

1. Create a new rails application: rails .

2. Install the tzinfo plugin: script/plugin install tzinfo_timezone

3. Edit app/controllers/application.rb

4. Add to it the following method (ok, it's 3 lines if we count the def and the end and not try to squeeze it into one with ; ):

def get_time
   render :text => TzinfoTimezone[params[:id]].utc_to_local(Time.now.getutc).strftime('%Y-%m-%d %H:%M:%S') rescue render :text => "ERROR - check your time zone", :status => 500
end

Now run it: script/server

Browse to: http://localhost:3000/application/get_time/Tokyo or http://localhost:3000/application/get_time/London. Check out the tzinfo plugin for the names of the supported time-zones. And you can add more to the mapping there.

You can improve performance a bit by turning sessions off completely. Do that either in the configuration or by adding session :off to the application controller class.

If you keep your computer clock accurate with ntp, you will get a pretty accurate time. The request is processed at a really high speed, so that shouldn't be a problem. And if the round trip to the server and back is quick, you will have a 1 second accuracy. Not bad for one line of code.

Posted in Rails, Ruby | no comments

Ruby and XMPP/Jabber Part 3: Adding html to the messages

Posted by Guy Naor Wed, 18 Oct 2006 15:02:00 GMT

If you used a Jabber client like GAIM or Trillian you know that it supports rich text messages. Those can be really nice to send some nicely formatted messages.

But I couldn't find much documentation on how to do it, outside the XEP/JEPs. To save you the pain of reading RFCs and XEPs/JEPs, I'll show you here how to send those nice rich messages.

The rich text is sent as an additional element of the message. The body still remains the same plain text body, and will be displayed by clients that do not support rich-text.

The type of markup that can be used is pretty limited - a small subset of regulat xhtml markup. The full details of the markup and how to use it are given in XEP-0071. But I'll go here over the important details.

To send the message we add a new element of type html with the namespace http://jabber.org/protocol/xhtml-im, and in it we put the html. The html should be the body part only. The following tags are supported: a, br, img, ul, li, ol, p, span. (Technically there are more tags that could match the namespace, but are not recommended.) The markup is styled with CSS using the style attribute of the tag. Supported styles are: background-color, color, font-family, font-size, font-style, font-weight, margin-left, margin-right, text-align, text-decoration.

Lets give it a try in irb (using the same login details from the previous post:

require 'xmpp4r/client'
include Jabber

# Login
jid = JID::new('test@yeush.com/Testing')
password = 'test'
cl = Client::new(jid)
cl.connect
cl.auth(password)

# Create a message
to = "test_with_me@yeush.com"
subject = "XMPP4R Rich-Text Test"
body = "Wow, I can do HTML now. But if you see this, your client doesn't support it"
m = Message::new(to, body).set_type(:normal).set_id('1').set_subject(subject)

# Create the html part
h = REXML::Element::new("html")
h.add_namespace('http://jabber.org/protocol/xhtml-im')

# The body part with the correct namespace
b = REXML::Element::new("body")
b.add_namespace('http://www.w3.org/1999/xhtml')

# The html itself
t = REXML::Text.new( "This is so <strong><span style='background: #003EFF; '><span style='font-size: large; '>COOL!!!</span></span></strong>. I can really do <strong>HTML</strong> now.", false, nil, true, nil, %r/.^/ )

# Add the html text to the body, and the body to the html element
b.add(t)
h.add(b)

# Add the html element to the message
m.add_element(h)

# Send it
cl.send m

It's pretty simple once you understand how to add the element to the message. Nothing fancy going on. And you can use other ways to build the markup. I'm using REXML as it's what XMPP4R considers the native elements. But assigning pre-formatted elements will also work.

Well, go send some nice messages, and come back for the next post about queries and callbacks. This will get us a bit deeper into the protocol, and get us ready for more interesting things.

UPDATE: A short explanation of the last parameter on the REXML::Text.new parameter is in order. As we are passing a pre-formatted text that include characters that are XHTML markup characters, we need to tell REXML to ignore those. We do that by passing a regular expression that will never match anything as the illegal parameter.

Posted in Ruby, XMPP4R/Jabber | no comments | no trackbacks