The Dev Blog

Improving Capistrano's put Command

Posted by Guy Naor Sat, 10 Mar 2007 10:00:00 GMT

Capistrano can be used for many tasks not directly related to deployment. One such task, which I use a lot, is getting content form staging servers, and putting it up on production servers. (See my post about Deploying to Staging and Production with Capistrano for how to manage a staging/production split in capistrano.)

Capistrano already has a put command for putting data on the server. The problem is that it expects a string, and so if uploading very large files (like a tar.gz of uploaded files, for example) this is VERY memory intensive. Imagine a 1GB file loaded into a string...

As capistrano uses sftp if available, I wrote a small helper that does the same thing that get_file does in capistrano (this function is new in capistrano 1.4). Only it uploads files instead of downloading. There is one slight inconvenience with the code, in that it uploads the files one by one. Capistrano's put works in parallel, uploading to all servers concurrently. Usually this isn't a problem, as the upstrean bandwidth usually limit the speed of concurrent uploads anyway. A perfect solution will be to use a modified version of put, that reads the files a chunk at a time and uploads in chunks. For my need this is overkill.

Open your deploy.rb recipies file, and add to it the following:

class Capistrano::Actor

  # A saner put replacement that doesn't read the whole file into memory...
  def put_file(path, remote_path, options = {})
    raise "put_file can only be used with SFTP" unless Capistrano::SFTP && options.fetch(:sftp, true)

    execute_on_servers(options) do |servers|
      servers.each do |server|
        logger.info "uploading #{File.basename(path)} to #{server}"
        sftp = sessions[server].sftp
        sftp.connect unless sftp.state == :open
        sftp.put_file path, remote_path
        logger.debug "done uploading #{File.basename(path)} to #{server}"
      end
    end
  end

end

To use it, call it just like get_file:

desc "Upload content to the remote server"
task :upload_content, :roles => :app do
  put_file 'content.tar.gz', 'content.tar.gz'
end

Note that it requires SFTP and will raise an error if not available.

As of capistrano 1.4, you can now put shared capistrano code in /etc/capistrano.conf and it will loaded into every single project. You can put the above code in /etc/capistrano.conf and have it available everywhere.

Posted in Rails, Ruby | no comments

How I learned to Love vim and Ditched Eclipse

Posted by Guy Naor Thu, 08 Mar 2007 13:03:00 GMT

For the longest time, since starting to work with Ruby and Rails, I used Eclipse with RadRails and RDT for coding. It's a pretty good IDE as far as IDEs go, and has nice Ruby/Rails tools to make development a lot easier. It also has a VERY nice subversion interface. One of the best I've seen as far as working the way I like to work.

On the down-side, it's a performance hog and the editor itself is so-so in features.

As I also do system admin chores on remote computers, I always use vi as well. And for many quick and dirty tasks, I use it even localy. Developing on a Linux machine, I always have some terminal windows open. Slowly I was pulled to do more things with vim, until I realized I prefer being in it than in Eclipse. Now I spend 90% of my time in vi, and most of my projects never even see Eclipse.

Why did I switch? First and foremost vi as an editor is really good. And lets keep the vi/emacs wars for another time. I'm sure emacs is a really good editor as well. I just have to choose my tools. Can't use all of them. Second thing is speed. I NEVER have to wait for vi. It's always super fast and super responsive.

But to make vi really useful with ruby and rails, some additions are needed. After all, I want synax highlighting, macros, easy way to open files, etc... So here is a list of vim scripts/plugins I use and recommed:

rails.vim, eruby.vim, ruby-macros.vim, rubycomplete.vim

There are many more additions and plugins on the vim site. Please note that some require the newer vim 7 to work.

Another option is getting the rails-vim gem: sudo gem install vim-ruby. Please note you have to then run the vim-ruby-install.rb script to complete installation.

So now that we have vim loaded with the additions, where do we start? Granted, the learning curve is steep! And don't get me started on using the esc key so much, that I keep doing it in Eclipse ;-). But as the editor is our main tool of work, learning it pays for itself later with productivity. vim has a lot of help written for it, and you can access it from within vim, but I prefer browsing it on the internet. So start with this short tutorial, and then bookmark the main help file. And for the rails related parts, take a look here. Some of the things it does are just awesome. I open files faster with :Rfind than on a tree in the IDE browser! And check out the partial extraction. That one is unbelievable.

vim also has the GUI version in gvim. And there you can open multiple tabs and have an IDE like file browser. Use that if you are more comfortable with the mouse and a real GUI. It's also easier to stasrt with as it has menus for the more common commands.

One thing I did that really got me into more advanced editing, is deciding that whenever I want to do something, and I don't know how to best do it, I stop resorting to hacking a solution with the things I know, and look for the real solution. In a few short weeks I learned more than in the last 5 years...

So give vi a try. Give yourself some learning time, and you'll never look back!

BTW, I also use svn from the command line. I'ts so much faster than from Eclipse that I don't mind the few keystrokes it takes to get things done. It's so much faster!

Posted in Rails, Ruby, Programming, Linux | 41 comments

A New Face for Famundo

Posted by Guy Naor Thu, 08 Mar 2007 05:46:00 GMT

Finally after lots of work, the new Famundo site is up an running. I believe it's VERY VERY nice. But go check it out! And do sign up for Famundo while you are there ;-).

We also have the new mascots - same family as the one I have with the logo here on the blog. They are officially called the Famundudes (Famundad, Famunmom, Famunkids, and and the members of the community they interact with).

There's an interesting story behind the site - the site is a fully page-cashed Rails application. And it has a small admin interface to manage all the changeable contents - like news and such. But the server I wanted to put it on started to miss-behave. So we generated all the needed pages by just accessing all the functionality, took the generated pages and uploaded it like a static content. Here's for you a nice little trick with rails for static sites.

The site that will run live with rails is coming soon. And it has a few tricks up it's sleeve - like the ability to switch serving of assets from our servers or from S3, and the ability to cache the html directly into S3. I'll write more about it when it's up.

Please go visit the new site and sign up for Famundo.

Posted in Rails, S3 | no comments

libical + ruby + linux = nirvana

Posted by Guy Naor Thu, 22 Feb 2007 13:25:00 GMT

We have a need for strong ical support in Famundo, and unfortunately all the ruby libraries for handling ical are nowhere near complete. A really great alternative is to use the well used and tested libical. This requires a wrapper that will let ruby use it, and the simplest way to do that, is to use SWIG to generate the wrapper, then compile it into an extension ruby can use.

Lucky for me, Rob Kaufman in this post contributed most of what's needed to get it going with SWIG. But it won't work on Linux. It seems it was compiled on a Mac, and trying to compile it on Linux just won't work.

Lucky for you, I have here the steps to make it compile on Linux (and probably any other system that supports the standard make tools). So lets jump right into the instructions.

1. Get the latest libical files and the zip file attached to Rob's post. You will also need to get the latest SWIG if it's not yet installed on your system.

2. Compile and install libical:

    tar tar xif libical-0.26-6.aurore.tar.bz2
    cd libical-0.26
    ./configure # you can run ./configure --help to see more options  
    make
    sudo make install 

3. Extract the files of the ruby SWIG based wrapper:

    unzip libical-ruby.zip
    cd libical-ruby/swig/
    make clean

4. Create an extconf.rb file to generate the makefile. The file should have the following in it:

    require 'mkmf'

    have_library("pthread")
    have_library("ical", "icaltime_null_time")
    have_library("icalss", "icalset_new_dir")

    create_makefile('LibicalWrap')

5. Generate the needed wrapper file using swig: swig -ruby -o LibicalWrap.c ical.i

6. Run: ruby extconf.rb This should generate the needed Makefile. If it fails, you might not have the libraries from libical installed correctly. If that is the case, redo step 2 above, but use the ./configure command with different parameters, to point it to the correct lib directory. For example, on 64bit FC4, you might want to try with this:

./configure --libdir=/usr/lib64 --includedir=/usr/include

7. Compile and install the wrapper:

    make clean
    make 
    sudo make install

8. We're done. You can now launch irb and type include 'LibicalWrap', and you are ready to go. With the libical-ruby.zip file, there are also some tests you can run on it to see if it works. And some ruby helpers to facilitate working with the library, as the library is C based, and so doesn't have nice class representations for ical objects.

Now that we have a good ical library, time to start using it. But that's outside the scope of this post.

Let me know if you have problems getting it to work. I'll be glad to help.

Posted in Rails, Ruby, Linux | 2 comments

A New Version of acts_as_rated plugin

Posted by Guy Naor Sun, 18 Feb 2007 13:09:00 GMT

I just uploaded to rubyforge a new version of the acts_as_rated plugin.

This version is thanks to Tiago Serafim who tested it and proposed the needed changes needed to make it work with MySQL.

Tiago also proposed a new method: rated_by?(rater) that returns true if the object is rated by the passed rater.

Thanks Tiago!

As usual full testing is provided. There is one failure in MySQL testing due to strangeness in MySQL average handling. They do fully pass with Postgres.

Posted in acts_as_rated, Rails, Ruby | 2 comments

I Love Ruby on Rails But...

Posted by Guy Naor Sat, 17 Feb 2007 12:00:00 GMT

...The last thing I want is start a language/framework war.

A funny little post I did about a world time server in a single line of rails code was posted on dzone with some comment about ruby, php and java, and caused a lot of heated comments.

So hereby I declare: I Love Rails, but I'm the last one to think that rails is the Holly Grail of languages and framework. It's really well written, it's a joy to write web apps in, but it's not the only game in town.

You like php? Love Java? Think C# is the best thing since sliced bread? All the power to you! Use them and enjoy them. And I'm sure there are enough projects where it makes more sense to to use those languages/framework.

I'm a true believer in one thing: know has many tools as you can comfortably manage, and use the one best suited for the task. For years I programmed in in C/C++ on Win32. I know a very large number of languages, and worked professionally with C/C++, Pascal, Perl, PHP, Ruby, dBaseII - dBaseIV (I think I'm giving away my ancientness here...) and a lot of other languages. Really, even assembler. Heck, I'm teaching my 9 year old daughter to program in Logo.

So please, don't use my posts for language wars, we have enough of those already.

Posted in Rails, Ruby, Programming | 1 comment

Adding More Control to the Displayed Data in AjaxScaffold

Posted by Guy Naor Thu, 15 Feb 2007 10:00:00 GMT

Wow, my 4th post of AjaxScaffold! This time a small change in it to let you better control the query used to retireve the data into the grid.

The problem I'm trying to solve, is including a join to the query used when laoding the grid. It's especially useful when doing filtering and searching. My example is the User model that has also a UserSetting association. I want the query to include a join to the user_settings table, as I want to be able to search on the fields from the joined table.

The easiest way to search/filter on AjaxScaffold is to define in your controller the method:

def conditions_for_#{plural_name}_collection
end

The value returned from this method is assigned internally by AjaxScaffold to the :conditions option of find. But if you now try to write a condition like:

['lower(name) LIKE ? OR lower(city) LIKE ?', name, city]

With city coming from the user_settings table, it will fail, as this field isn't part of the query. So we need to also change the query to include additional query parameters. All we need to do, is add another callback like the one above, that will let us adjust the options used for the find:

def adjust_options( options )
  options.merge!( { :joins => 'inner join user_settings on user_settings.user_id = users.id'} )
end

To implement the change, we need to add a call to adjustoptions before using the options in the code. Look in vendor/plugins/ajaxscaffoldp/lib/ajaxscaffoldplugin.rb for the function def #{prefix}tablesetup. Around line 200, after the code:

options = { :order => order,
                :conditions => conditions_for_#{plural_name}_collection,
                :direction => current_sort_direction(params),
                :per_page => #{rows_per_page} }

Add the following:

adjust_options(options) if self.respond_to? "adjust_options"

This is a hack, as I didn't want to do a big patch when a new and highly modified version AjaxScaffold is on it's way. Should be fine as an interim solution.

Posted in Rails | no comments

Testing: DRYing the Asserting of Array Similarity

Posted by Guy Naor Tue, 13 Feb 2007 11:21:00 GMT

After one too many instances of:

ph = contacts(:john).phones.collect{|p| p.number }
assert_equal 3, ph.size
assert_equal []. ['12345', '56789', '67890'] - ph

I decided it's time to DRY up array similarity checking. Using assert_equal on arrays is a so-so solution, given that you don't always know if the order in the array is the same. Returning an array from a has_many collection has a non-deterministic order by default. So asserting that what we get from a collection is a bit painfull, and require the code above to make absolutely sure we got what we wanted.

To make my life easier and DRYer, I added the following function to the test/test_helper.rb file:

def assert_array_similarity(expected, actual, message=nil)
  full_message = build_message(message, "<?> expected but was\n<?>.\n", expected, actual)
  assert_block(full_message) { (expected.size ==  actual.size) && (expected - actual == []) }
end

And now testing for array similarity is simply:

assert_array_similarity ['12345', '56789', '67890'], contacts(:john).phones.collect{|p| p.number }

Not bad, 1 line instead of 3 and better error reporting to boot.

Posted in Rails, Ruby | 8 comments

Solving Tough Deploy Problems with FastCGI and Rails

Posted by Guy Naor Mon, 12 Feb 2007 03:13:00 GMT

Sometimes you deploy a new rails app with fastcgi and it just doesn't work. No errors, no logs, nothing! Usually it's some permission problems, or a missing file. But it also might be some odd problems like a wrong shebang line somewhere.

The problem with those is that you get no error and no indication of what went wrong. When using the spawner script you might see the actual appication being launched again and again by the spawner, only to die immediately.

I found a neat trick to see what's really going on when this happens. On the deployment directory (on the server you deploy to) edit the file:

vendor/rails/railties/lib/commands/process/spawner.rb

And look for the lines that redirect STDIN, STDOUT and STDERR to /dev/null. Comment those lines, and launch the spawner from the command line. The errors will start flowing into the console. Just look at them and you should be on your way to solving the problem.

Here is some code from the spawner file with the lines commented out:

def daemonize #:nodoc:
  exit if fork                   # Parent exits, child continues.
  Process.setsid                 # Become session leader.
  exit if fork                   # Zap session leader. See [1].
  Dir.chdir "/"                  # Release old working directory.
  File.umask 0000                # Ensure sensible umask. Adjust as needed.
#  STDIN.reopen "/dev/null"       # Free file descriptors and
#  STDOUT.reopen "/dev/null", "a" # point them somewhere sensible.
#  STDERR.reopen STDOUT           # STDOUT/ERR should better go to a logfile.
end

Posted in Rails, Ruby | no comments

Hooked on Testing

Posted by Guy Naor Wed, 07 Feb 2007 10:50:00 GMT

I am now officially completely hooked on testing! The last few weeks proved to me how empowering tests can be. And not just for sleeping good at night, it has some other advantages that make every minute spent on testing worth the time.

When Rails 1.2 was released, I wanted to switch over a few of my applications. The ones that have a full test suite where a no brainer. Switch rails version (I keep it on my own repository), run tests, fix what's broken, and off we go! Even the small apps had some issues with the migration - some small changes that needed to be done. With the tests it's easy. But some other old apps that didn't have full testing - I just decided to wait a bit. It will require too much manual labor to make sure everything is in place. I had a similar experience with switching plugin versions.

Now every change I need to do, I can easily verify that it will work. No need for guessing or manual testing (which isn't reliable enough). This is a huge improvement from what I was used to in past, especially in my C++ development days, where testing was always external to the application, and it's so much better!

Regarding TDD (Test Driven Development) - I think it's amazingly well suited for functional testing, and even more so for integration testing. For models I still prefer to model the structure first and then test, especially for complex data models. It is still more natural for me to think of the structure. Maybe it's something I will change with time as well.

My next goal in testing is getting Selenium going and doing with it acceptance and integration testing.

A really good book all around, but on tests it really shines, is Beginning Ruby on Rails E-Commerce. They practice TDD from the first step of development, and really make the process clear. I highly recommend it even if you don't need an introduction to Rails or learn e-commerce. It's worth the money just to learn testing and TDD. They even have a chapter devoted to Selenium. It's now the third book I recommend for Rails developers: Programming Ruby, AWDR and Beginning Ruby on Rails E-Commerce.

Posted in Rails | 2 comments