Posted by Guy Naor
Wed, 28 Mar 2007 07:00:00 GMT
I've been using the security_extensions plugin to secure forms in Famundo and some other projects. It's a very simple plugin that adds protection against CSRF.
When upgrading one of my projects to Rails 1.2, I got a deprecation warnings from Rails, as this plugin requires start_form_tag and end_form to work. Thinking it was all easy to change, I replaced all calls to the new form_tag ... do format. This change resulted in lots of errors, all complains from erb on missing the _erbout variable.
After a lot of digging, I realized this error is caused by the way variables bindings work in Ruby, and the fact that erb uses it to pass along the output string it creates.
What are bindings? Bindings are (put very simply) the context for the variables in an execution block. It's what's used in Ruby to bind a variable to a block and have the block access it even after the variable went out of scope in the original code block, or the variable is re-defined in a new block. Here is some code to make it clear:
a = 5
level_a = lambda { puts a }
level_a.call
def level_b(blk)
a = 10
blk.call
end
level_b(level_a)
The best place I found to learn about bindings is this page.
How is this affecting the security_extensions plugin? The plugin needs to wrap the block given to the form, and inject into it the hidden field used to validate the form when it's posted back. When using the non-block accepting start_form_tag, it just appends a new field at the end:
def secure_form_tag(*args)
return start_form_tag(*args) + "\n" +
hidden_field_tag('session_id_validation', security_token)
end
The simple solution I thought will work, is very simple:
def secure_form_tag(url_for_options = {}, options = {}, *parameters_for_url, &block)
if block_given?
form_tag(url_for_options, options, *parameters_for_url) do
yield
end
hidden_field_tag("session_id_validation", security_token)
else
"#{form_tag(*args)} \n #{hidden_field_tag('session_id_validation', security_token)}"
end
end
It failed. I decided to try and call the external block directly:
def secure_form_tag(url_for_options = {}, options = {}, *parameters_for_url, &block)
if block_given?
form_tag(url_for_options, options, *parameters_for_url) do
block.call
end
hidden_field_tag("session_id_validation", security_token)
else
"#{form_tag(*args)} \n #{hidden_field_tag('session_id_validation', security_token)}"
end
end
Failed again! The problem is that the context of the internal block is completely different from the context of the block passed to the function, and so the _erbout variable isn't bound to the internal block, only to the external one.
The solution I used was to copy the binding from the passed block into the internal block, call the passed block, and then copy it back from the internal block to the passed block. Here is the code to do it:
def secure_form_tag(url_for_options = {}, options = {}, *parameters_for_url, &block)
if block_given?
_erbout = eval('_erbout', block)
form_tag(url_for_options, options, *parameters_for_url) do
concat(hidden_field_tag("session_id_validation", security_token), block.binding)
eval "_erbout = %q[#{_erbout}]"
yield
end
eval "_erbout = %q[#{_erbout}]", block
else
"#{form_tag(*args)} \n #{hidden_field_tag('session_id_validation', security_token)}"
end
end
This code does a lot of copying of strings, but as it's in very specific places that aren't performance sensitive, I rather get the nice way to use secured forms, and incur the performance penalty in this case.
The same trick can be use to extended other erb related methods that use blocks and need the _erbout bindings.
If there are better solutions, let me know. I'd love a simpler solution for this.
Posted in Ruby, Programming | 7 comments
Posted by Guy Naor
Mon, 12 Mar 2007 09:01:00 GMT
Following the explanation on how to load compressed content into S3, here is a tool to make it easy.
cp2s3 is a ruby script that make it easy to upload whole directory structures into S3 buckets. Some of the more important features (all controlled with command line switches):
- Compress specified files types (i.e. compress js and css files, but leave images alone)
- Recursive copy to load complete directory structures
- Use Ruby's Dir::glob for very complex file selections
- Save sha1 digest, size and modification date to the item meta-data on Amazon
- Upload only changed files (based on the sh1 digest)
- Upload to multiple buckets at once
- Mark files as public readable
- The S3 keys can be assigned through the command line, envirnment variables, or embedded in the script.
Installing it requires the installation of the AWS::S3 Ruby library, easiers done with
sudo gem install aws-s3
To use the digest switch or the modified only option, you will need the sh1sum command line utility (or write your own that will return a unique file digest).
Run it with no parameters to get a full list of options. And let me know if you would like to have anything added or modified. It was tested in production, uploading more than 80,000 files in one session into S3. (I use it regularly to upload Famundo assets into S3.)
Click on read more to see some usage examples and the full source, or just download the attached script. I just put it in my /usr/bin folder and rename it to just cp2s3.Then I can run it just like any other command.
Read more...
Posted in Ruby, S3 | no comments
Posted by Guy Naor
Sat, 10 Mar 2007 10:00:00 GMT
Capistrano can be used for many tasks not directly related to deployment. One such task, which I use a lot, is getting content form staging servers, and putting it up on production servers. (See my post about Deploying to Staging and Production with Capistrano for how to manage a staging/production split in capistrano.)
Capistrano already has a put command for putting data on the server. The problem is that it expects a string, and so if uploading very large files (like a tar.gz of uploaded files, for example) this is VERY memory intensive. Imagine a 1GB file loaded into a string...
As capistrano uses sftp if available, I wrote a small helper that does the same thing that get_file does in capistrano (this function is new in capistrano 1.4). Only it uploads files instead of downloading. There is one slight inconvenience with the code, in that it uploads the files one by one. Capistrano's put works in parallel, uploading to all servers concurrently. Usually this isn't a problem, as the upstrean bandwidth usually limit the speed of concurrent uploads anyway. A perfect solution will be to use a modified version of put, that reads the files a chunk at a time and uploads in chunks. For my need this is overkill.
Open your deploy.rb recipies file, and add to it the following:
class Capistrano::Actor
def put_file(path, remote_path, options = {})
raise "put_file can only be used with SFTP" unless Capistrano::SFTP && options.fetch(:sftp, true)
execute_on_servers(options) do |servers|
servers.each do |server|
logger.info "uploading #{File.basename(path)} to #{server}"
sftp = sessions[server].sftp
sftp.connect unless sftp.state == :open
sftp.put_file path, remote_path
logger.debug "done uploading #{File.basename(path)} to #{server}"
end
end
end
end
To use it, call it just like get_file:
desc "Upload content to the remote server"
task :upload_content, :roles => :app do
put_file 'content.tar.gz', 'content.tar.gz'
end
Note that it requires SFTP and will raise an error if not available.
As of capistrano 1.4, you can now put shared capistrano code in /etc/capistrano.conf and it will loaded into every single project. You can put the above code in /etc/capistrano.conf and have it available everywhere.
Posted in Rails, Ruby | no comments
Posted by Guy Naor
Thu, 08 Mar 2007 13:03:00 GMT
For the longest time, since starting to work with Ruby and Rails, I used Eclipse with RadRails and RDT for coding. It's a pretty good IDE as far as IDEs go, and has nice Ruby/Rails tools to make development a lot easier. It also has a VERY nice subversion interface. One of the best I've seen as far as working the way I like to work.
On the down-side, it's a performance hog and the editor itself is so-so in features.
As I also do system admin chores on remote computers, I always use vi as well. And for many quick and dirty tasks, I use it even localy. Developing on a Linux machine, I always have some terminal windows open. Slowly I was pulled to do more things with vim, until I realized I prefer being in it than in Eclipse. Now I spend 90% of my time in vi, and most of my projects never even see Eclipse.
Why did I switch? First and foremost vi as an editor is really good. And lets keep the vi/emacs wars for another time. I'm sure emacs is a really good editor as well. I just have to choose my tools. Can't use all of them. Second thing is speed. I NEVER have to wait for vi. It's always super fast and super responsive.
But to make vi really useful with ruby and rails, some additions are needed. After all, I want synax highlighting, macros, easy way to open files, etc... So here is a list of vim scripts/plugins I use and recommed:
rails.vim, eruby.vim, ruby-macros.vim, rubycomplete.vim
There are many more additions and plugins on the vim site. Please note that some require the newer vim 7 to work.
Another option is getting the rails-vim gem: sudo gem install vim-ruby. Please note you have to then run the vim-ruby-install.rb script to complete installation.
So now that we have vim loaded with the additions, where do we start? Granted, the learning curve is steep! And don't get me started on using the esc key so much, that I keep doing it in Eclipse ;-). But as the editor is our main tool of work, learning it pays for itself later with productivity. vim has a lot of help written for it, and you can access it from within vim, but I prefer browsing it on the internet. So start with this short tutorial, and then bookmark the main help file. And for the rails related parts, take a look here. Some of the things it does are just awesome. I open files faster with :Rfind than on a tree in the IDE browser! And check out the partial extraction. That one is unbelievable.
vim also has the GUI version in gvim. And there you can open multiple tabs and have an IDE like file browser. Use that if you are more comfortable with the mouse and a real GUI. It's also easier to stasrt with as it has menus for the more common commands.
One thing I did that really got me into more advanced editing, is deciding that whenever I want to do something, and I don't know how to best do it, I stop resorting to hacking a solution with the things I know, and look for the real solution. In a few short weeks I learned more than in the last 5 years...
So give vi a try. Give yourself some learning time, and you'll never look back!
BTW, I also use svn from the command line. I'ts so much faster than from Eclipse that I don't mind the few keystrokes it takes to get things done. It's so much faster!
Posted in Rails, Ruby, Programming, Linux | 41 comments
Posted by Guy Naor
Fri, 02 Mar 2007 11:46:00 GMT
If you have yet to check out Amazon's S3 service, go do that now. I'll wait for you to come back! It's a very simple storage server that is also really really cheap, and high-performance. Backed by Amazon's network, it's also pretty reliable.
I am in the process of moving a lot of the static content of Famundo into the service. But one important requirement is serving JavaScript and CSS files compressed, as we have pretty big files for both.
By default S3 will serve the files as uncompressed text files, increasing the load time for the clients, so a solution for compressed files serving was needed.
When saving files on S3 I can pass in HTTP headers that will be returned when the file is accessed. Using this along with pre-compressed JS and CSS files, we can have S3 serve the files compressed. The limitation here as opposed to serving it directly with a web-server, is that there is no content type negotiation, meaning, the files will always be served compressed. So if the clients accessing your files cannot accept compression, you are out of luck, and got to either serve uncompressed or do it from your content negotiating server. In Famundo we target only newer browsers (FF, IE6+ and Safari 1.2+), so pre-compressed files works for us.
Enough words, time for some code. I'm showing it using the Ruby AWS::S3 library, but you can use whichever library/language you want. The principle is the same. What we are doing, is compressing the files, then uploading the compressed files and the most important part - assigning to it the correct HTTP headers.
require 'rubygems'
require 'aws/s3'
require 'stringio'
require 'zlib'
AWS::S3::Base.establish_connection!(:access_key_id => 'YOUR S3 ACCESS KEY', :secret_access_key => 'YOUR S3 SECRET KEY')
strio = StringIO.open('', 'w')
gz = Zlib::GzipWriter.new(strio)
gz.write(open('test.css').read)
gz.close
S3Object.store('test.css', strio.string, 'the_bucket', :access => :public_read, "Content-Encoding" => 'gzip' )
You can now open it in your browser: http://s3.amazonaws.com/the_bucket/test.css and get it as CSS, though it was transfered compressed.
You could do the same with static html files as well. Don't do it with images, as there is nothing to gain by compressing them.
If there is anything else you would like to learn how to do with S3 (virtual serving, metadata manipulation, etc...), let me know.
Posted in Ruby, Programming, S3 | 2 comments
Posted by Guy Naor
Thu, 22 Feb 2007 13:25:00 GMT
We have a need for strong ical support in Famundo, and unfortunately all the ruby libraries for handling ical are nowhere near complete. A really great alternative is to use the well used and tested libical. This requires a wrapper that will let ruby use it, and the simplest way to do that, is to use SWIG to generate the wrapper, then compile it into an extension ruby can use.
Lucky for me, Rob Kaufman in this post contributed most of what's needed to get it going with SWIG. But it won't work on Linux. It seems it was compiled on a Mac, and trying to compile it on Linux just won't work.
Lucky for you, I have here the steps to make it compile on Linux (and probably any other system that supports the standard make tools). So lets jump right into the instructions.
1. Get the latest libical files and the zip file attached to Rob's post. You will also need to get the latest SWIG if it's not yet installed on your system.
2. Compile and install libical:
tar tar xif libical-0.26-6.aurore.tar.bz2
cd libical-0.26
./configure # you can run ./configure --help to see more options
make
sudo make install
3. Extract the files of the ruby SWIG based wrapper:
unzip libical-ruby.zip
cd libical-ruby/swig/
make clean
4. Create an extconf.rb file to generate the makefile. The file should have the following in it:
require 'mkmf'
have_library("pthread")
have_library("ical", "icaltime_null_time")
have_library("icalss", "icalset_new_dir")
create_makefile('LibicalWrap')
5. Generate the needed wrapper file using swig:
swig -ruby -o LibicalWrap.c ical.i
6. Run: ruby extconf.rb This should generate the needed Makefile. If it fails, you might not have the libraries from libical installed correctly. If that is the case, redo step 2 above, but use the ./configure command with different parameters, to point it to the correct lib directory. For example, on 64bit FC4, you might want to try with this:
./configure --libdir=/usr/lib64 --includedir=/usr/include
7. Compile and install the wrapper:
make clean
make
sudo make install
8. We're done. You can now launch irb and type include 'LibicalWrap', and you are ready to go. With the libical-ruby.zip file, there are also some tests you can run on it to see if it works. And some ruby helpers to facilitate working with the library, as the library is C based, and so doesn't have nice class representations for ical objects.
Now that we have a good ical library, time to start using it. But that's outside the scope of this post.
Let me know if you have problems getting it to work. I'll be glad to help.
Posted in Rails, Ruby, Linux | 2 comments
Posted by Guy Naor
Sat, 17 Feb 2007 12:00:00 GMT
...The last thing I want is start a language/framework war.
A funny little post I did about a world time server in a single line of rails code was posted on dzone with some comment about ruby, php and java, and caused a lot of heated comments.
So hereby I declare: I Love Rails, but I'm the last one to think that rails is the Holly Grail of languages and framework. It's really well written, it's a joy to write web apps in, but it's not the only game in town.
You like php? Love Java? Think C# is the best thing since sliced bread? All the power to you! Use them and enjoy them. And I'm sure there are enough projects where it makes more sense to to use those languages/framework.
I'm a true believer in one thing: know has many tools as you can comfortably manage, and use the one best suited for the task. For years I programmed in in C/C++ on Win32. I know a very large number of languages, and worked professionally with C/C++, Pascal, Perl, PHP, Ruby, dBaseII - dBaseIV (I think I'm giving away my ancientness here...) and a lot of other languages. Really, even assembler. Heck, I'm teaching my 9 year old daughter to program in Logo.
So please, don't use my posts for language wars, we have enough of those already.
Posted in Rails, Ruby, Programming | 1 comment
Posted by Guy Naor
Tue, 13 Feb 2007 11:21:00 GMT
After one too many instances of:
ph = contacts(:john).phones.collect{|p| p.number }
assert_equal 3, ph.size
assert_equal []. ['12345', '56789', '67890'] - ph
I decided it's time to DRY up array similarity checking. Using assert_equal on arrays is a so-so solution, given that you don't always know if the order in the array is the same. Returning an array from a has_many collection has a non-deterministic order by default. So asserting that what we get from a collection is a bit painfull, and require the code above to make absolutely sure we got what we wanted.
To make my life easier and DRYer, I added the following function to the test/test_helper.rb file:
def assert_array_similarity(expected, actual, message=nil)
full_message = build_message(message, "<?> expected but was\n<?>.\n", expected, actual)
assert_block(full_message) { (expected.size == actual.size) && (expected - actual == []) }
end
And now testing for array similarity is simply:
assert_array_similarity ['12345', '56789', '67890'], contacts(:john).phones.collect{|p| p.number }
Not bad, 1 line instead of 3 and better error reporting to boot.
Posted in Rails, Ruby | 8 comments
