Posted by Guy Naor
Fri, 19 May 2006 06:57:00 GMT
Filling a long form on the web just to have it fail for all kind of reasons when I submit, always bothered me. I see it as unfriendly and inconvenient to start tracking down the failure, fill in the fields that weren't saved (like passwords) and submit it again. Sometimes ad-infinitum. Especially when the error is a unique login name, or other similar fields, that might require a few tries until you find a legal one.
We found a better solution for that. We run a few local validations using JavaScript, and for validation that require database access, or some calculation not easily done on the client, we just send an Ajax call into the server, and get back an error message if something is wrong.
Now when a user submit the form, in most cases there will be no errors generated, as it's all pre-validated.
If you need some sample code or help, let me know.
Posted in Rails, Ruby | 2 comments | no trackbacks
Posted by Guy Naor
Sat, 13 May 2006 20:20:00 GMT
While working on our Famundo application, we used the symbolize_keys! method of one of the Hashs in rails, not realizing it was actually a HashWithIndifferentAccess. To our amazement, all the entries in the hash just disappeared.
Not ones to leave alone an inexplicable event, we decided to research it a bit. Time to dig into the rails code for Hash. This is the code used to symbolize the keys (in keys.rb):
1 def symbolize_keys!
2 keys.each do |key|
3 unless key.is_a?(Symbol)
4 self[key.to_sym] = self[key]
5 delete(key)
6 end
7 end
8 self
9 end
The source of the problem is that in a HashWithIndifferentAccess a symbol and a string are treated as one and the same, by converting the symbol into a string on assignment to the hash. Line 3 will be true for every key in the hash entered as either a symbol (that was converted to a string) or a string. The assignment in line 4 will actually assign the value to itself, and then line 5 will delete our key, leaving us with an empty hash. The best way to see it is to launch script/console, and enter the following:
>> h = HashWithIndifferentAccess.new
=> {}
>> h[:first] = "Item 1"
=> "Item 1"
>> h['second'] = "Item 2"
=> "Item 2"
>> h
=> {"second"=>"Item 2", "first"=>"Item 1"}
>> h.symbolize_keys!
=> {}
>> h
=> {}
The fix is pretty simple and involves making sure that we don't try to run symbolize_keys! on a HashWithIndifferentAccess. So the new method will now be:
def symbolize_keys!
unless self.kind_of? HashWithIndifferentAccess
keys.each do |key|
unless key.is_a?(Symbol)
self[key.to_sym] = self[key]
delete(key)
end
end
end
self
end
I submitted a ticket with a patch to the Rails trac. I hope it'll be included soon so that nobody else get bitten by this bug.
Posted in Rails, Ruby | 3 comments | no trackbacks
Posted by Guy Naor
Wed, 22 Mar 2006 20:06:00 GMT
To be able to serve large amount of data, especially pictures and similar files, I needed it to be served by the web server directly. Web servers are so optimized to serve this kind of files that anything developed in rails (or any other framework/language) will always be slower.
But there is a problem with serving those files in a system requiring authentication. Putting them anywhere the server can get at, will make them available to anyone knowing the URL - not good at all. Using caching (like the one built into rails) is possible, but becomes impractical for large number of files, and require cache management in the application.
So I decided to handle the different access modes for famundo in two separate ways. The private files requiring authentication for access will be delivered directly from the controllers of famundo, using send_file(). The public files will be served directly by the web server.
To make sure I don't have to copy and duplicate every file I want to share, I just symlink from the public area into the private area files. A note here: NEVER EVER point to a directory this way - only to files. If you point to a directory you open yourself up to directory traversal attacks.
So for a simple system, this wil be it. But for a system hosting large number of distinct users, each with his/her own data, some more changes are needed. Each family hosted in famundo will have it's own files and directories, and we need to keep them separate. And because we don't want to keep the files with the original file name, but only with some internal ID (keeps managemet MUCH easier, and obliviates the need for file name sanitation), we need an easy way to get at the file but still return the correct name to the browser. Enter mod_rewrite. Please note that I'm describing lighttpd mod_rewrite and not apache's. A similar thing can be done with apache, but the syntax is different.
For this example, we will have a family hosted as smith3.famundo.com. And the files we publish for this family will we internaly stored at: /data/private_data/families/s/m/smith3/files/ (Side note: the s and m before the full family name serve to distribute the families over multiple directories - I can explain the reasoning if there's an interest). Files are also distributed using part of the file ID. The file my_amazing_picture.png, that has ID 835 in the database, will be stored under /3/5/835.png. So the full path to the file will be: /data/private_data/families/s/m/smith3/files/3/5/835.png. In the public direcotry of rails, we add a directory structure for the files:
public/family_files/s/m/smitsh3/files, and in it we store the symlinks to the private files. So our famous picture will be a symlink:
public/family_files/s/m/smitsh3/files/3/5/835.png ->
/data/private_data/families/s/m/smith3/files/3/5/835.png.
But now we want a nice URL. We don't want to point every picture at this long ugly URL. mod_rewrite to the rescue! First thing first, enable mod_rewrite in the lighttpd config. Then add the following "black-magic" entry to the config file:
# Incoming URL: http://smith3.famundo.com/pub/file/835/my_file.png
# translated to: /published_files/s/m/smith3/file/3/5/835.png
$HTTP["host"] =~ "((.)(.)(.*)).famundo.com" {
url.rewrite-once = ( "/pub/(.+)/(\d*(\d)(\d))/.*(..*)$" =>
"/published_files/%2/%3/%1/$1/$3/$4/$2$5" )
}
A nice side effect of this trick is that we can give the file the true name in the URL. We don't even need to sanitize it, because we discard it when we retrieve the file. We retrieve the ID based name, but to the browser it appeared named correctly.
Some things to keep in mind:
- The family name need to be atleast 2 characters long.
- The file name based on the ID need to be atleast 2 digits. You can always use sprintf "%02d", id for that.
- Sharing a file is only creating a symlink in the published structure. Unsharing is just a delete of the same symlink.
- Different mappings can be created as long as the mapping can be represented in a regular expression.
- A more advanced mapping can be created by using the Lua language with lighttpd config file.
Posted in lighttpd, Rails | no comments | no trackbacks
Posted by Guy Naor
Mon, 06 Mar 2006 12:40:00 GMT
You already know I ABSOLUTELY love Capistrano (well, I do prefer SwitchTower as a name...). I also count security as a critical component in any production system (and no, having a firewall doesn't count as security). So to make sure I can comfortably use it in production, I had to make some security adjustment on the machines I deploy to.
Some important requirments:
- The web server should only have read access to the absolute minimum it really needs. In my case, as I'm using FastCGI processes launched from outside the server, only the public directory needs to be readable by it.
- The deployment user (set :user "deployer" in deploy.rb) should be a distinct user and should be the only one with write privileges to the rails application directory.
- The rails application runner will be itself a distinct user with only read access to the rails application.
- All other users can't even read the deployment directories.
- The deployment user can run as root using sudo only the minimum number of things like restarting the web server, or changing configuration.
To satisfy those requirement I created 2 users - one to deploy and one to run the rails application. I then set the privileges on the files (using chmod) so that the runner can read everything in the rails app directories and write just to the log directory. And the web server can read only the public directory. Then I edited the sudoers file (use visudo and direct editing to catch syntax errors) and allowed the deployment user rights to launch/stop/reload the web server, and to copy some configuration files. Note that it can't write to the files. It can just copy them from one specific location to another. It's risky letting someone to sudo run shell scripts, as then they can do anything like root. Here's a piece of my sudoers file:
Cmnd_Alias CP_LIGHTY_CONF = \
/bin/cp /etc/lighthttpd/maintenance.conf /etc/lighttpd/lighttpd.conf, \
/bin/cp /etc/lighttpd/application.conf /etc/lighttpd/lighttpd.conf, \
/etc/init.d/lighttpd
deployer localhost = CP_LIGHTY_CONF
This gives the deployer user the rights to copy the config files around, but not to ever write to them. So that the admin on the box set them as needed, but the deployer can change them as needed.
Please note that this is only one layer of the security. Just like the firewall is just a single layer. When planning a deployment, think of security as a layered mechanism, adding more and more layers (firewalls, host firewalls, privileges, monitoring, IDS, SELinux, etc...).
Posted in Rails, Linux | no comments | no trackbacks
Posted by Guy Naor
Sun, 05 Mar 2006 06:51:00 GMT
The spawner script in rails is pretty cool in that it will keep looking out for the FastCGI processes and make sure they are all working.
What I don't like about it, is that it launches the dispatcher again and again, letting it fail when the socket is already open. This is doing the unnecessary and puts more load on the system. But most of all it's just ugly!
As we're in a ruby script anyway, why not use it to see if the socket is in use before we launch the dispatcher? The only thing that needs to be changed is the spawn method (and you need to add a require 'socket' at the top of the script). We try to open a listening socket on the port we're being passed. If it opens, it means no process is listening on it and we can launch the dispatcher. If it is in use, an exception will be raised, and we just catch it, print a YES, and we're done.
def spawn(port)
print "Checking if something is already running on port #{port}..."
begin
srv = TCPServer.new('0.0.0.0', port)
srv.close
srv = nil
print "NO\n "
print "Starting FCGI on port: #{port}\n "
system("#{OPTIONS[:spawner]} -f #{OPTIONS[:dispatcher]} -p #{port}")
rescue
print "YES\n"
end
end
Now it will not even get to the dispatcher if the socket is already in use.
Posted in Rails, Ruby | 2 comments | no trackbacks
Posted by Guy Naor
Sat, 04 Mar 2006 00:59:00 GMT
I really like switchtower and the ease it brings to rails deployment. Suddenly deployment is just as easy as rails development. This is especially true for the multi-server deployment I'm working on.
But there was one small problem. My firewall doesn't give ANY direct access to the internal machines. All connections are NATed into the internal machines. This includes even ssh connections, which are NATed from different ports into different machines. So to the outside they all look like the same server (for example deploy.famundo.com), but a collection of ports redirect ssh connections to specific internal machines. For example: deploy.famundo.com:22222 => app1:22, deploy.famundo.com:22223 => web1:22. This clashes with the way switchtower works, as it expects ssh to be running on one specific port. Even using the ssh_options[:port] in deploy.rb will use the same port for all servers.
To solve this, I added support in switchtower for assigning an optional specific port to each server. If non is given, the default will be used. Assigning different ports is very easy:
role :db, "app1" # Will use port 22
role :app, "deploy.famundo.com:2224"
role :web, "deploy.famundo.com:2223"
The change is a simple addition to switchtower/lib/switchtower/ssh.rb (in svn diff format):
--- ssh.rb (revision 3755)
+++ ssh.rb (working copy)
@@ -23,6 +23,11 @@
methods = [ %w(publickey hostbased), %w(password keyboard-interactive) ]
password_value = nil
+ # If the server has a port assigned to it (www.sample.com:2222), use it, and clean the server name
+ if server =~ /(.+):(\d{1,5})/
+ server, port = $1, $2
+ end
+
begin
ssh_options = { :username => config.user,
:password => password_value,
Now I can use switchtower, and still keep my firewall setup! Sweet!
I'll post a ticket with this patch to the rail trac.
Posted in Rails | no comments | no trackbacks
Posted by Guy Naor
Wed, 01 Feb 2006 00:19:00 GMT
Here's a little something for rails novices. Might save you some time and pain.
I wanted nice URLs for my application, and one of the important parts was really short URLs for the main parts of the application. So you can do: http://www.famundo.com/library and get to the library module. Or http://www.famundo.com/photos to get to the library/photos area of the application.
That's easy with routes. In routes.rb I just had to add entries like:
map.connect 'library' , :controller => 'library_center', :action => 'show'
map.connect 'photos' , :controller => 'my_pictures', :action => 'show'
map.connect 'blog' , :controller => 'personal_blogs', :action => 'show'
But my system has a further complication. I use modules to separate the different parts of the application, so that everything is nicely organized in development. So for my maps I used:
map.connect 'library' , :controller => 'library/library_center', :action => 'show'
map.connect 'photos' , :controller => 'library/my_pictures', :action => 'show'
map.connect 'blog' , :controller => 'library/personal_blogs', :action => 'show'
At first it seemed to work fine, but when I got deeper in to the controllers actions, the resulting URLs where all messed up. After some work and some help on the list, the solution was VERY simple. I was missing an initial slash in my routes! When referencing the controller inside the module, I needed to add a leading slash. So that the routes are now working with:
map.connect 'library' , :controller => '/library/library_center', :action => 'show'
map.connect 'photos' , :controller => '/library/my_pictures', :action => 'show'
map.connect 'blog' , :controller => '/library/personal_blogs', :action => 'show'
Ah, what a small {color:red}/ can do!
Posted in Rails | no comments | no trackbacks
