category s3

cp2s3 - A Script for Smart Copy Into Amazon's S3

Posted by Guy Naor Mon, 12 Mar 2007 09:01:00 GMT

Following the explanation on how to load compressed content into S3, here is a tool to make it easy.

cp2s3 is a ruby script that make it easy to upload whole directory structures into S3 buckets. Some of the more important features (all controlled with command line switches):

Compress specified files types (i.e. compress js and css files, but leave images alone)
Recursive copy to load complete directory structures
Use Ruby's Dir::glob for very complex file selections
Save sha1 digest, size and modification date to the item meta-data on Amazon
Upload only changed files (based on the sh1 digest)
Upload to multiple buckets at once
Mark files as public readable
The S3 keys can be assigned through the command line, envirnment variables, or embedded in the script.

Installing it requires the installation of the AWS::S3 Ruby library, easiers done with

sudo gem install aws-s3

To use the digest switch or the modified only option, you will need the sh1sum command line utility (or write your own that will return a unique file digest).

Run it with no parameters to get a full list of options. And let me know if you would like to have anything added or modified. It was tested in production, uploading more than 80,000 files in one session into S3. (I use it regularly to upload Famundo assets into S3.)

Click on read more to see some usage examples and the full source, or just download the attached script. I just put it in my /usr/bin folder and rename it to just cp2s3.Then I can run it just like any other command.

A New Face for Famundo

Posted by Guy Naor Thu, 08 Mar 2007 05:46:00 GMT

Finally after lots of work, the new Famundo site is up an running. I believe it's VERY VERY nice. But go check it out! And do sign up for Famundo while you are there ;-).

We also have the new mascots - same family as the one I have with the logo here on the blog. They are officially called the Famundudes (Famundad, Famunmom, Famunkids, and and the members of the community they interact with).

There's an interesting story behind the site - the site is a fully page-cashed Rails application. And it has a small admin interface to manage all the changeable contents - like news and such. But the server I wanted to put it on started to miss-behave. So we generated all the needed pages by just accessing all the functionality, took the generated pages and uploaded it like a static content. Here's for you a nice little trick with rails for static sites.

The site that will run live with rails is coming soon. And it has a few tricks up it's sleeve - like the ability to switch serving of assets from our servers or from S3, and the ability to cache the html directly into S3. I'll write more about it when it's up.

Please go visit the new site and sign up for Famundo.

Posted in Rails, S3 | no comments

Serving Compressed Content from Amazon's S3

Posted by Guy Naor Fri, 02 Mar 2007 11:46:00 GMT

If you have yet to check out Amazon's S3 service, go do that now. I'll wait for you to come back! It's a very simple storage server that is also really really cheap, and high-performance. Backed by Amazon's network, it's also pretty reliable.

I am in the process of moving a lot of the static content of Famundo into the service. But one important requirement is serving JavaScript and CSS files compressed, as we have pretty big files for both.

By default S3 will serve the files as uncompressed text files, increasing the load time for the clients, so a solution for compressed files serving was needed.

When saving files on S3 I can pass in HTTP headers that will be returned when the file is accessed. Using this along with pre-compressed JS and CSS files, we can have S3 serve the files compressed. The limitation here as opposed to serving it directly with a web-server, is that there is no content type negotiation, meaning, the files will always be served compressed. So if the clients accessing your files cannot accept compression, you are out of luck, and got to either serve uncompressed or do it from your content negotiating server. In Famundo we target only newer browsers (FF, IE6+ and Safari 1.2+), so pre-compressed files works for us.

Enough words, time for some code. I'm showing it using the Ruby AWS::S3 library, but you can use whichever library/language you want. The principle is the same. What we are doing, is compressing the files, then uploading the compressed files and the most important part - assigning to it the correct HTTP headers.

require 'rubygems'
require 'aws/s3'
require 'stringio'
require 'zlib'

AWS::S3::Base.establish_connection!(:access_key_id => 'YOUR S3 ACCESS KEY', :secret_access_key => 'YOUR S3 SECRET KEY')

strio = StringIO.open('', 'w')
gz = Zlib::GzipWriter.new(strio)
gz.write(open('test.css').read) # Here is the file on your file system
gz.close
# 'test.css' is the name of the file on the S3 system, and 'the_bucket' is the bucket name
# Note the use of "Content-Encoding" => 'gzip', this is what make it all work
S3Object.store('test.css', strio.string, 'the_bucket', :access => :public_read, "Content-Encoding" => 'gzip' )