Speed up Travis CI builds by caching the bundle to S3

When we run our build suites on Travis Pro the bundling step takes the most time by a wide margin (aside from the test script itself).

Inspired by this Coderwall protip by Michał Czyż, I set about attempting to cache our completed gem bundle on S3.

Update: Checkout the bundle_cache gem for an improved version of this technique

How it works

The tarballed bundle and a SHA-2 sum of the Gemfile.lock are downloaded from S3 (if they exist)
Travis runs bundle install as normal (except that the bundle is installed to ~/.bundle). This shouldn’t take more than a few seconds if the bundle hasn’t changed.
Travis executes all the normal build steps, as usual
The bundle is tarballed and uploaded to S3 (us-east is the closest region to the new Travis workers), but only if the SHA-2 hash for the Gemfile.lock has changed

The bundle is uploaded with public-read permissions for easier downloading. This should not be a problem for most people and could be mitigated by using a really obscure filename, like a UUID.

Step by step instruction

Set up a bucket on S3 in the US Standard region (us-east-1) (and possibly a new user via IAM)
Install the travis gem with gem install travis
Log into Travis with travis login --auto (from inside your project respository directory)
Encrypt your S3 credentials with: travis encrypt AWS_S3_KEY="" AWS_S3_SECRET="" --add (be sure to add your actual credentials inside the double quotes)
Add the bundle_cache.rb and bundle_install.sh files from below to the script/travis folder
Modify your .travis.yml file to match the .travis.yml below

The BUNDLE_ARCHIVE variable will be used as the base for the uploaded bundle’s name
Setting AWS_S3_REGION is optional and defaults to us-east-1
Pay special attention to these keys:
- bundler_args
- env.global
- before_install
- after_script

If you have any questions or comments, you can comment on my gist, or tweet at me on Twitter.

	bundler_args: --without development --path=~/.bundle
	language: ruby
	rvm:
	- 1.9.3
	env:
	global:
	- BUNDLE_ARCHIVE="your-bundle-name"
	- AWS_S3_REGION="us-east-1"
	- AWS_S3_BUCKET="your-bucket-name"
	- RAILS_ENV=test
	- secure: "A_VERY_LONG_SERIES_OF_CHARACTERS_HERE"

	# Ensure Bundler >= 1.1, don't install rdocs, and fetch a cached bundle from S3
	before_install:
	- "echo 'gem: --no-ri --no-rdoc' > ~/.gemrc"
	- gem install bundler fog
	- "./script/travis/bundle_install.sh"

	before_script:
	- "cp config/database.example.yml config/database.yml"

	after_script:
	- "ruby script/travis/bundle_cache.rb"

	script: "bundle exec rake db:create db:test:load spec"

view raw .travis.yml hosted with ❤ by GitHub

	# encoding: UTF-8

	require "digest"
	require "fog"

	bucket_name = ENV["AWS_S3_BUCKET"]
	architecture = `uname -m`.strip

	file_name = "#{ENV['BUNDLE_ARCHIVE']}-#{architecture}.tgz"
	file_path = File.expand_path("~/#{file_name}")
	lock_file = File.join(File.expand_path(ENV["TRAVIS_BUILD_DIR"]), "Gemfile.lock")
	digest_filename = "#{file_name}.sha2"
	old_digest = File.expand_path("~/remote_#{digest_filename}")

	puts "Checking for changes"
	bundle_digest = Digest::SHA2.file(lock_file).hexdigest
	old_digest = File.exists?(old_digest) ? File.read(old_digest) : ""

	if bundle_digest == old_digest
	puts "=> There were no changes, doing nothing"
	else
	if old_digest == ""
	puts "=> There was no existing digest, uploading a new version of the archive"
	else
	puts "=> There were changes, uploading a new version of the archive"
	puts " => Old checksum: #{old_digest}"
	puts " => New checksum: #{bundle_digest}"
	end

	puts "=> Preparing bundle archive"
	`cd ~ && tar -cjf #{file_name} .bundle && split -b 5m -a 3 #{file_name} #{file_name}.`

	parts_pattern = File.expand_path(File.join("~", "#{file_name}.*"))
	parts = Dir.glob(parts_pattern).sort

	storage = Fog::Storage.new({
	:provider => "AWS",
	:aws_access_key_id => ENV["AWS_S3_KEY"],
	:aws_secret_access_key => ENV["AWS_S3_SECRET"],
	:region => ENV["AWS_S3_REGION"] \|\| "us-east-1"
	})

	puts "=> Uploading the bundle"
	puts " => Beginning multipart upload"
	response = storage.initiate_multipart_upload bucket_name, file_name, { "x-amz-acl" => "public-read" }
	upload_id = response.body['UploadId']
	puts " => Upload ID: #{upload_id}"

	part_ids = []

	puts " => Uploading #{parts.length} parts"
	parts.each_with_index do \|part, index\|
	part_number = (index + 1).to_s
	puts " => Uploading #{part}"
	File.open part do \|part_file\|
	response = storage.upload_part bucket_name, file_name, upload_id, part_number, part_file
	part_ids << response.headers['ETag']
	puts " => Uploaded"
	end
	end

	puts " => Completing multipart upload"
	storage.complete_multipart_upload bucket_name, file_name, upload_id, part_ids

	puts "=> Uploading the digest file"
	bucket = storage.directories.new(key: bucket_name)
	bucket.files.create({
	:body => bundle_digest,
	:key => digest_filename,
	:public => true,
	:content_type => "text/plain"
	})
	end

	puts "All done now."
	exit 0

view raw bundle_cache.rb hosted with ❤ by GitHub

	#!/bin/sh

	ARCHITECTURE=`uname -m`
	FILE_NAME="$BUNDLE_ARCHIVE-$ARCHITECTURE.tgz"

	cd ~
	wget -O "remote_$FILE_NAME" "https://$AWS_S3_BUCKET.s3.amazonaws.com/$FILE_NAME" && tar -xf "remote_$FILE_NAME"
	wget -O "remote_$FILE_NAME.sha2" "https://$AWS_S3_BUCKET.s3.amazonaws.com/$FILE_NAME.sha2"

	exit 0

view raw bundle_install.sh hosted with ❤ by GitHub