{"id":2749,"date":"2012-06-08T18:37:15","date_gmt":"2012-06-08T18:37:15","guid":{"rendered":"http:\/\/www.gubatron.com\/blog\/?p=2749"},"modified":"2012-06-08T18:37:15","modified_gmt":"2012-06-08T18:37:15","slug":"deploying-html5-apps-on-cloudfront-with-efficient-invalidation-requests","status":"publish","type":"post","link":"https:\/\/www.gubatron.com\/blog\/deploying-html5-apps-on-cloudfront-with-efficient-invalidation-requests\/","title":{"rendered":"Deploying HTML5 apps on CloudFront with efficient invalidation requests"},"content":{"rendered":"

So you decided to build your next web app\/site using nothing but HTML5 and Javascript.
\nNo server side processing for anything related to UI.<\/p>\n

This means you will be coding a lot of JavaScript.<\/p>\n

Wouldn’t it be nice to put all that static HTML and JS on your CloudFront CDN and not deal with web servers?<\/p>\n

Once you deploy it’s nice, but Amazon Cloudfront is not meant for you to be invalidating objects all the time, they actually charge you $0.005 per each invalidated file.<\/p>\n

They tell you to version your files, but if you’re going for a no webserver based deployment, 100% cloudfront deployment, you can’t play around with url rewrites or redirects, so you’ll want to invalidate the files that have changed.<\/p>\n

Here’s a script to invalidate only the files that have changed.<\/p>\n

Our configuration<\/strong>
\nCloudfront pointed to a custom origin.<\/p>\n

The custom origin web server Document Root points to
\n\/var\/www\/mysite\/production<\/p>\n

‘production’ is a symlink pointing to a folder inside \/var\/www\/mysite, in there we have copies of the entire website saved under version numbers, for example<\/p>\n

[bash]\/var\/www\/mysite\/1.0.1
\n\/var\/www\/mysite\/1.0.2
\n\/var\/www\/mysite\/1.0.3 -> production\/
\n\/var\/www\/mysite\/1.0.4[\/bash]<\/p>\n

What our script does, is it uses rsync to compare the files that changed between the current production directory (in this case 1.0.3 with the next version 1.0.4), and dump the new files on a temporary folder \/var\/www\/mysite\/tmp<\/p>\n

We then create a list with all the files in the \/tmp folder, and for each 1,000 files that have changed we send Cloudfront an invalidation request and move the ‘production’ symlink to the new version.<\/p>\n

Instead of invalidating every file (which would be costly), we just invalidate what changed, and we don’t have to change any of our code to reflect version numbers.<\/p>\n

All of this is handled by command line tools that let us do this in one step (one step deployment)<\/p>\n

Here’s some of the python code we use to get the list of files that changed in order to build the invalidation request with the minimum files necessary so Jeff Bezos doesn’t take all our money.<\/p>\n

[python]
\ndef getFilesToInvalidate(config, newVersion):
\n ”’Compares the files in the new version folder with what’s currently in production.
\n returns a list of the files that changed or got removed.”’
\n checkKey(config,’website.exports.path’)<\/p>\n

oldFiles = config[‘website.exports.path’] + os.path.sep + ‘production’
\n diffFiles = config[‘website.exports.path’] + os.path.sep + ‘diff’<\/p>\n

#the ..\/NEW_VERSION is important, the command needs relative paths to work
\n #rsync -av –compare-dest=..\/NEW_VERSION \/path\/to\/OLD_VERSION \/path\/to\/TEMP_CHANGED
\n rmDiffCmd = ‘rm -fr ‘ + diffFiles
\n cmd = ‘rsync -av –compare-dest=..\/%s %s\/ %s’ % (newVersion,
\n oldFiles,
\n diffFiles)
\n print cmd
\n os.system(rmDiffCmd)
\n os.system(cmd)
\n diffList = getFilesInPath(diffFiles)
\n result = []
\n for i in xrange(len(diffList)):
\n if diffList[i].find(‘#’)!=-1:
\n continue
\n result.append(diffList[i].replace(diffFiles,”))
\n os.system(rmDiffCmd)<\/p>\n

return result
\n[\/python]<\/p>\n

Then use the ‘result’ list to create your CloudFront invalidation request<\/a>.<\/p>\n

Screw server side processing to render web pages, clients can do it now with client-side templating and javascript.
\nAnd oh Amazon, you guys need to step up your CloudFront Console features, it’d be awesome to invalidate paths and what not, it’s ultimately about what the customer needs.<\/p>\n

Here’s toast to a faster cloud, cheers.<\/p>\n","protected":false},"excerpt":{"rendered":"

So you decided to build your next web app\/site using nothing but HTML5 and Javascript. No server side processing for anything related to UI. This means you will be coding a lot of JavaScript. Wouldn’t it be nice to put all that static HTML and JS on your CloudFront CDN and not deal with web […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[15,1],"tags":[1156,1167,1168,1162,1166,1163,1165,1164],"class_list":["post-2749","post","type-post","status-publish","format-standard","hentry","category-code","category-uncategorized","tag-aws","tag-cdn","tag-cloud-development","tag-cloudfront","tag-frugal","tag-invalidation-request","tag-minimal","tag-optimal"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5Unzf-Il","jetpack-related-posts":[{"id":2747,"url":"https:\/\/www.gubatron.com\/blog\/s3cmd-how-to-put-files-with-expiration-on-s3cloudfront-aws-s3\/","url_meta":{"origin":2749,"position":0},"title":"s3cmd – how to put files with expiration on S3\/CloudFront #AWS #s3","author":"gubatron","date":"May 23, 2012","format":false,"excerpt":"[bash]s3cmd put --add-header \"Cache-Control: 604800\" -P .\/* s3:\/\/mybucket.com\/path\/to\/files\/[\/bash] If you have your CloudFront Distribution pointed to your S3 bucket, files will expire after the number of seconds specified in Cache-Control have passed. You will need the wonderful s3cmd tool","rel":"","context":"In \"aws\"","block_context":{"text":"aws","link":"https:\/\/www.gubatron.com\/blog\/tag\/aws\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1324,"url":"https:\/\/www.gubatron.com\/blog\/html-5-is-out-and-about\/","url_meta":{"origin":2749,"position":1},"title":"HTML 5 is out and about","author":"gubatron","date":"July 11, 2009","format":false,"excerpt":"I believe that the ,