Cache busting with Grunt

In my previous post I talked about using Grunt to automate my build and deploy. In this post I'll talk about using Grunt to cache-bust my webapp.

Requirement

• I want to maximise the use of caching for JS and CSS files (with long expiry periods set in .htacess), but if a file is updated, the newer file should be requested instead of the cached version
• I use requireJS to conditionally load modules - these modules require cache busting, but CDN hosted library resources like jquery does not.
• I prefer to keep filenames the same for different builds. My reasoning is that an older version of a revved file (eg. script.123.js instead of script.js) would be deleted when there is a new build (eg. script.124.js), and if there are any cached references somewhere pointing to the older version, the server would return 404. Perhaps this fear is unfounded if .htacess is set up to never cache .html files?

Findings

I found a few half solutions, but unfortunately not one elegant solution that fits all my requirements

1. Use RequireJS urlArgs to cache-bust JS files

In requireJS config, append a url parameter with urlArgs to each URL that requireJS loads. I use @@hashcache as a placeholder for the build hash, generates the hash in my Gruntfile and then uses grunt-replace to replace it with a generated hash.
Good:

• Cache-busts all requireJS conditionally loaded modules 

Bad:

• urlArgs is not recommended for production
• All scripts loaded will have this urlArgs appended and it will cache-bust CDN resources. This means the caching benefit is lost for a widely used CDN-loaded resource like jquery
• Another solution is required for cache-busting CSS files, and also the first file requireJS loads via data-main="" in index.html

2. Rev file references and modify rewrite rules

I took notes from here and here for a solution that revs file references but keep actual filenames unchanged.

1. Use grunt-hashres to rev and replace references to JS & CSS files, but don't rename actual files

Index.html references revved CSS & JS files: 2. Update .htaccess rewrite rule to point revved to original files


Good:

• hash is generated for each file - if a file isn't changed, its hash isn't updated - maximises the advantage of caching

Bad:

• Does not rev JS modules loaded by RequireJS 
• I have a feeling my rewrite rule may be a tad too broad? Could this accidentally effect other files?

3. Cache-busting requireJS modules

The above solution is able to cache-bust anything except for conditionally loaded requireJS modules not referenced in index.html. For each of these modules, I need to:
1. Configure module path in requireJS config so it'll load a revved file path, with the original path as backup

2. Configure module path in Gruntfile so the optimiser knows where to look for the module

3. Use grunt-replace to replace references to @@cachehash with a hash for each build

Good:

• cache-busts without using urlArgs; Only busts files that require it, and not CDN files

Bad:

• require fiddly manual configuration for each conditionally-loaded file

• Same hash is used for all files per build. So even if a file hasn't updated since last acess, it will still be reloaded. Not a big issue but not as cache-optimised as the previous solution.

Conclusion

I'm using both 2 and 3 for my project. Although the cobined solution covered all my requirements, I think this is still a work in progress as  it feels clunky with all the configuration required for 3. What I need is a grunt task that does file revving with support for requireJS! I'll continue to investigate, and would appreciate any suggestions!

Build and deploy with Grunt

I've recently started using Grunt for building and deploying my webapp. I won't go through how to get Grunt up and running but rather talk about my experience with configuring grunt tasks for my project.

Quick note about the project

The Listing Generator is a webapp that generates HTML listings to be pasted into eBay. The project made up of two sections.

• main - the webapp tool for generating HTML eBay listings; this tool is not customer facing
• ebay - scripts & stylesheet that the generated eBay listings reference; these are loaded by customers when they view a eBay listing and is separately hosted from main. 

These two sections are hosted separately but are fairly tightly coupled so i've kept them in the same project.

 My build process

1. Validate scripts
2. Clean
3. Copy files
4. Optimise JS
5. Optimise CSS
6. Optimise images
7. Cache-busting
8. Deployment

1. Validate scripts

Task: grunt-contrib-jshint
Jshint provides many configuration options which can be specified in Gruntfile or a .jshintrc file. I found that .jshintrc was not a good option if you want different configurations for different scripts - as overriding/merging the settings from the .jshintrc file is not possible - it's all or nothing! Also .jshintrc is JSON format so comments are a bit tricky. I prefer to specify a set global options and override it in the subtasks all within the Gruntfile. The downside is the Gruntfile will be a bit more bulky.

Jshint also generates a lot of formatting errors about missing semicolon, trailing spaces, inconsistent indentation etc which i suppress as i'm not too concerned about formatting. The ones i founded helpful: "undef" which catches leaking global variables, "unused" which catches variables declared but not used. Here's the configuration options I've used:

jshint: {
 options:{
  // enforcing options
  "bitwise": true,
  "camelcase": false,
  "curly": false,
  "eqeqeq": false,
  "immed": true,
  "latedef": true,
  "newcap": true,
  "noarg": true,
  "regexp": true,
  "undef": true,
  "unused": true,
  "strict": true,
  "trailing": false, // trailing whitespace

  // relaxing options
  "asi":true, // missing semicolon
  "eqnull":true,
  "esnext": true, // ECMAScript 6 warning
  "laxbreak": true,// unsafe line breakings
  "smarttabs":true, // mixed tabs & spaces
  "sub":true,  // square brackets instead of dot notation

  // environments
  "browser": true,

  // supress other warnings - I suppress these as I use truthy/falsy comparison a lot
  '-W041': true, // strict compare
  '-W099':true // Use '===' to compare with '0'
 },
 grunt:{
  options:{
   "node": true // node environment
  },
  files:{
   src:['Gruntfile.js']
  }
 },
 main: { // non-customer-facing scripts - can be less strict

  //- turning on devel & nonstandard to allow console.log() and escape()
  options:{
   'devel':true,

   'nonstandard':true,
   ignores:['src/scripts/lib/**'], // don't lint library files
   globals:{
    require:false, // used by requirejs
    define:false,
    gapi:false // used by google analytics
   }
  },
  files:{
   src:['src/scripts/**/*.js',
   'test/spec/**/*.js']
  }
 },
 ebay: { //customer-facing scripts
  options:{
   globals:{
    "GH_config":false, // ebay specified globals
    "domain":false,
    "ebayItemID":false,
    "eBayTRItemId":false,
    "ActiveXObject":false,
    "_rg":false      
   }
  },
  files:{
   src:['ebay/**/*.js']
  }
 } 

2. Clean distribution folder

Task: grunt-contrib-clean
I setup subtasks for cleaning all, or just 'main' or 'ebay' part of the project

clean: {
 all:['dist'],
 main:['dist/main'],
 ebay:['dist/ebay']
}

3. Copy files that don't require optimisation

Task: grunt-copy-to
I first tried to use grunt-contrib-copy which is maintained by the Grunt team, but couldn't get it to work consistently. It would sometimes fail silently by creating folders but not copy any files over. I spent ages debugging to no avail. Once I switched over to grunt-copy-to it worked as expected. grunt-copy-to also has the advantage of only copying newer files - making the build process faster.
As most of the source files are optimised and copied by other tasks, I just needed this task to copy everything in the deploy folder to dist/main.

copyto: {
 main: {
  files: [{cwd:'deploy/',src: ['.*','**'], dest: 'dist/main/'}]
 }
}

Note: '**' does not match an extension only file (eg. .htacess) so use ['.*','**'] to match everything in a directory

4. Optimize JS

Tasks: grunt-contrib-requirejs, grunt-contrib-uglify

'Main' is a requireJS project with text plugin and I uses grunt-contrib-requirejs with the inlineText option enabled for optimisation to generate two Javascript modules: bootstrap.js and drive.js (conditionally loaded from bootstrap.js)

requirejs: {
 main:{
  options:{
   baseUrl: "src/scripts/",
   mainConfigFile: "src/scripts/bootstrap.js",
   dir: "dist/main/scripts/",
   paths:{
    jquery:'empty:', // jquery load via CDN - don't optimise
    handlebars:'empty:',
   },
   modules:[{
    name:'bootstrap',
    exclude:['jquery','handlebars']
   },{
    name:'drive',
    exclude:['jquery','handlebars','bootstrap'] // exclude bootstrap as this script is conditional-loaded from bootstrap
   }],
   inlineText:true, // use text plugin
   useStrict: true,
   removeCombined:true // don't copy combined js to dist
  }
 }
}

I used grunt-contrib-uglify for "ebay" project as it does not use requirej.

uglify: {
 ebay: {
  options: {
   banner: '<%= banner %>'
  },
  files:{
   'dist/ebay/info.js':'src/ebay/info.js',
   'dist/ebay/rg.js':'src/ebay/rg.js'
  }
 }
}

5. Optimise CSS

Task: grunt-contrib-cssmin
As I only have one CSS file each for each sub-project, it's a fairly straightforward css minification.

cssmin: {
 main:{
  files: {
   'dist/main/styles/styles.css': 'src/styles/styles.css'
  }
 },
 ebay:{
  files: {
   'dist/ebay/styles.css': 'src/ebay/styles.css'
  }
 }
}

For other projects where I use SASS I use grunt-contrib-compass.

6. Optimise images

Task: grunt-contrib-imagemin
You can set optimisation levels for the images - I just leave it at the default setting

imagemin: {
 ebay: {
  files: [{
   expand: true,
   cwd: 'src/ebay',
   src: '{,*/}*.{gif,jpeg,jpg,png}',
   dest: 'dist/ebay'
  }]
 }
}

7.Cache-busting

This was slightly tricky and subject to another blog post!

8. Deploy on to server with FTP

Task: grunt-ftp-deploy
Deployment via FTP. The login info is stored in a .ftppass file which should be excluded from git. I have  'ebay' build deploying to a versioned directory, which makes rollback easier when required! Also don't add this task to the 'default' grunt task just in case you deploy something accidentally! I also increment the version with npm version before i deploy.

'ftp-deploy': {
 main: {
  ...
 },
 ebay: {
  auth: {
   host: 'ftp.example.com',
   port: 21,
   authKey: 'key1' // username/password stored in .ftppass
  },
  src: 'dist/ebay',
  dest: '/path/to/server/<%= pkg.version %>', // deploy as the current package.json version
 }
}

Other things to add to my Gruntfile

• Add watch tasks for starting the server and autorefresh on file change
• Add unit-tests with grunt-mocha-phantom I've been putting this off, which is bad :-/

Tips & resources

1. Grunt website's sample gruntfile is a good place to start
2. Yo webapp generator generates a great Gruntfile, although it does a lot of things and probably needs better documentation to explain what each task does!
3. David Tucker's blog post on Grunt is very detailed helped me a lot in setting up my Gruntfile.
4. Use load-grunt-tasks to replace all those grunt.loadNpmTasks(...)statements in the Gruntfile.
   Just add require('load-grunt-tasks')(grunt)