Cache busting with Grunt

In my previous post I talked about using Grunt to automate my build and deploy. In this post I'll talk about using Grunt to cache-bust my webapp.

Requirement

• I want to maximise the use of caching for JS and CSS files (with long expiry periods set in .htacess), but if a file is updated, the newer file should be requested instead of the cached version
• I use requireJS to conditionally load modules - these modules require cache busting, but CDN hosted library resources like jquery does not.
• I prefer to keep filenames the same for different builds. My reasoning is that an older version of a revved file (eg. script.123.js instead of script.js) would be deleted when there is a new build (eg. script.124.js), and if there are any cached references somewhere pointing to the older version, the server would return 404. Perhaps this fear is unfounded if .htacess is set up to never cache .html files?

Findings

I found a few half solutions, but unfortunately not one elegant solution that fits all my requirements

1. Use RequireJS urlArgs to cache-bust JS files

In requireJS config, append a url parameter with urlArgs to each URL that requireJS loads. I use @@hashcache as a placeholder for the build hash, generates the hash in my Gruntfile and then uses grunt-replace to replace it with a generated hash.
Good:

• Cache-busts all requireJS conditionally loaded modules 

Bad:

• urlArgs is not recommended for production
• All scripts loaded will have this urlArgs appended and it will cache-bust CDN resources. This means the caching benefit is lost for a widely used CDN-loaded resource like jquery
• Another solution is required for cache-busting CSS files, and also the first file requireJS loads via data-main="" in index.html

2. Rev file references and modify rewrite rules

I took notes from here and here for a solution that revs file references but keep actual filenames unchanged.

1. Use grunt-hashres to rev and replace references to JS & CSS files, but don't rename actual files

Index.html references revved CSS & JS files: 2. Update .htaccess rewrite rule to point revved to original files


Good:

• hash is generated for each file - if a file isn't changed, its hash isn't updated - maximises the advantage of caching

Bad:

• Does not rev JS modules loaded by RequireJS 
• I have a feeling my rewrite rule may be a tad too broad? Could this accidentally effect other files?

3. Cache-busting requireJS modules

The above solution is able to cache-bust anything except for conditionally loaded requireJS modules not referenced in index.html. For each of these modules, I need to:
1. Configure module path in requireJS config so it'll load a revved file path, with the original path as backup

2. Configure module path in Gruntfile so the optimiser knows where to look for the module

3. Use grunt-replace to replace references to @@cachehash with a hash for each build

Good:

• cache-busts without using urlArgs; Only busts files that require it, and not CDN files

Bad:

• require fiddly manual configuration for each conditionally-loaded file

• Same hash is used for all files per build. So even if a file hasn't updated since last acess, it will still be reloaded. Not a big issue but not as cache-optimised as the previous solution.

Conclusion

I'm using both 2 and 3 for my project. Although the cobined solution covered all my requirements, I think this is still a work in progress as  it feels clunky with all the configuration required for 3. What I need is a grunt task that does file revving with support for requireJS! I'll continue to investigate, and would appreciate any suggestions!

Build and deploy with Grunt

I've recently started using Grunt for building and deploying my webapp. I won't go through how to get Grunt up and running but rather talk about my experience with configuring grunt tasks for my project.

Quick note about the project

The Listing Generator is a webapp that generates HTML listings to be pasted into eBay. The project made up of two sections.

• main - the webapp tool for generating HTML eBay listings; this tool is not customer facing
• ebay - scripts & stylesheet that the generated eBay listings reference; these are loaded by customers when they view a eBay listing and is separately hosted from main. 

These two sections are hosted separately but are fairly tightly coupled so i've kept them in the same project.

 My build process

1. Validate scripts
2. Clean
3. Copy files
4. Optimise JS
5. Optimise CSS
6. Optimise images
7. Cache-busting
8. Deployment

1. Validate scripts

Task: grunt-contrib-jshint
Jshint provides many configuration options which can be specified in Gruntfile or a .jshintrc file. I found that .jshintrc was not a good option if you want different configurations for different scripts - as overriding/merging the settings from the .jshintrc file is not possible - it's all or nothing! Also .jshintrc is JSON format so comments are a bit tricky. I prefer to specify a set global options and override it in the subtasks all within the Gruntfile. The downside is the Gruntfile will be a bit more bulky.

Jshint also generates a lot of formatting errors about missing semicolon, trailing spaces, inconsistent indentation etc which i suppress as i'm not too concerned about formatting. The ones i founded helpful: "undef" which catches leaking global variables, "unused" which catches variables declared but not used. Here's the configuration options I've used:

jshint: {
 options:{
  // enforcing options
  "bitwise": true,
  "camelcase": false,
  "curly": false,
  "eqeqeq": false,
  "immed": true,
  "latedef": true,
  "newcap": true,
  "noarg": true,
  "regexp": true,
  "undef": true,
  "unused": true,
  "strict": true,
  "trailing": false, // trailing whitespace

  // relaxing options
  "asi":true, // missing semicolon
  "eqnull":true,
  "esnext": true, // ECMAScript 6 warning
  "laxbreak": true,// unsafe line breakings
  "smarttabs":true, // mixed tabs & spaces
  "sub":true,  // square brackets instead of dot notation

  // environments
  "browser": true,

  // supress other warnings - I suppress these as I use truthy/falsy comparison a lot
  '-W041': true, // strict compare
  '-W099':true // Use '===' to compare with '0'
 },
 grunt:{
  options:{
   "node": true // node environment
  },
  files:{
   src:['Gruntfile.js']
  }
 },
 main: { // non-customer-facing scripts - can be less strict

  //- turning on devel & nonstandard to allow console.log() and escape()
  options:{
   'devel':true,

   'nonstandard':true,
   ignores:['src/scripts/lib/**'], // don't lint library files
   globals:{
    require:false, // used by requirejs
    define:false,
    gapi:false // used by google analytics
   }
  },
  files:{
   src:['src/scripts/**/*.js',
   'test/spec/**/*.js']
  }
 },
 ebay: { //customer-facing scripts
  options:{
   globals:{
    "GH_config":false, // ebay specified globals
    "domain":false,
    "ebayItemID":false,
    "eBayTRItemId":false,
    "ActiveXObject":false,
    "_rg":false      
   }
  },
  files:{
   src:['ebay/**/*.js']
  }
 } 

2. Clean distribution folder

Task: grunt-contrib-clean
I setup subtasks for cleaning all, or just 'main' or 'ebay' part of the project

clean: {
 all:['dist'],
 main:['dist/main'],
 ebay:['dist/ebay']
}

3. Copy files that don't require optimisation

Task: grunt-copy-to
I first tried to use grunt-contrib-copy which is maintained by the Grunt team, but couldn't get it to work consistently. It would sometimes fail silently by creating folders but not copy any files over. I spent ages debugging to no avail. Once I switched over to grunt-copy-to it worked as expected. grunt-copy-to also has the advantage of only copying newer files - making the build process faster.
As most of the source files are optimised and copied by other tasks, I just needed this task to copy everything in the deploy folder to dist/main.

copyto: {
 main: {
  files: [{cwd:'deploy/',src: ['.*','**'], dest: 'dist/main/'}]
 }
}

Note: '**' does not match an extension only file (eg. .htacess) so use ['.*','**'] to match everything in a directory

4. Optimize JS

Tasks: grunt-contrib-requirejs, grunt-contrib-uglify

'Main' is a requireJS project with text plugin and I uses grunt-contrib-requirejs with the inlineText option enabled for optimisation to generate two Javascript modules: bootstrap.js and drive.js (conditionally loaded from bootstrap.js)

requirejs: {
 main:{
  options:{
   baseUrl: "src/scripts/",
   mainConfigFile: "src/scripts/bootstrap.js",
   dir: "dist/main/scripts/",
   paths:{
    jquery:'empty:', // jquery load via CDN - don't optimise
    handlebars:'empty:',
   },
   modules:[{
    name:'bootstrap',
    exclude:['jquery','handlebars']
   },{
    name:'drive',
    exclude:['jquery','handlebars','bootstrap'] // exclude bootstrap as this script is conditional-loaded from bootstrap
   }],
   inlineText:true, // use text plugin
   useStrict: true,
   removeCombined:true // don't copy combined js to dist
  }
 }
}

I used grunt-contrib-uglify for "ebay" project as it does not use requirej.

uglify: {
 ebay: {
  options: {
   banner: '<%= banner %>'
  },
  files:{
   'dist/ebay/info.js':'src/ebay/info.js',
   'dist/ebay/rg.js':'src/ebay/rg.js'
  }
 }
}

5. Optimise CSS

Task: grunt-contrib-cssmin
As I only have one CSS file each for each sub-project, it's a fairly straightforward css minification.

cssmin: {
 main:{
  files: {
   'dist/main/styles/styles.css': 'src/styles/styles.css'
  }
 },
 ebay:{
  files: {
   'dist/ebay/styles.css': 'src/ebay/styles.css'
  }
 }
}

For other projects where I use SASS I use grunt-contrib-compass.

6. Optimise images

Task: grunt-contrib-imagemin
You can set optimisation levels for the images - I just leave it at the default setting

imagemin: {
 ebay: {
  files: [{
   expand: true,
   cwd: 'src/ebay',
   src: '{,*/}*.{gif,jpeg,jpg,png}',
   dest: 'dist/ebay'
  }]
 }
}

7.Cache-busting

This was slightly tricky and subject to another blog post!

8. Deploy on to server with FTP

Task: grunt-ftp-deploy
Deployment via FTP. The login info is stored in a .ftppass file which should be excluded from git. I have  'ebay' build deploying to a versioned directory, which makes rollback easier when required! Also don't add this task to the 'default' grunt task just in case you deploy something accidentally! I also increment the version with npm version before i deploy.

'ftp-deploy': {
 main: {
  ...
 },
 ebay: {
  auth: {
   host: 'ftp.example.com',
   port: 21,
   authKey: 'key1' // username/password stored in .ftppass
  },
  src: 'dist/ebay',
  dest: '/path/to/server/<%= pkg.version %>', // deploy as the current package.json version
 }
}

Other things to add to my Gruntfile

• Add watch tasks for starting the server and autorefresh on file change
• Add unit-tests with grunt-mocha-phantom I've been putting this off, which is bad :-/

Tips & resources

1. Grunt website's sample gruntfile is a good place to start
2. Yo webapp generator generates a great Gruntfile, although it does a lot of things and probably needs better documentation to explain what each task does!
3. David Tucker's blog post on Grunt is very detailed helped me a lot in setting up my Gruntfile.
4. Use load-grunt-tasks to replace all those grunt.loadNpmTasks(...)statements in the Gruntfile.
   Just add require('load-grunt-tasks')(grunt)

Integrating Listing Generator with Google Drive App

My sisters have an ebay store selling dresses. To improve the workflow for creating ebay listings, I made a Listing Generator. It takes a bunch of inputs (description, editorial, measurements etc) via a form and outputs a HTML/CSS listing. The generator also outputs JSON data which can be copied into a textfile and saved for future edits.

Listing Generator form input

Listing generator output to HTML for ebay and JSON for saving to file

The generator is built with javascript/jquery with handlebars template. Because it is completely client-side, it can load local JSON files but has no way of saving changes back as local files, hence every time a listing is generated or an edit is made, the JSON needs to be manually copied and saved to a file... quite a tedious and error-prone workflow.

To improve the process, I decided to integrate the Listing Generator to Google Drive since all our listing JSON files are stored on Google Drive already. This would allow files to be created, edited and saved all without leaving the browser. This integration took me about two days, overall went fairly well although a few things took some setting up & trial and error.

1. Enable Drive API and Drive SDK

Google's helpful Quick Start Guide for enabling Drive API also contains some handy sample code. I also enabled Drive SDK to allow the app to integrate with the Drive UI. I used the below authorisation scopes for create, edit , save files and be installed to user's Drive:

https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/drive.install

After saving these Drive SDK settings in Google console, I could install the app to my Drive account and see my app integrated with the Drive UI.

2. Integration

I modified Google's sample code to integrate Listing Generator with Drive API:

• authenticate with gapi.auth.authorize
  

  function connect($do,retryIfFail){
   gapi.auth.authorize({
    'client_id': CLIENT_ID, 
    'scope': SCOPES,
    'immediate': retryIfFail?true:false // try immediate mode first 
   },function(authResult){
    if (authResult && !authResult.error) {
     $do.resolve(authResult)
     // Access token has been successfully retrieved, requests can be sent to the API.
    } else {
     if(retryIfFail){
      console.log('connection with immediate mode failed - retrying with non-immediate mode')
      connect($do,false);
     }else{
      $do.reject('authentication failed');
     }
    }
   });
   return $do.promise();
  }
  

  The code to upload/update a file is slightly modified from Google's example. If fileId is given, it saves to an existing file, otherwise a new file is created. It's not very pretty code with all the boundary, delimiter and all that string concatenation.. but hey, whatever works.
  An attempt to authorise with immediate mode is made first, and if it fails, it re-attempts with non-immediate mode. Immediate mode will not trigger a popup authorisation screen, but it only works for users that have already authorised the app before. (It took me awhile to figure out why my authorisation calls were failing when I was only using immediate mode) Non-immediate mode will force the user to reauthorise  via popup window. If the user has authorised once via this process, immediate mode should work for the user from that point. (You may need to have immediate mode triggered on a mouse-click to prevent popup blockers)
• parse URL parameter state.action to determine if opening existing file or creating a new one
• if open:
• get file id from URL parameters state.id
• load metadata with gapi.client.request
• download file metadata.downloadUrl
• when saving content to the same file, pass the loaded metadata and file id; use PUT method
• if create:
• get folder id from URL parameters state.folderId
• load folder metadata with gapi.client.request
• when saving content to file, pass new metadata with title (new file name), mimetype and folderId specified; use method POST

 function updateFile(fileId, fileMetadata, fileData) {
   const boundary = '-------314159265358979323846';
   const delimiter = "\r\n--" + boundary + "\r\n";
   const close_delim = "\r\n--" + boundary + "--";

   var $do = $.Deferred();

     var contentType = 'application/json';
     var base64Data = btoa(fileData);
     var multipartRequestBody =
         delimiter +
         'Content-Type: application/json\r\n\r\n' +
         JSON.stringify(fileMetadata) +
         delimiter +
         'Content-Type: ' + contentType + '\r\n' +
         'Content-Transfer-Encoding: base64\r\n' +
         '\r\n' +
         base64Data +
         close_delim;

     var request = gapi.client.request({
         'path': '/upload/drive/v2/files' + (fileId?'/'+fileId:''),
         'method': fileId?'PUT':'POST',
         'params': {'uploadType': 'multipart', 'alt': 'json'},
         'headers': {
           'Content-Type': 'multipart/mixed; boundary="' + boundary + '"'
         },
         'body': multipartRequestBody});
      request.execute(function(file){
       $do.resolve(file);
      });

  return $do.promise();
   
 }

Google Drive UI integration - creating a new file with the app

3. Publish app as a Google Chrome Web Store app

Even though my app is for internal use and would never be released publicly, I still had to publish it to the Chrome Web Store (albeit to a whitelist of users) for Drive UI integration to work for anyone else but me. This required registering an account (and paying $5), uploading a manifest.json file with the app information, an icon, screenshot & promo images. It's a fairly straightforward process, with a simpler setup and cheaper than Apple's process for getting an app up and running!

Overall, I'm pretty happy with the end result. Generating/editing files with a much smoother workflow, and as a bonus: our files are now automatically version controlled in Google Drive!