Blog

Where we discuss our work, thoughts, and process

Automatic file minification on nginx

For our HiFi content management system we wanted to automatically combine and compress CSS and Javascript files. On complex sites, this minification can have a significant impact on page load times. You can read more details about the benefits and our template-side implementation in a previous post. Here I am going to discuss the technical details of how we set things up on our nginx server.

Getting Started

There is a very useful and stable Minify project on Google Code. Both CSS and Javascript minification libraries are included and they can easily be switched out if you prefer another library. The normal way to use Minify is to install it in your root directory and call its main file with a query string to specify the specific files that should be included: /min/?f=js/jquery.js,js/jquery.cycle.js,js/setup.js. However with this setup, even after caching the minified files server-side, the PHP interpreter is involved in delivering the file, which is relatively inefficient. Ideally, after a file is cached the first time nginx would serve the compressed file directly.

There is some sample code included in the Minify download that uses Apache url rewriting to take a more "normal"-looking url and point it to the Minified version. This was almost what we were looking for, except for two things: it requires a very specific url structure—with all JS and CSS files stored inside the Minify folder—and we are not using Apache. For HiFi, we decided on Nginx, as the configuration for running many sites off of the same code-base is much simpler. Nginx has a HttpRewriteModule, which is similar to Apache's mod_rewrite, so I figured there must be a way to replicate the Minify sample code in the new environment.

Aside: Template Tags

A quick preview of how HiFi handles stylesheets and scripts to give you a bit of context: We added two new tags to our templating language: {% css %} and {% js %}. These are designed to be used every time a script or stylesheet is included in a template, and they do everything needed to hook into the minification system. Each tag takes a comma-separated list of file names that will be included. For example, a standard js tag might be {% js 'jquery,jquery.cycle,setup' %}.

By default, these tags will output straight or tags for each file in the list. However, adding min as the last parameter of either tag will cause it to instead output a single tag containing a special url that will get the Minified version. This way, it is just a matter of adding and removing that single parameter to switch between development and production modes. You could even set up a global variable to change it for all the tags in a layout at once.

The url format for serving the minified file is scripts/jquery,jquery.cycle,setup_123456.mjs. There are three things to note here: The first is that we use the .mjs extension (or .mcss). That change is what tells Nginx to serve the Minified version. The second thing is the timestamp we are appending to the filename. Since we will be setting far-future expires headers (so visitors' browsers know to cache the files locally), we need a way to manually expire the files when we have changed them. Our template tag automatically resets the timestamp whenever the source files change, but you could also increment it manually. Finally, this script assumes that JS files will be in a folder called /scripts and CSS files will be in /styles. Eventually, it would be nice to be able to place files in any arbitrary location, but that makes things considerably more complicated.

How It’s Going to Work

For maximum speed, we are going to enable two different types of cacheing: of the minified files on the server and in the visitor's web browser. Whenever a particular combination of files is requested, nginx will check to see if there is already a cached copy. If there is—and there will be except for the first time after a source file is modified—the server will return that file directly, without involving the PHP interpreter at all. This operation is very fast, as it required almost no processing.

If nginx cannot find a cached copy of the file, it will pass the request off to our minification script. This code uses the Minify library to combine and compress the source files, returns the end result to the visitor's browser, and caches it for future use. It will actually cache two versions: one as plain-text and one that has been further compressed with gzip. Nginx uses content negotiation to determine which of the two should be returned to the browser.

What happens when you change a source file? Since we will be telling the visitor's browser to cache our asset files indefinitely, you will need to change the timestamp/version number on the file when you make updates. By doing that, you will invalidate both the server-side and the client-side caches and cause the files to be re-minified.

The Code

Nginx Configuration

Open up the nginx.conf file on your server. In the http block, add the following code:

# Logic for serving minified CSS & JS
location ~* \.(mjs|mcss)$ {
  set $domain      www.domain.com;        # Change this to your site's domain name
  set $root_fcgi   /var/sites/site_name;  # Change this to the public root folder of your site
  set $root_cache  $root_fcgi/cache;      # Change this to a folder in which to cache the minified files
  set $min_dir     /var/sites/min;        # Change this folder to wherever you put the Minify files
  
  include fastcgi_params;
  fastcgi_param SITE_ROOT $root_fcgi;
  fastcgi_param SCRIPT_FILENAME $min_dir/nginx-mininification.php;
  fastcgi_param PATH_INFO nginx-minification.php;
  fastcgi_param SERVER_NAME $domain;
  fastcgi_param CACHE_DIR $root_cache;
  
  root $root_cache;
  
  expires max;
	
  gzip_static on;   # You will need to have installed Nginx using the --with-http_gzip_static_module flag for this to work
  gzip_http_version 1.1;
  gzip_proxied expired no-cache no-store private auth;
  gzip_disable "MSIE [1-6]\.";
  gzip_vary on;
  
  # If there is not already a cached copy, create one
  if (!-f $request_filename) {
    root $root_fcgi;
    fastcgi_pass 127.0.0.1:9000;
  }
}

Make sure to set the four variables at the beginning to reflect the way your server is actually configured.

Minify and Cache

First, copy the Minify files to anywhere you like on your server. They do not need to be in a publicly-accessible location, since Nginx will be proxying requests to Minify behind the scenes. The Minify config.php file contains sensible defaults, but check it to be sure.

Create a new file inside the Minify directory, called nginx-minification.php. Paste the following code into it:

 $sources,
  'quiet' => true,
  'encodeMethod' => '',
  'lastModifiedTime' => 0
));
if (! $output['success']) {
  send404();
}

// Clear old cached files
if ($handle = @opendir($cachedir)) {
  while (false !== ($file = @readdir($handle))) {
    if ($file != '.' && $file != '..' && stristr($file, $filename)) {
      @unlink($cachedir . '/' . $file);
    }
  }
  @closedir($handle);
}

// Cache the output
error_reporting(0);
if (false === file_put_contents($cache, $output['content']) ||
  false === file_put_contents($cache.'.gz', gzencode($output['content'],9))) {
  echo "/* File writing failed. Your cache directory, {$cachedir}, must be writable by PHP. */\n";
  exit();
}

// And return it to the client
unset($output['headers']['Last-Modified'], $output['headers']['ETag']);
foreach ($output['headers'] as $name => $value) {
  header("{$name}: {$value}");
}
echo $output['content'];

Testing It Out

You should be ready to go. Try converting your script and style tags to use the new url format ("/scripts/jquery,setup_123.mjs" or "/styles/layout,typography,homepage_123.mcss"), making sure that the source files are in either the scripts or the styles folders. If it doesn't work, make sure all the paths in the Nginx configuration file are set correctly and that the cache folder exists. If you have suggestions or problems, please post them in the comments! I'm no expert on the ins and outs of nginx, so I am sure there are better ways to do certain things here.

Comments

Christine Fürst's avatar
Christine Fürst
Thanks! I am curious about NGINX. Is this website served via nginx? Cheers, @stinie
Mario Estrada's avatar
Mario Estrada
Does the PHP code works? It does not start with '<?php' and the first line is '$sources,' and I think it should be '$sources = array('. Or maybe I'm not fully understanding how the PHP script gets called. Is it being appended to another script? I'm interested in implementing something similar to the minification detailed here.
Jeff's avatar
Jeff
I think the example code was a snippet from googles gen.php. http://code.google.com/p/minify/source/browse/trunk/mincache/gen.php
Dumitru's avatar
Dumitru
Well, I modified a lot the script to work. This is my version of the minificator.php:


<?php<br />
//Send a 404 error in case of failure
function send404() {
header('HTTP/1.0 404 Not Found');
exit('File not found');
}

//Get vars enabled by Nginx
$cacheDir = $_ENV['CACHE_DIR'];
$siteRoot = $_ENV['SITE_ROOT'];
$domainName = $_ENV['SERVER_NAME'];
$minifyRequest = $_ENV['SCRIPT_NAME'];

//Define minify library directory and cache directory
define('MINIFY_MIN_DIR', realpath(dirname(__FILE__)));
define('MINCACHE_DIR' , $cacheDir);

//Validate the URLs to minify - security can be improved
if (!isset($minifyRequest) || !preg_match('@^.*m(css|js)$@', $minifyRequest, $m)) {
send404();
}

//Strip the modified time and the fake extension from the request URI (group of files to minify)
$group = preg_replace('/_([0-9] )\.m(css|js)$/', '', $m[0]);

//Create an md5 checksum from the request URI
$m[0] = md5($m[0]);

//Prepare minify options
$spec = array(
'group' => $group,
'ext' => $m[1],
'plainFilename' => "{$m[0]}.{$m[1]}"
,'deflatedFilename' => "{$m[0]}.{$m[1]}.zd"
,'typeMap' => MINCACHE_DIR . '/' . $m[0]
,'plainFile' => MINCACHE_DIR . "/{$m[0]}.{$m[1]}"
,'deflatedFile' => MINCACHE_DIR . "/{$m[0]}.{$m[1]}.gz"
,'ctype' => ($m[1] == 'js' ? 'application/x-javascript' : 'text/css')
);

//Detect if the client browser supports gzip encoding and search for a cached version to return and exit
if (substr_count($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip') && file_exists($spec['deflatedFile']))
{
header('Content-Type:' . $spec['ctype']);
header('Content-Encoding:gzip');
readfile($spec['deflatedFile']);
exit();
}

//Search for a cached version (non-gziped) of the file to return and exit
if (file_exists($spec['plainFile'])) {
header('Content-Type:' . $spec['ctype']);
readfile($spec['plainFile']);
exit();
}

//Load the Minify config file
require MINIFY_MIN_DIR . '/config.php';

//Setup new include path for the minify library
set_include_path($min_libPath . PATH_SEPARATOR . get_include_path());

//Set a new autoload function for the Minify files
function min_autoload($name) {
require str_replace('_', DIRECTORY_SEPARATOR, $name) . '.php';
}
spl_autoload_register('min_autoload');

//Search if there is an documentRoot set in to Minify configs
if ($min_documentRoot) {
$_SERVER['DOCUMENT_ROOT'] = $min_documentRoot;
} elseif (0 === stripos(PHP_OS, 'win')) {
Minify::setDocRoot(); // IIS may need help
}

//$min_serveOptions['minifierOptions']['text/css']['symlinks'] = $min_symlinks;

//Parse the list of files to minify
$sources = array();
$files = explode(',', $spec['group']);
foreach ($files as $file) {
$file = realpath($siteRoot . $file);
if (($file !== false) && (file_exists($file)))
{
// file OK
$sources[] = $file;
}
else
{
send404();
}
}

//Minify the group of files
$output = Minify::serve('Files', array(
'files' => $sources,
'quiet' => true,
'encodeMethod' => '',
'lastModifiedTime' => 0,
));

//If minication failed - generate an error and stop
if (!$output['success']) {
send404();
}

// Cache the output in plaintext and gziped
error_reporting(0);
if (false === file_put_contents($spec['plainFile'], $output['content']) ||
false === file_put_contents($spec['deflatedFile'], gzencode($output['content'], 9)))
{
echo " File writing failed. Your cache directory, {$cachedir}, must be writable by PHP. \n";
exit();
}

//Unset unused headers from the output
unset($output['headers']['Last-Modified'], $output['headers']['ETag']);

//Set nedded headers for output
foreach ($output['headers'] as $name => $value) {
header("{$name}: {$value}");
}

//Flush the content
echo $output['content'];
Burçlar's avatar
Burçlar
i am using nginx and it is adding queries end of the javascript files . How i can disable this ?
aaabdo's avatar
aaabdo
not work at all :( i see this in github easy and works fine https://github.com/nginx-modules/nginx-minify

Leave a comment