Javascript aggregation in Drupal is just what it sounds like: it aggregates Javascript files that are added during a page request into a single file. Modules and themes add Javascript using Drupal’s API, and the Javascript aggregation system takes care of aggregating all of that Javascript into one or more files. Drupal does this in order to cut down on the number of HTTP requests needed to load a page. Fewer HTTP requests is generally better for front-end performance.
In this article we’ll take a look at the API for adding Javascript, paying specific attention to the options affecting aggregation in order to make best use of the system. We’ll also take a look at some common pitfalls to look out for and how you can avoid them using Advanced Aggregation (AdvAgg) module. This article will mostly be looking at Drupal 7, however much of what we’ll cover applies equally to Drupal 8, and we’ll take a look specifically at what’s changed in Drupal 8.
If fewer is better, why multiple aggregates?
Before we dig in, I said that fewer HTTP requests is generally better for front-end performance. You may have noticed that Drupal creates more that one Javascript aggregate for each page request. Why not a single aggregate? The reason Drupal does this is to take advantage of browser caching. If you had a single aggregate with all the Javascript on the page, any difference in the aggregate from page to page would cause the browser to download a different aggregate with a lot of the same code. This situation arises when Javascript is conditionally added to the page, as is the case with many modules and themes. Ideally, we would group Javascript that appears on every page apart from conditionally added Javascript. That way the browser would download the every page Javascript aggregate once and subsequently load it from cache on other page requests on your website. Drupal’s Javascript aggregation attempts to give you the ability to do exactly that, striking a balance between making fewer HTTP requests, and leveraging browser caching.
The Basics
Let’s take a step back and briefly go over how to use Javascript aggregation in Drupal 7. You can turn on Javascript aggregation by going to Site Configuration > Development > Performance in your Drupal admin. Check off "Aggregate Javascript files" and Save. That’s it. Drupal will start aggregating module and theme Javascript from there on out.
Using the API
Once you’ve enabled Javascript aggregation, great! You’re on your way to sweet, sweet performance gains. But you’ve got modules and themes to build, and Javascript to go along with them. How do you make use of Javascript aggregation provided by Drupal? What kind of control do you have over the process?
There are a couple API entry points for adding Javascript to the page. Most API entry points have parameters that let you tell Drupal how to aggregate your Javascript. Let’s look at each in turn.
drupal_add_js
This is a loaded function that is used to add a Javascript file, externally hosted Javascript, a setting, or inline Javascript to the page. When it comes to Javascript aggregation, only Javascript files are aggregated by Drupal, so we’re going to focus on adding files. The other types (inline and external) do have important impacts on Javascript aggregation though, which we’ll get into later.
hook_library/drupal_add_library
Drupal also provides a mechanism to register Javascript/CSS "libraries" associated with a module. This is nice because you can specify the options to pass to drupal_add_js in one place. That gives you the flexibility to call drupal_add_library in multiple places, without having to repeat the same options. The other nice thing is that you can specify dependencies for your library and Drupal will take care of resolving those dependencies and putting them in the right order for you.
I like to register every script in my module as a library, and only add scripts using drupal_add_library (or the corresponding #attached method, see below). This way I’m clearly expressing any dependencies and in the case where my script is added in more than one place, the options I give the aggregation system are all in one place in case I need to make a change. It’s also a nice way to safely deprecate scripts. If all the dependencies are spelled out, you can be assured that removing a script won’t break things.
Instructions for using hook_library and drupal_add_library are well documented. However, one important thing to note is the third parameter to drupal_add_library, every_page, which is used to help optimize aggregation. At registration time in hook_library, you typically don’t know if a library will be used on every page request or not. Especially if you’re registering libraries for other module authors to use. This is why drupal_add_library has an every_page parameter. If you’re adding a library unconditionally on every page, be sure to set every_page parameter to TRUE so that Drupal will group your script into a stable aggregate with other scripts that are added to every page. We’ll discuss every_page in more detail below.
#attached
#attached is a render array property that lets you add Javascript, CSS, libraries and arbitrary data to the page. #attached is the preferred approach for adding Javascript in Drupal. In fact, drupal_add_js and drupal_add_library are gone in Drupal 8 in favor of #attached. #attached is nice since it is part of a render array and can be easily altered by other modules and themes. It’s also essential for places where render arrays are cached. With caching in play, functions that build out a render array are bypassed in favor of a cached copy. In that scenario, you need to have your Javascript addition included in #attached on the render array, because a call to drupal_add_js/drupal_add_library in your builder function would otherwise be missed.
Drupal developers often fall back on drupal_add_js for lack of a render array in context. It’s common to add Javascript in places like hook_init, preprocess and process functions, where there is no render array available to add your Javascript via #attached. In these scenarios, consider whether your Javascript is more appropriately added via a hook_page_build, or hook_node_view, where there is a render array available to add your script via #attached. Not only will you begin to align yourself with the Drupal 8 way of doing things, but you open yourself up to being able to use the render_cache module, which can gain you some performance. See the documentation for drupal_process_attached for more information.
Controlling aggregation
Let’s take a close look at the parameters affecting Javascript aggregation. Drupal will create an aggregate for each unique combination of the scope, group, and every_page parameters, so it’s important to understand what each of these mean.
scope
The possible values for scope are ‘header’ and ‘footer’. A scope of ‘header’ will output the script in the
tag, a scope of ‘footer’ will output the script just before the closing tag.
group
The ‘group’ option takes any integer as a value, although you should stick to the constants JS_LIBRARY, JS_DEFAULT and JS_THEME to avoid excessive numbers of aggregates and follow convention.
every_page
The ‘every_page’ option expects a boolean. This is a way to tell the aggregation system that your script is guaranteed to be included on every page. This flag is commonly overlooked, but is very important. Any scripts that are added on every page should be included in a stable aggregate group, one that is the same from page to page, such that the browser can make good use of caching. The every_page parameter is used for exactly that purpose. Miss out on using every_page and your script is apt to be lumped into a volatile aggregate that changes from page to page, forcing the browser to download your code anew on each page request.
What can go wrong
There are a few problems to watch out for when it comes to Javascript aggregation. Let’s look at some in turn.
Potential number of aggregates
The astute reader will have noticed that there is a potential for a lot of aggregates to be created. When you combine scope, group and every_page options, the number of aggregates per page can get out of control if you’re not careful. With two values for scope, three for groups and two for every_page, there is potential to climb to twelve aggregates in a page. Twelve seems a little out of control. Sure, we want to leverage browser caching, but this many HTTP requests is concerning. If you’re getting high into the number of aggregates it’s likely that modules and themes adding Javascript are not using the API as best as they could.
Misuse of Groups
As was mentioned, JS_LIBRARY, JS_DEFAULT and JS_THEME are default constants that ship with Drupal to group Javascript. When adding Javascript, modules and themes should stick to these groups. As soon as you deviate, you introduce a new aggregate. Often when people deviate from the default groups, it’s to ensure ordering. Maybe you need your script to be the very first script on the page, so you set your group to JS_LIBRARY - 100. Instead, use the ‘weight’ option to manage ordering. Set your group to JS_LIBRARY and set the weight very low to ensure your first in the JS_LIBRARY aggregate. Another issue I see is themes adding script explicitly under the JS_THEME group. This is logical, after all, it is Javascript added by the theme! However, it’s better to make use of hook_library, declare your dependencies and let the group default to JS_DEFAULT. The order you need is preserved by declaring your dependencies, and you avoid creating a new aggregate unessesarily.
Inline and External Scripts
Extra care should be employed when adding external and inline scripts. drupal_add_js and friends preserve order really well, such that they track the order the function is called for all the script added to the page and respect that. That’s all well and good. However, if an inline or external script is added between file scripts, it can split the aggregate. For example if you add 3 scripts: 2 file scripts and 1 external, all with the same scope, group, and every_page values, you might think that Drupal would aggregate the 2 file scripts and output 2 script tags total. However, if drupal_add_js gets called for the file, then external, then file, you end up getting two aggregates generated with the external script in between for a total of 3 script tags. This is probably not what you want. In this case it’s best to specify a high or low weight value for the external script so it sits at the top or bottom of the aggregate and doesn’t end up splitting it. The same goes for inline JS.
Scripts added in a different order
I alluded to this in the last one, but certain situations can arise where scripts get added to an aggregate in a different order from one request to the next. This can be for any number of reasons, but because Drupal is tracking the call order to drupal_add_js, you end up with a different order for the same scripts in a group and thus a different aggregate. Sadly the same code will sit in two aggregates on the server, with slightly different source order, but otherwise they could have been exactly equal, produced a single aggregate and thus cached by the browser from page to page. The solution in this case is to use weight values to ensure the same order within an aggregate from page to page. It’s not ideal, because you don’t want to have to set weights on every hook_library / drupal_add_js. I’d recommend handling it on a case by case basis.
Best Practices
Clearly, there is a lot that can go wrong, or at least end up taking you to a place that is sub-optimal. Considering all that we’ve covered, I’ve come up with a list of best practices to follow:
Always use the API, never ‘shoehorn’ scripts into the page
A common problem is running into modules and themes that sidestep the API to add their Javascript to the page. Common methods include using drupal_add_html_head to add script to the header, using hook_page_alter and appending script markup to $variables[‘page_bottom’], or otherwise outputting a raw
Use hook_library
hook_library is a great tool. Use it for any and all Javascript you output (inline, external, file). It centralizes all of the options for each script so you don’t have to repeat them, and best of all, you can declare dependencies for your Javascript.
Use every_page when appropriate
The every_page option signals to the aggregation system that your script will be added to all pages on the site. If your script qualifies, make sure your setting every_page = TRUE. This puts your script into a "stable" aggregate that is cached and reused often.
AdvAgg
Advanced CSS/JS Aggregation (AdvAgg) is a contributed module that replaces Drupal’s built in aggregation system with it’s own, making many improvements and adding additional features along the way. The way that you add scripts is the same, but behind the scenes, how the groups are generated and aggregated into their respective aggregates is different. AdvAgg also attempts to overcome many of the scenarios where core aggregation can go wrong that I listed above. I won’t cover all of what AdvAgg has to offer, it has an impressive scope that would warrant it’s own article or more. Instead I’ll touch on how it can solve some of the problems we listed above, as well as some other neat features.
Out of the box the core AdvAgg module supplies a handful of transparent backend improvements. One of those is stampede protection. After a code release with JS/CSS changes, there is a potential that multiple requests for the same page will all trigger the calculation and writing of the same new aggregates, duplicating work. On high traffic sites, this can lead to deadlocks which can be detrimental to performance. AdvAgg implements locking so that only the first process will perform the work of calculating and writing aggregates. Advagg also employs smarter caching strategies so the work of calculating and writing aggregates is done as infrequently as possible, and only when there is a change to the source files. These are nice improvements, but there is great power to behold in AdvAgg’s submodules.
AdvAgg Compress Javascript
AdvAgg comes with two submodules to enhance JS and CSS compression. They each provide a pluggable way to have a compressor library act on the each aggregate before it’s saved to disk. There are a few options for JS compression, JSqueeze is a great option that will work out of the box. However, if you have the flexibility on your server to install the JSMIN C extension, it’s slightly more performant.
AdvAgg Modifier
AdvAgg Modifier is another sub module that ships with AdvAgg, and here is where we get into solving some of the problems we listed earlier. Let’s explore some of the more interesting options that are made available by AdvAgg Modifier:
Optimize Javascript Ordering
There are a couple of options under this fieldset that seek to solve some of the problems laid out above. First up, "Move Javascript added by drupal_add_html_head() into drupal_add_js()" does just what it says, correcting the errant use of the API by module/theme authors. Doing this allows the Javascript in question to participate in Javascript aggregation and potentially removes an HTTP request if it was a file that was being added. Next “Move all external scripts to the top of the execution order” and “Move all inline scripts to the bottom of the execution order” are both an attempt at resolving the issue of splitting the aggregate. This is more or less assuming that external files are more library-esque and inline Javascript is meant to be run after all other Javascript has run. That may or may not be the case depending on what you’re doing and you should definitely use caution. Such as that is, if it works for you, it could be a simple way to avoid splitting the aggregate without having to do manual work of adjusting weights.
Move JS to the footer
As we discussed, you typically add Javascript to one of two scopes, header or footer. A common performance optimization is to move all the Javascript to the footer. AdvAgg gives you a couple choices in how you do that ranging from some to all Javascript moved to the footer. This can easily break your site however, and you definitely need to test and use caution here. The reason of course is that there could be other script that aren’t declaring their dependencies properly, and/or are ‘shoehorning’ their Javascript into the page that are depending on a dependency existing in the header. These will need to be fixed on a case by case basis (submit patches to contrib modules!). If you still have Javascript that really does need to be in the header, AdvAgg extends the options for drupal_add_js and friends to include a boolean scope_lock that when set to TRUE, will prevent AdvAgg from moving the script to the footer. There always seems to be some stubborn ads or analytics provider that insists on presence in the header, and is backed by the business that forces you to make at least one of these exceptions. Another related option is "Put a wrapper around inline JS if it was added in the content section incorrectly". This is an attempt to resolve the problem of inline Javascript in the body depending on things being in the header. AdvAgg will wrap any inline Javascript in a timer that waits on jQuery and Drupal to be defined before running the inline code. It uses a regular expression to search for the scripts, which while neat, feels a little brittle. There is an alternate option to search for the scripts using xpath instead of regex, which could be more reliable. If you try it, do so with care, test first.
Drupal 8
Not a lot is different when it comes to Javascript aggregation in D8, but there have been a few notable changes. First, the aggregation system has been refactored under the hood to be pluggable. You can now cleanly swap out and/or add to the aggregation system, whereas in Drupal 7 it was a difficult and messy affair to do so. Modules like AdvAgg should have a relatively easier time implementing their added functionality. Other improvements to the aggregation system that didn’t quite make the D8 release will now have a clearer path to implementation for the same reason. In terms of its function and the way in which Javascript is grouped for aggregation, everything is as it was in Drupal 7.
The second change centres around adding Javascript to the page. drupal_add_js and drupal_add_library have been removed in Drupal 8 in favor of #attached. The primary motivation is to improve cacheability. With Javascript consistently added using #attached, it’s possible to do better render array caching. The classic example being that if you were using drupal_add_js in your form builder function, your Javascript would be missed if the cached render array was used and the builder function skipped. Caching aside, it’s nice that there is a single consistent way to add Javascript. Also related to this change, is hook_library definitions have gone the way of the dodo. In their place, *.libraries.yml files are now used to define your Javascript and CSS assets in libraries, continuing the Drupal 8 theme of replacing info hooks with YML files.
A third somewhat related change that I’ll point out is that Drupal core no longer automatically adds jQuery, Drupal.js and Drupal.settings (drupalSettings in Drupal 8). jQuery still ships with core, but if you need it, you have to declare it as a dependency in your *.libraries.yml file. There have also been steps taken to reduce Drupal core’s reliance on jQuery. This isn’t directly related to Javascript aggregation per se, but it’s a big change nonetheless that you’ll have to keep in mind if you’re used to writing Javascript in Drupal 7 and below. It’s a welcome improvement with potential for some nice front end performance gains. For more information and specific implementation details, check out the guide for adding Javascript and CSS from a theme and adding Javascript and CSS from a module.
HTTP 2.0
Any article about aggregating Javascript assets these days deserves a disclaimer regarding HTTP 2.0. For the uninitiated, HTTP 2.0 is the new version of the HTTP protocol that while maintaining all the semantics of HTTP 1.1, significantly changes the implementation details. The most noted change in a nutshell is that instead of opening a bunch of TCP connections to download all the resources you need from a single origin, HTTP 2.0 opens a single connection per origin and multiplexes resources on that connection to achieve parallelism. There is a lot of promise here because HTTP 2.0 enables you to minimize the number of TCP connections your browser has to open, while still downloading assets in parallel and individually caching assets. That means that you could be better off not aggregating your assets at all, allowing the browser to maximize caching efficiency without incurring the cost of multiple open TCP connections. In practice, the jury is still out on whether no aggregation at all is a good idea. It turns out that compression doesn’t do as well on a collection of individual small files as it does on aggregated large files. It’s likely that some level of aggregation will continue to be used in practice. However as hosts more widely support HTTP 2.0 and old browsers fall off, this will become a larger area of interest.
Conclusion
Whew, we’ve come to the end at last. My purpose was to go over all of the nuances of Javascript aggregation in Drupal. It’s an important topic when it comes to front end performance, and one that deserves careful consideration when you're trying to eek every last bit of performance out of your site. With Drupal 8 and HTTP 2.0 now a reality, it’ll be interesting to watch as things evolve for the better.