In my previous post, Modern Decoupling is More Performant, we discussed how saving HTTP round-trips has a very positive impact on performance. In particular, we demonstrated how the JSON API module could help your application by returning multiple entities in a single request. Doing so eliminates the need for making an individual request per entity. However, this is only possible when fetching entities, not when writing data and only if those entities are related to the entry point (a particular entity or collection).
Sometimes you can solve this problem by writing a custom resource in the back-end every time, but that can lead to many custom resources, which impacts maintainability and is tiresome. If your API is public and you don’t have prior knowledge of what the consumers are going to do with it, it’s not even possible to write these custom endpoints.
The Subrequests module completes that idea by allowing ANY set of requests to be aggregated together. It can aggregate them even when one of them depends on a previous response. The module works with any request, it's not limited to REST or any other constraint. For simplicity, all the examples here will make requests to JSON API.
Why Do We Need It?
The main concept of the Subrequests module is that instead of sending multiple requests to your Drupal instance we will only send a single request. In this master request, we will provide the information about the requests we need to make in a JSON document. We call this document blueprint.
A blueprint is a JSON document containing the instructions for Drupal to make all those requests in our name. The blueprint document contains a list of subrequest objects. Each subrequest object contains the information about a single request being aggregated in the blueprint.
Imagine that our consumer application has a decoupled editorial interface. This editorial interface contains a form to create an article. As part of the editorial experience, we want the form to create the article and a set of tags in the Drupal back-end.
Without using Subrequests, the consumer application should execute the following requests when the form is submitted:
- Query Drupal to find the UUID for the tags vocabulary.
- Query Drupal to find the UUID of the user, based on the username present in the editorial app.
- Create the first tag in the form using the vocabulary UUID.
- Create the second tag in the form using the vocabulary UUID.
- Create the article in the form using the user UUID and the newly created tags.
We can query for the user and the vocabulary in parallel. Once that is done, and using the information in the vocabulary response, we can create the tag entities. Once those are created, we can finally create the article. In total, we would be making five requests at three sequential levels. And, this is not even a complex example!
Sequential requests introduce a performance penalty.
A JavaScript pseudo-code for the form submission handler could look like:
console.log('Article creation started…');
Promise.all([
httpRequest('GET', 'https://cms.contentacms.io/api/vocabularies?filter[vid-filter][condition][path]=vid&filter[vid-filter][condition][value]=tags'),
httpRequest('GET', 'https://cms.contentacms.io/api/users?filter[admin][condition][path]=name&filter[admin][condition][value]=admin'),
])
.then(res => {
const [vocab, user] = res;
return Promise.all([
Promise.resolve(user),
httpRequest('POST', 'https://cms.contentacms.io/api/tags', bodyForTag1, headers),
httpRequest('POST', 'https://cms.contentacms.io/api/tags', bodyForTag2, headers),
])
})
.then(res => {
const [user, tag1, tag2] = res;
const body = buildBodyForArticle(formData, user, tag1, tag2);
return httpRequest('POST', 'https://cms.contentacms.io/api/articles', body, headers);
})
.then(() => {
console.log('Article creation finished!');
});
Using Subrequests
Our goal is to have JavaScript pseudo-code that looks like:
console.log('Article creation started…');
const blueprint = buildBlueprint(formData);
httpRequest('POST', 'https://cms.contentacms.io/api/subrequests?_format=json', blueprint, headers)
.then(() => {
console.log('Article creation finished!');
});
We've reduced our application code to a single POST request that contains a blueprint in the request body. We have reduced the problem to the blueprint creation. That is a big improvement in the developer experience of consumer applications.
Sequential requests as processed by Subrequests avoid unnecessary round trips.
Parallel Requests
In our current task we need to perform two initial HTTP requests that can be run in parallel:
- Query Drupal to find the UUID for the tags vocabulary.
- Query Drupal to find the UUID of the user based on the username in the editorial app.
That translates to the following blueprint:
[
{
"requestId": "vocabulary",
"action": "view",
"uri": "/api/vocabularies?filter[vid-filter][condition][path]=vid&filter[vid-filter][condition][value]=tags",
"headers": ["Accept": "application/vnd.application+json"]
},
{
"requestId": "user",
"action": "view",
"uri": "/api/users?filter[admin][condition][path]=name&filter[admin][condition][value]=admin",
"headers": ["Accept": "application/vnd.application+json"]
}
]For each subrequest, we can observe that we are providing four keys:
requestIdA string used to identify the subrequest. This is an arbitrary value generated by the consumer application.actionIdentifies the action being performed. A "view" action will generate a GET request. A "create" action will generate a POST request, etc.uriThe URL where the subrequest will be sent .headersAn object containing the headers specific for this subrequest.
The response to this blueprint (after adjusting the permissions in Drupal to view users and vocabularies) will return the response to both subrequests:
{
"vocabulary": {
"headers": {
"content-id": ["<vocabulary>"],
"status": [200]
},
"body": "{\"data\":[{\"type\":\"vocabularies\",\"id\":\"47ce8895-0df6-44a4-af43-9ef3b2a924dd\",\"attributes\":{\"status\":true,\"dependencies\":{\"module\":[\"recipes_magazin\"]},\"_core\":\"HJlsFfKP4PFHK1ub6QCSNFmzAnGiBG7tnx53eLK1lnE\",\"name\":\"Tags\",\"vid\":\"tags\",\"description\":\"Use tags to group articles on similar topics into categories.\",\"hierarchy\":0,\"weight\":0},\"links\":{\"self\":\"http:\\/\\/localhost\\/api\\/vocabularies\\/47ce8895-0df6-44a4-af43-9ef3b2a924dd\"}}],\"links\":{\"self\":\"http:\\/\\/localhost\\/api\\/vocabularies?filter%5Bvid-filter%5D%5Bcondition%5D%5Bpath%5D=vid\\u0026filter%5Bvid-filter%5D%5Bcondition%5D%5Bvalue%5D=tags\"}}"
},
"user": {
"headers": {
"content-id": ["<user>"],
"status": [200]
},
"body": "{\"data\":[{\"type\":\"users\",\"id\":\"a0b7af80-e319-4271-899f-f151d3fbfc8e\",\"attributes\":{\"internalId\":1,\"name\":\"admin\",\"mail\":\"admin@example.com\",\"timezone\":\"Europe\\/Madrid\",\"isActive\":true,\"createdAt\":\"2017-09-15T15:47:26+0200\",\"updatedAt\":\"2017-09-15T20:06:15+0200\",\"access\":1505565434,\"lastLogin\":\"2017-09-15T20:06:07+0200\"},\"relationships\":{\"roles\":{\"data\":[]}},\"links\":{\"self\":\"http:\\/\\/localhost\\/api\\/users\\/a0b7af80-e319-4271-899f-f151d3fbfc8e\"}}],\"links\":{\"self\":\"http:\\/\\/localhost\\/api\\/users?filter%5Badmin%5D%5Bcondition%5D%5Bpath%5D=name\\u0026filter%5Badmin%5D%5Bcondition%5D%5Bvalue%5D=admin\"}}"
}
}In the (simplified) response above we can see that for each subrequest, we have one key in the response object. That key is the same as our requestId in the blueprint. Each one of the subresponses contains the information about the response headers and the response body. Note how the response body is an escaped JSON object.
This blueprint is not sufficient to create an article with two tags, but it's a great start. Let's build on top of that to create the tags and the article.
Dependent Requests
The next task we need to execute is the creation of the two tag entities:
- Create the first tag in the form using the vocabulary UUID.
- Create the second tag in the form using the vocabulary UUID.
To do this, we will need to expand the blueprint. However, we don't know the vocabulary UUID at the time we are writing the blueprint. What we do know is that the vocabulary UUID will be in the subresponse to the vocabulary subrequest. In particular, we can find the UUID in data[0].id.
We will use that information to create a blueprint that can create tags. Since we don't know the actual value of the vocabulary UUID, we will use a replacement token. At some point, during the blueprint processing by Drupal, the token will be resolved to the actual UUID value.
Replacement Tokens
We can use replacement tokens anywhere in the body or the URI of our subrequests. For those to be resolved, a token needs to be formatted in the following way:
{{<requestId>.<"body"|"headers">@<json-path-expression>}}In particular, the replacement token for our vocabulary UUID will be:
{{vocabulary.body@$.data[0].id}}What this replacement says is:
- Use the subresponse for the
vocabularysubrequest. - Take the body from that subresponse.
- Extract the string under
data[0].id, by executing the JSON Path expression$.data[0].id. You can execute any JSON Path expression as long as it returns a string. JSON Path is a very powerful way to extract data from an arbitrary JSON object, in our case the body in subresponse to thevocabularysubrequest.
This is what our blueprint looks like after adding the subrequests to create the tag entities. Note the presence of the replacement tokens:
[
{
"requestId": "vocabulary",
"action": "view",
"uri": "/api/vocabularies?filter[vid-filter][condition][path]=vid&filter[vid-filter][condition][value]=tags",
"headers": {"Accept": "application/vnd.api+json"}
},
{
"requestId": "user",
"action": "view",
"uri": "/api/users?filter[admin][condition][path]=name&filter[admin][condition][value]=admin",
"headers": {"Accept": "application/vnd.api+json"}
},
{
"action": "create",
"requestId": "tags-1",
"body": "{\"data\":{\"type\":\"tags\",\"attributes\":{\"name\":\"My First Tag\"},\"relationships\":{\"vocabulary\":{\"data\":{\"type\":\"vocabularies\",\"id\":\"{{vocabulary.body@$.data[0].id}}\"}}}}}",
"uri": "/api/tags",
"headers": {"Content-Type": "application/vnd.api+json"},
"waitFor": ["vocabulary"]
},
{
"action": "create",
"requestId": "tags-2",
"body": "{\"data\":{\"type\":\"tags\",\"attributes\":{\"name\":\"My Second Tag\",\"description\":null},\"relationships\":{\"vocabulary\":{\"data\":{\"type\":\"vocabularies\",\"id\":\"{{vocabulary.body@$.data[0].id}}\"}}}}}",
"uri": "/api/tags",
"headers": {"Content-Type": "application/vnd.api+json"},
"waitFor": ["vocabulary"]
}
]Note that to use a replacement token in a subrequest, we need to add a dependency on the subresponse that contains the information. That's why we added the waitFor key in our tag subrequests.
Finishing the Blueprint
Using the same principles that we used for the tags we can add the subrequest for:
- Create the article in the form using the user UUID and the newly created tags.
That will leave our completed blueprint as:
[
{
"requestId": "vocabulary",
"action": "view",
"uri": "/api/vocabularies?filter[vid-filter][condition][path]=vid&filter[vid-filter][condition][value]=tags",
"headers": {"Accept": "application/vnd.api+json"}
},
{
"requestId": "user",
"action": "view",
"uri": "/api/users?filter[admin][condition][path]=name&filter[admin][condition][value]=admin",
"headers": {"Accept": "application/vnd.api+json"}
},
{
"action": "create",
"requestId": "tags-1",
"body": "{\"data\":{\"type\":\"tags\",\"attributes\":{\"name\":\"My First Tag\"},\"relationships\":{\"vocabulary\":{\"data\":{\"type\":\"vocabularies\",\"id\":\"{{vocabulary.body@$.data[0].id}}\"}}}}}",
"uri": "/api/tags",
"headers": {"Content-Type": "application/vnd.api+json"},
"waitFor": ["vocabulary"]
},
{
"action": "create",
"requestId": "tags-2",
"body": "{\"data\":{\"type\":\"tags\",\"attributes\":{\"name\":\"My Second Tag\",\"description\":null},\"relationships\":{\"vocabulary\":{\"data\":{\"type\":\"vocabularies\",\"id\":\"{{vocabulary.body@$.data[0].id}}\"}}}}}",
"uri": "/api/tags",
"headers": {"Content-Type": "application/vnd.api+json"},
"waitFor": ["vocabulary"]
},
{
"action": "create",
"requestId": "article",
"headers": {"Content-Type": "application/vnd.api+json"},
"body": "{\"data\":{\"type\":\"articles\",\"attributes\":{\"body\":\"Custom value\",\"default_langcode\":\"1\",\"langcode\":\"en\",\"promote\":\"1\",\"status\":\"1\",\"sticky\":\"0\",\"title\":\"Article Created via Subrequests!\"},\"relationships\":{\"tags\":{\"data\":[{\"id\":\"{{tags-1.body@$.data.id}}\",\"type\":\"tags\"},{\"id\":\"{{tags-2.body@$.data.id}}\",\"type\":\"tags\"}]},\"type\":{\"data\":{\"id\":\"article\",\"type\":\"contentTypes\"}},\"owner\":{\"data\":{\"id\":\"{{user.body@$.data[0].id}}\",\"type\":\"users\"}}}}}",
"uri": "/api/articles",
"waitFor": ["user", "tags-1", "tags-2"]
}
]More Powerful Replacements
Imagine that instead of creating an article for a single user, we wanted to create an article for each one of the users on the site. We cannot write a simple blueprint, like the one above, since we don't know how many users there are in the Drupal site. Hence, we cannot write an article creation subrequest for each user.
To solve this problem we can tweak the user subrequest, so instead of returning a single user it returns all the users in the site:
[
…
{
"requestId": "user",
"action": "view",
"uri": "/api/users",
"headers": {"Accept": "application/vnd.api+json"}
},
…
]
Then in our replacement tokens, we can write a JSON Path expression that will return a list of user UUIDs, instead of a single string. Subrequests will accept JSON Path expressions that return either strings or an array of strings for the replacement tokens.
In our article creation subrequest we will need to change {{user.body@$.data[0].id}} by {{user.body@$.data[*].id}}. The Subrequests module will create a duplicate of the article subrequest for each replacement item. In our case this will have the effect of having a copy of the article creation subrequest per each available user in the user subresponse.
The Final Response
The modified blueprint that generates one article per user will have a response like:
Six articles are returned from a single subrequest.
We can see how a single subrequest can generate n subresponses, and we can use each one of those to generate n other subresponses, etc. This highlights how powerful this technique is. In addition, we have seen that we can combine different type of operations. In our example, we mixed GET and POST in a single blueprint (to get the vocabulary and create the new tags).
Conclusion
Sub requests is a great way to fetch or write many resources in a single HTTP request. This allows us to improve performance significantly while maintaining almost the same flexibility that custom code provides.
Further Your Understanding
If you want to know more about the blueprint format you can read the specification. The Subrequests module comes with a JSON schema that you can use to validate your blueprint. You can find the schema here.
The hero image was downloaded from Frankenphotos and use without modifications with a CC BY 3.0 license.