JSON-RPC to Decouple Everything Else

At this point, you may have read several DrupalCon retrospectives. You probably know that the best part of DrupalCon is the community aspect. During his keynote, Steve Francia, made sure to highlight how extraordinary the Drupal community is in this regard.

One of the things I, personally, was looking forward to was getting together with the API-First initiative people. I even printed some pink decoupled t-shirts for our joint presentation on the state of the initiative. Wim brought Belgian chocolates!

Mateu, Wim and Gabe in decoupled T-Shirts (with multiple consumers)

I love that at DrupalCon, if you have a topic of interest in an aspect of Drupal, you will find ample opportunity to talk about it with brilliant people. Even if you are coming into DrupalCon without company, you will get a chance to meet others in the sprints, the BoFs, the social events, etc.

During this week, the API-First initiative team discussed an important topic that has been missing from the decoupled Drupal ecosystem: RPC requests. After initial conversations in a BoF, we decided to start a Drupal module to implement the JSON-RPC specification.

The decision log after finishing the BoF at DrupalCon

Wikipedia defines RPC as follows:

In distributed computing, a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction.

The JSON API module in Drupal is designed to only work with entities because it relies heavily on the Entity Query API and the Entity subsystem. For instance, it would be nearly impossible to keep nested filters that traverse non-entity resources. On the other hand, core’s REST collections based on Views, do not provide pagination, documentation or discoverability. Additionally, in many instances, Views will not have support for what you need to do.

We need RPC in Drupal for decoupled interactions that are not solely predicated on entities. We’re missing a way to execute actions on the Drupal server and expose data that is not based on entities for read and write. For example, we may want to allow an authenticated remote agent to clear caches on a site. I will admit that some interactions would be better represented in a RESTful paradigm, with CRUD actions in a stateless manner on resources that represent Drupal’s internals. However because of Drupal’s idiosyncrasies sometimes we need to use JSON-RPC. At the end of the day, we need to be pragmatic and allow other developers to resolve their needs in a decoupled project. For instance, the JS initiative needs a list of permissions to render the admin UI, and those are stored in code with a special implementation.

Why the current ecosystem was not enough

After the initial debate, we came to the realization that you can do everything you need with the current ecosystem, but it is error-prone. Furthermore, the developer experience leaves much to be desired.

Custom controllers

One of the recommended solutions has been to just create a route and execute a controller that does whatever you need. This solution has the tendency to lead to a collection of unrelated controllers that are completely undocumented and impossible to discover from the front-end consumer perspective. Additionally, there is no validation of the inputs and outputs for this controller, unless you implement said validation from scratch in every controller.

Custom REST resources

Custom REST resources have also been used to expose this missing non-entity data and execute arbitrary actions in Drupal. Custom REST resources don’t get automatic documentation. They are also not discoverable by consumers. On top of that, collection support is rather limited given that you need to build a custom Views integration if it’s not based on an entity. Moreover, the REST module assumes that you are exposing REST resources. Our RPC endpoints may not fit well into REST resources.

Custom GraphQL queries and mutators

GraphQL solves the problem of documentation and discovery, given it covers schemas as a cornerstone of the implementation. Nevertheless, the complexity to do this both in Drupal and on the client side is non-trivial. Most important, bringing in all the might of GraphQL for this simple task seems excessive. This is a good option if you are already using GraphQL to expose your entities.

The JSON-RPC module

Key contributor Gabe Sullice (Acquia OCTO) and I discussed this problem at length and in the open in the #contenta Slack channel. We decided that the best way to approach this problem was to introduce a dedicated and lightweight tool.

The JSON-RPC module will allow you to create type-safe RPC endpoints that are discoverable and automatically documented. All you need to do is to create a JsonRpcMethod.

Each plugin will need to declare:

• A method name. This will be the plugin ID. For instance: plugins.list to list all the plugins of a given type.
• The input parameters that the endpoint takes. This is done via annotations in the plugin definition. You need to declare the schema of your parameters, both for validation and documentation.
• The schema of the response of the endpoint.
• The PHP code to execute.
• The required access necessary to execute this call.

This may seem a little verbose, but the benefits clearly surpass the annoyances. What you will get for free by providing this information is:

• Your API will follow a widely-used standard. That means that your front-end consumers will be able to use JSON-RPC libraries.
• Your methods are discoverable by consumers.
• Your input and outputs are clearly documented, and the documentation is kept up to date.
• The validation ensures that all the input parameters are valid according to your schema. It also ensures that your code responds with the output your documentation promised.
• The module takes care of several contrived implementation details. Among those are: error handling, bubbling the cacheability metadata, specification compliance, etc.

As you can see, we designed this module to help Drupal sites implement secure, maintainable, understandable and reliable remote procedure calls. This is essential because custom endpoints are often the most insecure and fragile bits of code in a Drupal installation. This module aims to help mitigate that problem.

Usage

The JSON-RPC module ships with a sub-module called JSON-RPC Core. This sub-module exposes some necessary data for the JS modernization initiative. It also executes other common tasks that Drupal core handles. It is the best place to start learning more about how to implement your plugin.

Let's take a look at the plugins.list endpoint.

/**
 * Lists the plugin definitions of a given type.
 *
 * @JsonRpcMethod(
 *   id = "plugins.list",
 *   usage = @Translation("List defined plugins for a given plugin type."),
 *   access = {"administer site configuration"},
 *   params = {
 *     "page" = @JsonRpcParameterDefinition(factory = "\Drupal\jsonrpc\ParameterFactory\PaginationParameterFactory"),
 *     "service" = @JsonRpcParameterDefinition(schema={"type"="string"}),
 *   }
 * )
 */
class Plugins extends JsonRpcMethodBase {

In the code, you will notice the @JsonRpcMethod annotation. That contains important metadata such as the method's name, a list of permissions and the description. The annotation also contains other annotations for the input parameters. Just like you use @Translation, you can use other custom annotations. In this case, each parameter is a @JsonRpcParameterDefinition annotation that takes either a schema or a factory key.

If a parameter uses the schema key, it means that the input is passed as-is to the method. The JSON schema will ensure validation. If a parameter uses the factory key that class will take control of it. One reason to use a factory over a schema is when you need to prepare a parameter. Passing an entity UUID and upcasting it to the fully-loaded entity would be an example. The other reason to choose a factory is to provide a parameter definition that can be reused in several RPC plugins. An example of this is the pagination parameter for lists of results. The class contains a method that exposes the JSON schema, again, for input validation. Additionally, it should have a ::doTransform() method that can process the input into a prepared parameter output.

The rest of the code for the plugin is very simple. There is a method that defines the JSON schema of the output. Note that the other schemas define the shape of the input data, this one refers to the output of the RPC method.

  /**
   * {@inheritdoc}
   */
  public static function outputSchema() {
    // Learn more about JSON-Schema
    return [
      'type' => 'object',
      'patternProperties' => [
        '.{1,}' => [
          'class' => [ 'type' => 'string' ],
          'uri' => [ 'type' => 'string' ],
          'description' => [ 'type' => 'string' ],
          'provider' => [ 'type' => 'string' ],
          'id' => [ 'type' => 'string' ],
        ],
      ],
    ];
  }

Finally, the ::execute() method does the actual work. In this example, it loads the plugins of the type specified in the service parameter.

  /**
   * {@inheritdoc}
   *
   * @throws \Drupal\jsonrpc\Exception\JsonRpcException
   */
  public function execute(ParameterBag $params) {
    // [Code simplified for the sake of the example]
    $paginator = $params->get('page');
    $service = $params->get('service');
    $definitions = $this->container->get($service)->getDefinitions();
    return array_slice($definitions, $paginator['offset'], $paginator['limit']);
  }

Try it!

The following is a hypothetical RPC method for the sake of the example. It triggers a backup process that uploads the backup to a pre-configured FTP server.

Visit JSON-RPC to learn more about the specification and other available options.

To trigger the backup send a POST request to /jsonrpc in your Drupal installation with the following body:

{
    "jsonrpc": "2.0",
    "method": "backup_migrate.backup",
    "params": {
        "subjects": ["database", "files"],
        "destination": "sftp_server_1"
    },
    "id": "trigger-backup"
}

This would return with the following response:

{
    "jsonrpc": "2.0",
    "id": "trigger-backup",
    "result": {
        "status": "success",
        "backedUp": ["database", "files"]
        "uploadedTo": "/…/backups/my-site-20180524.tar.gz"
    }
}

This module is very experimental at the moment. It’s in alpha stage, and some features are still being ironed out. However, it is ready to try. Please report findings in the issue queue; that's a wonderful way to contribute back to the API-First initiative.

Many thanks to Gabe Sullice, co-author of the module, and passionate debater, for tag teaming on this module. I hope that this module will be instrumental in coming improvements to the user experience both in core's admin UI and actual Drupal sites. This module will soon be part of Contenta CMS.

Header photo by Johnson Wang.