For the past several weeks, long-running debates in the Drupal community have been in the spotlight. Since DrupalCon London in August, discussions about the maintainability of Drupal Core, the nature of Drupal as a web framework, and its proper target audience have generated hundreds of comments. At the heart of the current debate are some key questions:
- Drupal core has become larger and more difficult to maintain in recent releases. How can we make it easier for increasingly overloaded core developers to maintain the software?
- The rapid growth of Drupal has led to a much more diverse pool of users. How can we account for the different -- and sometimes incompatible -- needs of that very diverse userbase?
- The same complexity and tight coupling that overloads core developers also makes building major products and sites on top of Drupal difficult; how can we simplify that work?
Those Sound Serious
In many ways, these concerns overlap with each other. The traditional solution to serving more diverse needs is to add more functionality to Drupal core, but that exacerbates the problem of developer burn-out and increases the complexity of building focused products on top of Drupal. On the other hand, pruning lots of code from Drupal core would reduce the maintenance overhead and the complexity of customizing it, but would leave site builders and newcomers with a less useful tool.
During the post-DrupalCon Code Sprint in London, old discussions about removing modules from Drupal core were reignited. Patches to remove our old friends, the Blog and Profile modules, were committed to the Drupal 8 branch. The fate of other "feature modules" like Poll, Forum, and Trigger, is actively under discussion, though it's unlikely that all of them would be removed. While everyone seems to have a hitlist of modules they rarely use, and a whitelist of ones they'll fight for, the tactical discussions about individual patches makes it easy to lose sight of the strategic purpose of "refocusing" Drupal core.
A Way Forward
Core developers Nathaniel Catchpole (catch) and Daniel Kudwien (sun) have been two of the most active participants in the conversation, and my "Product, Framework, or Platform?" session from DrupalCon London summarizes my take on some of those issues. Last week we had a chance to talk about some of these issues, and while there's still a lot to iron out, quite a bit is becoming clear. Rather than simply hunting for modules that experienced developers often ignore -- and evicting them from Drupal core -- our community needs to assess what Drupal core is and what it should be. There's broad agreement that we need to establish a set of heuristics that our community can use to weigh new functionality against its shared vision of what Drupal should be, and examine existing functionality to see if it's still appropriate for core. These are hard decisions, because our community has rarely invested in building consensus around Drupal's purpose, target audience, and internal architectural philosophy. The current explosion of interest in these topics, and the demand for viable solutions, tells us that the time is right to tackle them.
What is Drupal?
One of the key problems we face in talking about Drupal is the relatively clunky terms we use to describe different parts of our relatively complex software stack. Often, discussions pit Drupal's potential as a "Development Framework" against its possible future as a "Product for end users", but that binary split gives a false picture of how our system really works. In his 8/31 blog post, Nathaniel Catchpole addressed this issue by naming five distinct layers of the Drupal software stack, and I think they're a great way to break things down.
The Web Framework
The foundation of that stack is the essential "web framework" code for Drupal. It's the portion of Drupal that provides essential services any web application would require. Rendering data into HTML and other formats, identifying and loading plugins, handling HTTP requests and responses, caching, database and storage access, and so on all live at this layer; today, most of this code is found inside of Drupal's includes directory.
The Web Framework portions of Drupal are actually the least "Drupally." Serious core developers and maintainers of large modules are familiar with it by necessity, and its performance and stability is important, but none of the functionality at this level is unique to Drupal. In the future, we can ease our complexity and maintenance problem by using code from other PHP web frameworks instead of reinventing the wheel. In Drupal 8, for example, we're replacing certain parts of Drupal's low-level framework layer with well-supported code from the Symfony Project.
The Building Blocks
Moving up in the stack, we find the "building blocks" layer that Catch referred to as the "Platform" in his blog post. This is where we begin to encounter Drupal-specific approaches to CMS and web building problems. The Entity system, users, nodes, taxonomy terms and vocabularies, actions, roles and permissions, FieldAPI, text formats, and our concept of "Blocks and regions" live here, and can be used to assemble features and functionality. Reusable user interface tools and UX components are also here: things like drag-and-drop tools, Drupal 7's Overlay mechanism, and so on are all reusable tools used to craft usable interfaces. Most of the functionality at this layer is invisible or irrelevant to the people who visit a Drupal-powered web site: they don't see Views and flags and nodes, just galleries and favorites and articles. That type of visitor-oriented functionality still has to be assembled out of the building blocks. Most of the "Big ideas" in Drupal's history made their splash at this layer: Node Types, CCK, Views, Context, and so on. As Effulgentsia points out in his comment on Catch's post, the Web Framework and Building Block layers are what most people want to keep when they talk about "streamlining" Drupal or making it into a simple framework. Although a Drupal site could run using nothing but the components that exist at this layer, it would be hard to build it without the piece that comes next...
The Site Building Application
If the raw materials of a Drupal site live in the Building Blocks layer, the tools for assembling those materials into a working web site live here, in the "Site Building Application." It contains the tools that are used when building a new site from scratch via Drupal's own web interface. Examples include administration forms for creating content types and modifying their fields, as well as modules like Views UI that exist solely to create and configure new site elements. (This layer doesn't include day to day user management and content management, however -- just the tools used to actually create and configure a site.)
Often, we're used to lumping this part of the Drupal stack into the same category as the building blocks themselves. A well-built Drupal site, however, doesn't actually need the Site Building Application to run smoothly: like the booster rockets on the space shuttle, its job is to 'launch' the site into existence. Later, if new features need to be built, an administrator can also use it to make the changes. Why bother discussing it as a unique layer in the Drupal stack? Over the past several years, more and more tools have been built to provide alternate ways to set up and configure a site. Modules like Views, Flag, Rules, and more can export their configuration to files, and canned modules can provide ready-to-use functionality to users who don't have the time or the experience to build them from scratch. Tools like Features allow even more complicated functionality to be bundled together, and command-line tools like Drush can handle some of these setup tasks without ever touching a web browser. Installation profiles can automate the process as well, setting up a fully-configured site for end users without the intervention of a traditional "Site Builder."
User-Facing features
We've talked about the framework, the building blocks, and the site-building tools that we're all familiar with. Now comes the layer that excites everyone: user-facing functionality.
Features like photo galleries, blogging, discussion forums, online stores, social media integration and real-time chat all live here. Content administration tools like the node editing forms, user management screens, and so on also live here: the people who manage a Drupal site every day are its users, too! The Workbench Suite, developed by Palantir, is a great example of a feature/functionality module that's aimed at a site's managers and editors rather than its visitors. Regardless of who the audience is, functionality that lives in this "Feature Layer" is about the day to day operation of the site, rather than the creation and configuration of the site itself. In the olden days of Drupal, features that lived at this layer were almost always custom-coded in PHP. Want a custom content type? Write a module. Want special behaviors like a calendar or a listing page or an RSS feed? Write a module. Many of the "Feature Modules" that ship with Drupal core (blog, Forum, Poll, Tracker, and so on) come from this era. Over the last five years or so, tools like CCK, Views, and others have changed the game. Rather than downloading a custom Photo Gallery module, we combine Views, Content Types, Image Fields, ImageCache, jQuery Sliders, and so on. Sometimes the "recipes" for great features get passed around as blog posts, screencasts, and so on. Tools like the Features module have also emerged, allowing site builders to export the configuration choices they've made into a standalone module. It's still 100% configuration and combination of Drupal's Building Blocks, but it brings back some of the simplicity we lost when we moved away from the "Just turn on the module, and you have a Gallery" era. As the standard solidifies, projects like The Open App Standard may provide another approach to packaging and distributing these end-user features, but the place they occupy in the Drupal ecosystem is the same.
End-User Products
The capstone on this Drupal stack is one we're not as used to talking about in the community: products for end users. When all the tinkering and programming and theming is done, a "product" is the end goal. A single-user blog for a book lover, a portfolio site for an artist or photographer, a local newspaper's web site, and so on are all concrete "products" formed by the careful use of all of Drupal's other layers. What really lives at this layer? Often, they're one-off web sites built for a particular person or organization's needs. Increasingly, the world of Installation Profiles and Drupal Distributions has started capturing common types of sites in the form of reusable products like Open Atrium, Drupal Commons, and OpenPublish. Hosted services like Drupal Gardens, Buzzr, and SubHub are also products, because they package a particular administrative interface with a particular set of features, and target users with a set of needs more specific than simply "building a Drupal Site." At the annual Do It With Drupal conference, one of our tracks is a parade of Fantasy Sites -- clones of popular sites like Stack Overflow, BaseCamp, Flickr, and others built in Drupal. They're built on a short timeline using existing Drupal building blocks, and deconstructed for the audience by the team that created them. It's consistently one of the most popular events at the conference, because people love seeing how compelling sites were created. One of the greatest challenges faced by new users of Drupal is the spartan quality of the standard installation profile that it ships with. Although it pre-configures a handful of content types, it doesn't target a particular use case or type of web site, and leaves many users shaking their head as they evaluate the whole platform. Real products living at this layer of the Drupal stack are all about making clear and conscious choices, then implementing them with the tools Drupal provides.
The Drupal Platform
There we have it. The five layers of the Drupal Platform! At the very top, most visible to visitors and new evaluators, is the array of "Products" that can be built with Drupal. From hosted site-building services to high-end enterprise intranets to mom and pop web stores, it's where we shine. Although Drupal Core doesn't offer much on this front by default, successful Drupal products show newcomers its power and flexibility. These products can be thought of as combinations of features -- choices about what the purpose of each site really is, and decisions about what user-facing functionality should be combined to serve that purpose. The Site Building Application is our "Product Zero" -- the piece that site builders use to construct their own features and products. Books are written about using it properly, new modules expand on its functionality, and tools like Features and the Drupal 8 Config Management Initiative are designed to simplify the process of capturing and reusing the things that are built with it. The Site Building Tool would be useless, though, without the raw materials provided by Drupal Core and popular contrib modules -- the conceptual building blocks of every Drupal site. When earth-shaking changes hit the Drupal world, they usually start here with new ideas like "Content Types" or "Views" or "Fields." The building blocks, too, would be useful without the hard-working application framework that is Drupal's foundation. Though it rarely gets much attention, that very bottom layer of the Drupal stack is doing its work every time a page is served up. The boundaries and dividing lines discussed in Catch's blog post and this one aren't perfect, but they give us a useful way to describe and discuss the way Drupal actually works today, as well as where we want it to be in the future. Looking at Drupal as a "five-layer platform" can also help us identify places where our existing code and modules bleed across those boundaries. That 'bleed" between layers makes maintenance and ongoing core development harder, and it makes building new and innovative products on top of Drupal more difficult.
What does it mean?
So after all those words, what have we gained? Does this way of discussing Drupal help us understand some of the burning questions our community faces? Looking at our ecosystem and our codebase through the five-layer-lens can help us, and I think there are several critical conclusions we can draw.
- The long tail of Drupal users build features and products with the middle layer -- the "Site Building Application." They may know PHP and write custom modules to fill gaps, but the "Framework" and "Building Block" layers are a means to an end for them. Even code-centric developers need something at that level, whether it's Drush or custom-written tools.
- Calls to split the "framework" and "product" portions of Drupal core into separate projects would allow developers to obtain a slimmed down version that only includes the first two layers. However, the question of what should be bundled into the primary "Welcome to Drupal" introductory download would still remain. We can't avoid strategic choices simply by changing our infrastructure.
- One of the biggest pain points articulated by developers is the huge number of interconnections and cross-dependencies between different parts of Drupal Core's code. While it doesn't solve all of our problems, segmenting our code effectively along the boundaries discussed above can help. This would take discipline, though: loosey-goosey interaction between different subsystems and different layers is a big part of Drupal coding today.
- Dissatisfaction with canned features like Blog, Forum, and Tracker spiked when the community shifted to a "building blocks" approach. Those modules are useful, but no longer reflect our preferred way of solving problems. They're easier to understand, but less flexible, and have stopped evolving.
- Our preferred way of solving problems -- combining small building blocks into reusable features, and features into finished products -- is baffling for newcomers who have not yet mastered the building blocks themselves.
Where do we go from here?
There's no single step that will solve the kinds of problems that core Drupal developers are facing, or that newcomers to the project face as they try to ramp up. We can tackle individual challenges, though, as long as we keep the big picture in mind. One way we can improve the situation is refocusing the tools that ship with Drupal core. We can simplify our code and serve site-builders more effectively by strengthening the building block functionality that comes with core, replacing the "canned" features like Blog and Forum with ones constructed from the building blocks, then shedding the canned feature modules that we've been dragging along for years. Finally, using use-case targeted installation profiles to give Drupal evaluators and new site builders a "running start" with common types of sites would help smooth our notoriously brutal learning curve. Focusing on just one or two of those steps means that Drupal is likely to suffer, either from code bloat or a crippled experience for site building evaluators. The importance of Drupal's many constituencies makes this holistic approach a necessity. When it comes to the nuts and bolts of Drupal's codebase, we've also accumulated a lot of technical debt over the years: we need to pry apart tightly coupled systems, reduce our dependence on oddball "Drupalisms" when we could use native PHP language features, and determine where our approach to extensibility is hampering developers rather than helping them. There's a lot of great work going on in those areas, but it often happens in isolation. Treating it as part of a holistic effort to reduce Drupal's complexity is important. Over the next several weeks, I'll be posting more about several of the important projects that are converting the current discussions into actionable issues on Drupal.org. In the meantime, check out these "Meta-Issues" where important posts and code tasks are being collected, and weigh in here -- what do you think the future holds for Drupal's different layers?