Mike and Matt talk with the leads of Drupal's "Automatic Updates" initiative to discuss the security, limitations, status, and gotchas of automatic updates.
Episode Guests
Lucas Hedding
Senior Drupal Engineer at MTech LLC. In his spare time, he is a frequent core contributor, core contribution mentor, project application reviewer, and general nice guy. He's from the Midwest in the US but now lives in Nicaragua where he splits his time between his family, clients and running a largely Nicaraguan-based Drupal services business.
David Strauss
Pantheon’s CTO, David also helps build the systemd layer that runs on many of the world’s Linux systems, protects Drupal’s integrity via the Security Team, co-organizes the annual All Systems Go! Conference, and contributes to AMP as a member of the Technical Steering Committee.
Tim Lehnen
Chief Technology Officer at the Drupal Association which entails managing the team that builds the tools that the community uses to build Drupal. Tim lives in Portland, OR and spends his free time tinkering with arduino, raspberry pi, and VR - all the trappings of the sci fi dystopia ;)
Mentioned in this Episode
Transcript
Transcript
- Matt Kleve:
- For January 9th, 2020 it's the Lullabot Podcast! What are you still all about podcast, episode 243. I'm Matt Kleve senior developer at Lullabot. With me is always close to the show, senior front end developer, Mike Herchel. Hey Mike.
- Mike Herchel:
- Hey Matt. How are you doing? Happy new decade!
- Matt Kleve:
- Happy new decade. Although some people will fight you over that.
- Mike Herchel:
- Those people can go to hell.
- Matt Kleve:
- Depending on whether you start counting with zero or one.
- Mike Herchel:
- Yeah, we're developers. We're zero indexed.
- Matt Kleve:
- Okay. I suppose. Yeah, that works. It depends on what language and what structure in that language we're talking about. So yeah, I suppose we're ready to start with zero so that's fair. That's good. So we're also going to talk about something that I think people have been talking about for a while, right?
- Mike Herchel:
- I don't know. My opinion is almost a little under the radar but I'm honestly super excited about it.
- Matt Kleve:
- It's something that you hear about every once in a while and it's in the Drupal world so we're going to be talking Drupal today, which we often do on the Lullabot podcast. Lullabot's a strategy design development company that primarily does lots of website work in Drupal, right?
- Mike Herchel:
- Yeah, among other things but a lot of Drupal.
- Matt Kleve:
- Yeah. And so today auto updates in Drupal core.
- Mike Herchel:
- Automatic updates. That sounds a little scary and it sounds a little undrupally.
- Matt Kleve:
- Well, I suppose. Let's see. We have the right people to talk to today that have been working on it and know everything there is to know and we'll all be smarter at the end of this one, right?
- Mike Herchel:
- Yeah. First up we have Lucas Hedding who is a senior Drupal engineer at MTech LLC as a frequent core contributor mentor, project application reviewer and in general a nice guy according to our notes here. He's from the Midwest in the US but lives in Nicaragua and he splits his time to his family, clients and running a largely Nicaraguan based Drupal services business. Welcome Lucas.
- Lucas Hedding:
- Hi. Nice to be on the show with you guys.
- Mike Herchel:
- Yeah, thanks for coming.
- Matt Kleve:
- You welcome. I think you're the only first time Lullabot podcast guest today.
- Lucas Hedding:
- Oh, well it's to be that honor.
- Matt Kleve:
- Glad you're here. Also with us today, we have Pantheon CTO, David Strauss. David helps build the system D layer that runs on many of the world's Linux systems. He protects Drupal's integrity via the Drupal security team. He core organizes the annual all systems go conference and contributes to amp, the Accelerated Mobile Pages project. Did I get that right?
- David Strauss:
- [inaudible 00:02:47] drops the accelerated mobile pages.
- Matt Kleve:
- Oh, okay.
- David Strauss:
- It's just AMP.
- Matt Kleve:
- It's just AMP? Okay. But my issue with AMP David is that when I Google AMP, it wants to give me Google maps. Anyway, David is a member of the amp technical steering committee. Welcome David Strauss.
- David Strauss:
- Thank you.
- Matt Kleve:
- And you've been on before. We talked with David about Drupal performance a couple of times several years ago with press flow, and more recently I'm talking about performance on Pantheon and CDNs and all that good stuff.
- Mike Herchel:
- Next up we have a chief technology officer of the Drupal association who manages the team that builds the tools that the community uses to build Drupal. Tim Lehnen lives in Portland, Oregon and spends his free time tinkering with Arduino raspberry pies, VR, all the trappings of the sci fi dystopia. Welcome T Lehnen. Welcome Tim Lehnen.
- Tim Lehnen:
- Great to be here. It seems only appropriate in 2020 to have all these trappings of the sci fi dystopia. I decided that I might as well lean in to the whole thing. That's the world we're going to live in.
- Matt Kleve:
- And Tim, you've been with us before talking about the drupal.org infrastructure team in 2017, and then in 2018 we've talked about changing the tooling that's happening on drupal.org.
- Tim Lehnen:
- Yeah, absolutely. Good to be back.
- Matt Kleve:
- So it sounds like you're all involved with this automatic updates and Drupal core stuff. So if we could just get the big picture, maybe what's going on, tell me about the initiative, that kind of thing.
- Tim Lehnen:
- Yeah, maybe I'll kick it off a little bit, just because the I think big picture idea came together when the Drupal association met with a number of different folks, core contributors and folks like that. I think probably the first really big conversation about this idea was back at Midwest Drupal summit, maybe in 2017 or something like that. And the notion of having automatic updates in Drupal is much older than that. But we came together to talk about how could we actually get it done in a Drupal context? Other software projects, in fact, these days, sort of most modern projects have some form of self updating, automatic updating feature. It's something WordPress does it, it's something that Mac OS does, that windows does, that all sorts of things do and many different software projects implement automatic updates in different ways.
- Tim Lehnen:
- So we wanted to come up with a solution that would work for Drupal and that would meet the unique needs of the kind of platforms that get built and using Drupal as a base. So there are a couple of goals behind that, right? Because when you think about automatic updates, you think who's the audience for that kind of a thing. And for the most part it's those kinds of folks out there who are maybe on more of a set it and forget it attitude towards their site maintenance thing. Not the kind of folks who have big development teams or maintenance teams who can do their dev stage prod deployment workflows and all that kind of stuff. So I would say that one of the major goals is just more accessibility and maintainability of Drupal for this sort of small to medium sized folks with this initiative. But I think it has some implications that are good across the board. So I don't know. Is that a good enough rough overview? David, Lucas, anything to add?
- David Strauss:
- I think one of the use cases that we've really been trying to meet at least early on is the site owners that are located in very different time zones than a lot of the security team. Because in many cases we've been setting up the updates in a queued up way where they might need a stamp in the middle of the night in order to rapidly apply the updates. And one of the things that we've learned from some of the major updates to Drupal over the past five years is that attackers can be on the heels of an update in a matter of hours. So that creates a complex situation for people who are having to choose between getting a night's sleep or putting their site at risk.
- David Strauss:
- And it really can make things a lot more equitable around the world for maintaining Drupal sites if it's possible for things to be set up in a place where even if it can't be completely set up and forget it, at least they can know that when an update is in the works and about to come out, that they can get their site into a good place on their schedule in a way where when the update actually lands, then the update just rolls out to their site.
- Matt Kleve:
- Yeah. So David in case somebody isn't familiar with how a security update might happen, can you walk through that process? I'm trying to remember it off the top of my head but I think you're probably a better person to just give the high level there. It's Wednesdays at Eastern time afternoon or something like that?
- David Strauss:
- I don't know much about the actual Eastern time because I'm out in California, but it's definitely on Wednesdays when we do the releases and get them out. And we'll usually let people know ahead of time if there is a major security update coming out on a Wednesday so that people can be ready to go for the deployment. We've also made a number of advancements over the past five to 10 years in improving the experience for administrators that need to be rolling out updates. For example, we always split out the actual security update into a separate point release from any kind of feature bug fix updates so that there's always a sort of minimum jump that you can make when the update comes out that has the lowest risk possible for the site while still being able to get the site past any security risks.
- Matt Kleve:
- And given the nature of open source, when there's a security announcement, everybody should know about it but when there's security announcement, everybody who isn't necessarily acting in a good way will also know about it.
- David Strauss:
- Correct. And we often see attackers be able to develop at least some attempts against something within six hours of a release coming out.
- Lucas Hedding:
- That's a pretty good segue into what are the features of what we've done with our first revision and that the very first parts that we worked on this past summer or this whole concept of a public safety announcement or a PSA will let folks know, hey, we've got an especially important security release coming out in the next couple of days, and now that'll be broadcasted on the site just like the sort of more annoying. You've got security updates but it'll happen before and it's only going to happen on these highly sensitive ones, these Drupal again type event.
- Mike Herchel:
- Interesting. So the first stage was just creating the framework for an announcement that will happen actually prior to the release that says, "Hey, you're going to need to update?"
- Lucas Hedding:
- Yeah. It gives folks that manage these sites to really take a deeper look at the site. As an engineer on many sites, what we've found is that there's a lot of things you just didn't necessarily keep up to date. Hey, I didn't factor everything in to get. Oh my goodness, I did an update last time on the site on live or some of these smaller sites that's sometimes the easiest way, download a couple of modules, whatever. Well, anyways, it lets you get a little bit ahead of the game. Hey, tomorrow there's going to be a security update. I'm going to need to be paying attention to all my sites. And that was just the first phase of the first phase.
- Matt Kleve:
- Gotcha. And when was that first phase of the first phase completed?
- Lucas Hedding:
- Tim was that?
- Tim Lehnen:
- Gosh, I want to say we had the initial work, maybe before that, maybe it was even July or August. And then in the early alpha releases of the build, that was kind of the first thing there. Essentially, it subscribes to a Jason feed from drupal.org of these PSA announcements that the security team can choose to publish to you to when they have something to put out. So as Lucas said, that's going to give you a days head up, a week's heads up. It'll depend on how far in advance they're able to do those sorts of things. But yeah, I think that came through over the summer, roughly speaking. And then we got into this sort of real meat and potatoes of things.
- Matt Kleve:
- So when you say meat and potatoes, you're talking about self updating?
- Tim Lehnen:
- Yeah. There's sort of two elements. Lucas did a lot of work around making sure you're ready for a self updating first, which I think is one of those key parts.
- Lucas Hedding:
- Yeah. There's three parts. There's the PSA and then there's what we initially called pre-flight checks, but then we weren't sure that was the best name so we decided eventually on readiness checks. And that just makes sure that hey, you're ready to update.
- Matt Kleve:
- What are the things required to [crosstalk 00:12:41].
- Lucas Hedding:
- An actual updated of the site. Some examples, just sort of wrap your mind around it. Are you mounted on a read only file system? Do you have enough disk space? Do you have database updates that are sort of sitting out there? Those would be things that are pretty critical. You can't update files if the hard drive is read only.
- Matt Kleve:
- Sure. And if you don't have enough hard drive space, like the basic mechanical requirements.
- Lucas Hedding:
- Yeah.
- Matt Kleve:
- Yeah.
- Lucas Hedding:
- So we looked at it like there's these errors and there's these warnings like hey, you've hacked a file, you modified a file in tracker module. I don't know why you modified tracker module.
- David Strauss:
- I know why he's modified tracker module.
- Matt Kleve:
- To disable it?
- Lucas Hedding:
- Because she didn't want to use it.
- Matt Kleve:
- Yeah.
- David Strauss:
- I forked at one point in time for drupal.org.
- Lucas Hedding:
- Well, okay. But are you going to want to keep node module which is probably a lot more in use on your site from being upgraded if only the node module was what was fixed with a security release.
- Matt Kleve:
- So is your code base in line with what it should be? We don't want to make replacements that are going to break changes that you have made for one reason or another.
- Lucas Hedding:
- Yeah, I say hack but it's really usually a patch in most cases.
- Matt Kleve:
- That sounds a lot like, what was it, hacked module or something like that that would go through and see where the files have deviated.
- Lucas Hedding:
- It does but you know what? Hacked module has a lot better tools now. The Drupal association worked very closely with David and myself and others that were contributing to this initiative. And we've now got SHA-256 sums of every single file of every single release, of every single project, theme, core, distribution back through the ages and you can now download this archive of the SHA sums and verify that nothing has been modified from what you got off of drupal.org, and you can do it securely.
- Matt Kleve:
- You said for every release, what happens if I'm running a bleeding edge version of a contributed module or something?
- David Strauss:
- Like if you're running a dev release?
- Matt Kleve:
- Correct.
- David Strauss:
- The generation, I'd need to talk to Neil Drum a little bit because he's working on some of this, but the generation of the SHA sums is on every patch release of every module of core of things like that. I think actually it may exist for the tip of the branch Lucas, I'm not certain. We'd have to double check.
- Lucas Hedding:
- It wasn't or if it is, we don't really support it in the module because we intentionally don't check against that. Who's to say that your head in my head are not the same thing.
- Matt Kleve:
- Sure. And it would be every commit through the tree and that would be a lot.
- Lucas Hedding:
- Yeah. So we're just doing releases.
- Mike Herchel:
- So what does happen if there's some type of new mega Drupal [inaudible 00:16:03] vulnerability in the node module but for example, I might have my DB blog module patched or something like that?
- Lucas Hedding:
- You'll have gotten warnings and if it was a super critical and you even had gotten the PSA the day before, and maybe you weighed the pros and cons of whether or not you needed to take that patch out or not and you decided to leave it in. And the next day the update is out there and we checked to see if we can apply it. The only thing that changed was no dot module will update you.
- Tim Lehnen:
- Okay. Even if other components are patched or hacked or things like that, as long as they're not effected by what's in the update, it should still work in most cases? Of course, there's interdependencies sometimes that might not be caught which is why that throws a warning.
- Matt Kleve:
- But it does check. So for example, say if I download this patch early before the auto update module can get to work and I apply this patch, when auto updates gets to work, it'll fail to apply the patch and give a message?
- David Strauss:
- It'll [inaudible 00:17:15] to apply the patch because you've got a modified system.
- Matt Kleve:
- Gotcha.
- Mike Herchel:
- What type of notifications are given to the user at that point? Does it say like, "Hey, you have this Uber critical security update?"
- Tim Lehnen:
- You can enable or disable the notifications. I'm trying to remember if you can enable or disable a email or not or if that's just sort of bundled in with the notifications. I have a feeling that I said that that was a separate thing. And so then the site administrator, whoever gets the emails with the uptick dot module when something is out of date, those same people get the emails for a super important security thing is coming out.
- Matt Kleve:
- Got you.
- Tim Lehnen:
- Or if a update failed or if it was successful, you also would get that signal notification. There's no way to turn that one off. I decided that that one seemed a no brainer, don't give people too many options, just let him know that something happened important.
- Mike Herchel:
- So this is going into Drupal core. That's the ultimate goal. Correct?
- Tim Lehnen:
- That is the ultimate goal. Yeah. There's some gates we'll have to get through to get it into Drupal core. So we've been talking so far, I think we sort of talked around what capabilities are included so far. So we talked about GSAs, we talked about readiness checking and we didn't go into detail about how we apply the the changed code base but Lucas sort of alluded to it, right? You checked to see if there are any unexpected changes and make sure it can apply and if it applies, it is pretty much a simple file overlay of the code base from the take the changed files and put them in place of the older versions of those files that you have in your code base.
- Tim Lehnen:
- The trick is in its current form, in this first release, it supports Drupal 7 and Drupal 8 core updates only. We don't have any contrib updating supported yet. We also don't have good support for composer based, primarily composer utilizing installations so there's some problems to solve. So as we've talked with the core team about this to start, we know that they want a couple of things. They want to see support for a wider arrange of some of these other things, they also want to make sure that there's some more robust tooling around sort of rollback or even sort of staging the changes so there's an idea for how we would do that, that maybe David can talk about it [inaudible 00:20:02]. But yeah, part of the next set of things here is we've been working with core to identify, okay, so what are the things we need to resolve before we can begin to actually move this into core, and what are just the other feature paths that we need to keep working on to cover all the use cases that will help protect the community and help maintain people sites.
- David Strauss:
- I think we'd be leaving something behind if we didn't mention that for as long as something like this has been discussed, there has been a pretty decent resistance to it as far as I wouldn't want my web server updating itself. How have we solved that in this era and why is this a good idea?
- Lucas Hedding:
- I think part of what we've through these tools is assistance even for the people who don't want to use the auto updating itself in the sense that even if you don't actually take that final leap of having and apply the update when the release comes out, we're still providing a lot of assistance to help people keep their sites up to date in the sense that what we were calling the pre-flight checks or the readiness checks, the PSA distribution. So in some ways we've tried to accomplish a sliding scale of capabilities where an administrator can choose where they're comfortable automating and they'll still be able to enjoy all the more basic capabilities of the tooling even if they don't take that final leap of having them applied automatically.
- Matt Kleve:
- Yeah. And that said, as Lucas has sort of architected the actual implementation that goes in there, between the readiness checks and the way that the sort of file overlays applied, there are some protections in place. I think we've aired certainly on the more cautious side of not trying to apply updates to avoid breaking sites rather than applying something when the site might not be ready to receive it. So if you are the kind of maintain or who pins your kernel versions and all this stuff all the way up and down your stack, you can still use those sort of workflows. You by no means have to have this turned on but I think the notion of having this as a sort of hopefully eventually the same default for sort of an out of the box installation just makes sense.
- Matt Kleve:
- There are a million and a half or more websites. Some percentage of those will be very well maintained by a team of expertise that wants to manage this on their own. But a lot of them, we want to make sure they're protected too. So yeah, that configurability that David described I think makes a big difference. And then if we just go ahead and speak to the sort of the future of this, right now you have this system where if something gets screwed up in the process, it sort of automatically rolls back the attempts to do the file overlay. But when we start doing more complex things like doing dependency resolution for composer updates and things like that where you can't just do a simple kind of undo a command to just undo that work. One of the things that David proposed is creating sort of like a boot loader for your Drupal site and that's something that I think kind of directly addresses that point of giving people control but also taking a lot of the work out of the effort. Do you want to describe that idea to David?
- David Strauss:
- Sure. A lot of it's inspired by what I've seen from the world of Libostree, Core OS and Chromium OS. And even to some degree, the design of network hardware, like Cisco appliances where what happens in a lot of these cases is the active system is effectively read only and there's an alternate route system, whether it's a firmware image or it's a more complex file system that allows all of the manipulations to occur in an area that is actually isolated from the current running system. And that substantially reduces the risk of applying complex sets of updates where we might be having to update many files. Some of those files might have interdependencies that are not easy to know which one needs to be applied first. These designs also introduced the ability to atomically switch to the updated state once it's available as well as an intrinsic ability to roll back, at least the file portions of the updates so that if an update fails, there's always that backup image that you were last active on that remains available until it's discarded.
- Mike Herchel:
- So that's kind of interesting. To make sure I understand this. So you're somehow going to create a duplicate file system that's isolated from the actual file system of Drupal and apply your updates there and then somehow flip a switch, or you're going to verify the updates and some pile flip a switch and kind of make the updated environment, the actual file system environment? Am I understand that properly?
- David Strauss:
- The only part that I'm trying to juggle in terms of the description and understanding against my own mental model of it is that there's no particular place that is the primary file system in a perpetual way. Get you're always sort of on a branch and you can update the branch that you're not on and then whenever you're ready to, you can switch to that branch and then once you've switched to a different branch than you can update any of the other ones that you're not on. Some designs intrinsically have two like Cisco's firmware design and Core OS, some designs support an arbitrary number like Libostree. But one of the big benefits is providing a workspace in order to perform complex operations. If we're trying to spider dependencies for a composer build, we may have a cascading set of changes that need to be applied as soon as one version gets bumped.
- David Strauss:
- For example, if a module applies to security update by moving from the one point X to two point X branch of an HTTP client module, that may have some large cascading changes to the tree that need to be then applied in a way where if you have any partial application of that change set, things may just be broken. So right now we can sort of escape some of that with the current auto updates because we queue everything up to be able to rapidly apply changes generally to only a handful of files, and we're able to structure that data in a way where the client has a packaged version of the change set that needs to get applied basically, where it doesn't really have to perform a lot of calculations on the client side.
- David Strauss:
- And even though a partial application of an update might cause a problem, the chance that that will occur or be over a long period of time is very low with the types of updates that are rolling out through the system as designed right now. The kind of AB switching or however, we've sort of started calling it an AB switching pattern allows us to have that working area be separate from the active site, which also simplifies some of the tooling around administrator tools because there's always risk in updating yourself in the sense that if you update yourself in a way that you break yourself then it can be very hard to roll back and update or provide any interactivity as those are occurring.
- David Strauss:
- And a lot of the design of all of these phases has been built around the idea that we can't make things worse in the sense that we can't make things less secure by virtue of having auto updates enabled, we can't create a substantial risk of site breakage because auto updates are enabled. Because all of these things will chill adoption of these features and the goal is to have as little chilling of the adoption of these features as possible.
- Matt Kleve:
- Yeah, that totally makes sense. Are there any other web content management systems or applications that do any type of boot loading or boot loader type architecture that you're describing?
- David Strauss:
- It's certainly present for the model in Java for war files where let's say you have a Tomcat server and you drop in a war file for an application and you want to update the application and you replaced the war file, I believe that's a fairly atomic switch.
- Matt Kleve:
- That's interesting.
- Tim Lehnen:
- I want say symphony four has the concept of swapping out your phone and controller or some of these parts. Does that sound right to anyone else?
- David Strauss:
- I think you might be right. I don't know how complete their implementation and the idea is.
- Tim Lehnen:
- it's an API. You still have to build to the API.
- David Strauss:
- But to the point, I think this is treading some new ground to try and come up with a method of update that relies on this sort of a framework. But I think ultimately that's where we resolve that earlier question about how do you drive maximal adoption in spite of that whole category of users who might be otherwise scared off by the idea of the software updating itself.
- Matt Kleve:
- Yeah, we got pretty deep there. Can we just kind of back up and clarify exactly where we are today versus where we're going in the future? So today we have the PSA stuff and the pre-flight stuff that's been renamed but I liked the name pre-flight. What was it called?
- David Strauss:
- Readiness checks.
- Matt Kleve:
- Readiness checks. There we go. So we can do PSAs and readiness checks, and that's through a contrib module? Am I getting that right? Okay.
- David Strauss:
- Correct.
- Tim Lehnen:
- And we can do core updates on seven and eight as long as the site isn't based on a composer workflow.
- Matt Kleve:
- And what is this contrib module called?
- Tim Lehnen:
- Automatic updates.
- Matt Kleve:
- Automatic updates. Straight up. Cool. And so these automatic updates that are updating core in a very basic way are still based on Drupal being writeable by Apache but that's better than not having an updated Drupal site and that's why we're willing to make that trade?
- Tim Lehnen:
- It is. And we do it all in the same HTTP request and if anything blows at this and knocks it over and something fails, immediately we'll step back all within the same HTTP request.
- Mike Herchel:
- One thing I also wanted to dig into that you were just mentioning there is in the idea of the writeable file system for something like PHP, because one thing we also looked at for these designs is to not increase the attack surface in the sense that for any level of control an attacker might gain over a site, we didn't want their power over the site to be better just because auto updates is enabled or they have a configuration that allows auto updates to be used. And one of the areas is of course rating PHP files to disk that then are running for the application. But one of the things we looked at is how things like say twig templates get compiled and other derived assets get created for say aggregated JavaScript.
- Matt Kleve:
- The way I understand it, Mike, as a friend editor, you can probably talk circles around me. The pretty twig template you see ends up being compiled to PHP and then executed as such. That's something that happens fairly transparently to most people.
- Mike Herchel:
- Yeah I know that there's compiled to IG files. I've never poked at them.
- Matt Kleve:
- Okay. Am I right? I'm right, David, right?
- David Strauss:
- Yes. To my understanding, they're compiled to executable PHP because it makes a substantial performance difference to have it that way because things like the op cash and PHP can then actually cash the interpreted version of that code in a way where it can be rapidly loaded out of memory without really parsing the vials off desk when those templates get used. But what it does do is meant that we opened the door to the idea of the active Drupal application being able to write files to a part of the file system that then will get executed as PHP. And there's nothing particularly right or wrong about that but it's just a door that we've already gone through for Drupal, and so being able to update itself and its own PHP files doesn't necessarily represent any new territory that an attacker can then take advantage of to compromise a site in a greater way simply because you're using automatic updates and have a writeable file system for Drupal's own core PHP and not just for twig templates.
- Mike Herchel:
- That's interesting. So the way I'm understanding what you're saying is that being able to update its own files isn't a big deal because we are a generate executable files for twig? Yeah, and if the hacker could do write our own against its own file system, it could also be so against twig so it's already in.
- David Strauss:
- Correct.
- Matt Kleve:
- Yeah, the other area there, and I don't know if we want to spend kind of the whole time on the security side of it.
- Mike Herchel:
- Well I think it's something that we can't gloss over either because that's been an argument that's been held for a long time.
- Matt Kleve:
- Yeah.
- David Strauss:
- Yeah. So if you say, okay, it's safe to write, or it's in principle safe to write these executable files, then just walking you through the same logic we use, then the next step in the chain that needs to be secured is the delivery of the files that this thing is going out and choosing to drop on there. So this is where we built the architecture that actually secures the delivery of the update from drupal.org all the way through to when the automatic updates module applies that to make sure that you're not getting poisoned update files.
- Matt Kleve:
- So you're not using FTP?
- David Strauss:
- No.
- Lucas Hedding:
- Oh goodness no. Absolutely not.
- Tim Lehnen:
- We could though. Because we're not actually reliant on the delivery path of the files as part of the trust framework.
- David Strauss:
- Absolutely correct. We have built a higher level security controls. It doesn't really matter whether you're doing it over [inaudible 00:36:01], over a fancy handshake or carrier pigeons.
- Matt Kleve:
- That's really interesting.
- Mike Herchel:
- Yeah. I will give the layman's version because that's my level of understanding. And then maybe if we want to dig in. So there's a package signing architecture basically, right? So as Lucas mentioned earlier on the drupal.org side what we've done is we have a hardware security modules that we've used to generate a root key. And we use that root key through these hardware security modules to generate intermediate keys that expire after a certain amount of time. So we might have an intermediate key that let's say last 30 days and then after that, the automatic updates module won't trust any files signed by that intermediate key anymore, right? So then we generate that key using our hardware security modules and the root key to make a new intermediate, we put that intermediate on drupal.org's signing infrastructure.
- Mike Herchel:
- And then when drupal.org is doing that hashing of the files, it's also sort of signing the package manifests so that the automatic updates module can verify, okay, this package that's coming from a site that tells me it's drupal.org was signed by an intermediate key that matches the route public key or was not and therefore should be discarded. And that means that just as David and Lucas were saying, it doesn't matter how you got that package, the method that the package gets to you could have been totally insecure but as long as that verification matches or doesn't match we can decide whether or not it's safe to apply that package.
- David Strauss:
- Correct. Totally correct.
- Matt Kleve:
- I like it.
- Lucas Hedding:
- Yeah.
- David Strauss:
- I did want to share what motivated us to do it this way as well, which is that one of our concerns in terms of broad availability of using these tools is to be able to secure the transit of updates even to hosts that may not actually be in a great configuration from a modern trust anchor perspective in the sense that sometimes the hosts may have old versions of PHP, they may have old versions of root certificates for certificate authorities, generally for use with X509 and TLS. And so we wanted to build a foundation that could be modern and trustworthy even if you're deployed to a target that is sort of equally, or not equally but has not necessarily been well-maintained.
- David Strauss:
- And at the same time we didn't want to make any compromises to the delivery of these updates to very modern systems in terms of we didn't want to pick up old style designs in order to maximize compatibility. So we have this interesting middle ground which is ultimately the same exact kind of design for trust that the Python update framework TUF uses, which is the idea that you want to anchor the trust in the project and not necessarily rely on certificate authorities and HTTPS alone. It also opens up the ability to use mirrors more broadly for the update distribution, which is another important thing because preventing a site from getting an update is another way to attack a site.
- Matt Kleve:
- We're talking automatic updates with Drupal core with some of the folks that are helping make it happen on the Lullabot podcast. Coming up right after this we'll talk a little bit more about where we're going.
- Mike Herchel:
- Hey, it's Floral Drupal camp organizer and DrupalEasy podcast host, Mike Anello. How are you doing Mike?
- Mike Anello:
- Hey Mr. Herchel, how are you? Thanks for having me.
- Mike Herchel:
- I'm doing pretty good. It's kind of chilly outside.
- Mike Anello:
- I know. It's like low 70s. I don't know how you and I are going to survive down here over the winter.
- Mike Herchel:
- Yeah, I know. In February, specifically around February 20th, it's going to be about the same, right? That's what temperature?
- Mike Anello:
- Yeah. If only there was a good reason for people to come down to Florida the weekend of February 21st, 22nd and 23rd.
- Mike Herchel:
- Now that you mention it, Mike Anello, organizer of Florida Drupal camp, isn't Florida Drupal camp that same exact weekend?
- Mike Anello:
- Oh my goodness, Mike Herschel co-organizer of Florida Drupal camp, I think it is.
- Mike Herchel:
- Yes. And it's going to be nice, warm and sunny. We're going to guarantee that. We have foresight into that. How many days is Florida Drupal camp?
- Mike Anello:
- It's three days. The first day is a training day. That's Friday the 21st, the second day is Saturday the 22nd is wall-to-wall sessions and food and fun. And then the third day, Sunday, the 23rd is like a two thirds day. We end a little bit early. We have a few sessions but it's mainly a contribution day.
- Mike Herchel:
- Yeah. It's a contribution day. And when we say a contribution day, we actually have a good amount of people contributing, correct?
- Mike Anello:
- We do. And we've got arguably one of the top contributor, trainer, organizer, herders who's going to be there, Amy June Heinlein.
- Mike Herchel:
- Yeah, absolutely. And she gets stuff done and points people in the correct way. I'm one of the organizers for the new Defoe Drupal theme, which is going to be named [inaudible 00:41:47] which there'll be a subsequent podcast on. But I'm planning doing a sprint then, we're going to be kind of hopefully deep, deep, deep in to the teaming process and where are you going to need a lot of help?
- Mike Anello:
- So where and when can people register for this amazing event?
- Mike Herchel:
- Oh, that's a good question. They can register right now at fldrupal.camp, and the early bird tickets are $50. I believe the tickets go up to $75 January 17th. The campus located in Orlando, which Orlando you can get flights to from anywhere and yeah.
- Mike Anello:
- All right, well come on down to the camp.
- Mike Herchel:
- Yeah, I agree. All right, well thanks. Mike Anello organizer of Florida Drupal camp for being on this podcast.
- Mike Anello:
- Oh no, thank you.
- Mike Herchel:
- Hey, welcome back to the Lullabot podcast. We are talking about automatic updates in Drupal and how it's a little scary, but it's going to be awesome.
- Matt Kleve:
- It is partially, right? For core updates through a contrib module.
- Mike Herchel:
- Yeah. To tell you the truth [inaudible 00:42:59] just a little bit scary but hopefully going to be a little less so.
- Matt Kleve:
- Hey Tim, I have a quick question if we could.
- Tim Lehnen:
- Sure.
- Matt Kleve:
- We talked a little bit about the PSAs, and we're able to deploy core updates and roll them back and it works like it should mostly at this point. What was required on Drupal site infrastructure wise to make that happen?
- Tim Lehnen:
- Oh, gosh. There were a number of different components creating just the feed to serve the PSAs was new development work.
- Matt Kleve:
- And that's something that lives on drupal.org ?
- Tim Lehnen:
- Yes, exactly.
- Matt Kleve:
- Okay.
- Tim Lehnen:
- That lives on and is published from drupal.org. And in fact that's documented on drupal.org's API documentation. So that same feed of PSAs could be used in other applications than just automatic updates. And then beyond that we had to collaborate with Lucas and others to actually implement the signing infrastructure, both the hashing of the packages when releases are made, but also everything from the hardware secure security modules all the way up through to the sort of signing Oracle virtual machine that's part of our hosting infrastructure.
- Lucas Hedding:
- Did a lot of this work. And I would say it's probably the coolest part of it is just well, there's the security signing of everything, but we now have a quasi patch approach which anyone could use it too, where if you want to take any version of any module or a core and give it a version range in the URL, you'll get the full contents of every single file that is the diff between those two versions. That's how we got around updating things so that we don't have to worry about whether or not you've got patch utils or anything else to do a true patch. It's just the whole node dot module file because copying and pasting will always work. If applied might not.
- Tim Lehnen:
- Yeah. So all of that gets delivered from drupal.org and can be used and will be extended for the rest of this work, but could also be creatively extended by other folks as well. It'll be interesting to see as this matures, whether people who operate things like the LTS program for Drupal decide to extend the readiness checks or decide to do something else with these sort of differential crossing patches and things like that. It'll be really cool.
- David Strauss:
- I think we mentioned that the European union funded some of this work initially.
- Matt Kleve:
- Yeah. That was my very next question.
- Tim Lehnen:
- Well I was going to say that tag one was also a very instrumental in a lot of the work too. They provided project management and other components to this initiative for the last year, and tag one happens to be one of those LTS vendors. And so we catered a little bit in our design to making sure that this would accommodate them and might drop wizard and others that want to sort of keep the ball rolling once Drupal 7 stops having full support.
- Matt Kleve:
- Yeah, it's valuable. It's awesome that a Tiguan was able to do that. Thank you Tiguan. So they provided project management support and you said the European commissioner, European union provided assets or how did that work?
- David Strauss:
- So the European commission has a program, it's called ISI FOSSA.
- Matt Kleve:
- Who is the European commission as an noble American? I don't know.
- David Strauss:
- Sure. As an entity, the European commission is a body that is part of the larger sort of European Union apparatus. And the European commission has several different divisions that are all focused on programs for the good of the European Union member States and the kind of population within that, and to sort of support a larger scale in European Union operations. So one of the divisions within the European commission, I think it's called digit, it's their digital services division and they're who we've been working with. And the team over there is great and they implement and maintain a lot of the European union and European commission's digital presence and most of that's built on Drupal.
- Matt Kleve:
- Ah, there it is. Okay. I got it.
- David Strauss:
- Yeah. So they've reached out. We've collaborated with them on a few other things or related to funding a security bug bounty program and stuff like that as well. And so probably 18 months to two years ago, we pitched them on this idea of the automatic updates initiative and they were very generous in their funding and allowed us to as the association to contract with Lucas, to work with other members of the community, to work closely with Taiwan and all the various contributors to actually get us as far as we've gotten. So they had a one year funding program that took us through all of 2019 and got us to our first stable release. And so we'll talk about this a little bit more as we get to the end but we'll be following up with other organizations as well to see if we can find some of that stuff that we've been talking about, the second phase two.
- Tim Lehnen:
- That's awesome. And maybe segue into where we're going and there was a little bit of conversation about this earlier. Some of the things that were mentioned were composure support and contrib support. So working for a company that does large Drupal 8 websites, I believe pretty much all of our projects are composer now.
- Matt Kleve:
- Yeah. Mike and I were in a call this morning with some Lullabots and we know we were going to talk automatic updates on the podcast today and that was everybody's first question is, how is this going to work with composer? And it sounds like there is some ideas in play but a lot of work left.
- Tim Lehnen:
- That's absolutely true.
- Lucas Hedding:
- Unfortunately, that's the truth. We've got a plan, mental plan but we always get back to the point where I've got a 256 megabyte livable small machine and I can't run composer on it and composer takes up too much memory. How are we going to handle my use case? But I feel like that's a little bit like a phase two or phase three because we've got a lot of hosts that can support a running composer on the server. So let's learn some things from those. Just the whole approach the whole time has been let's try to introduce new features almost like a sprint, almost like you're running the open source project like a scrum project. And every time we do something, we tweak something, we implement something.
- Lucas Hedding:
- But even before composer, we want to do this, we went into the weeds about AB and being able to have two index.phps that can respond to requests and swap those out. So there's a lot of things that can happen even before we get to the composer. But maybe a first implementation is don't do anything with AB. We'll work independently on high memory composer update. Nothing's stopping you from doing that right now except well, a lot of time and effort and probably funding.
- Mike Herchel:
- Yeah, all the usual resources.
- Matt Kleve:
- Do you think it would be interesting for us to be seeing the telemetry data on how often this set up for a site that is using composer makes it viable for us to run auto updates in the sense that we suspect that a substantial portion of the sites that are using composer as their model for code project management may have read only file systems in the production site.
- David Strauss:
- Yeah, that makes a lot of sense like the large enterprisey type sites that Lullabot creates typically are not going to use auto updates. So maybe that first use case is when people are downloading the tarballs and selling stuff via that way.
- Tim Lehnen:
- And one of the concerns with that is just that like, as we say, understanding the right use cases and understanding how we can accommodate them and where the real value and impact is. And another element of this is as of the latest Drupal 8 eight release, for example, even the tarballs on drupal.org are technically composer ready installs. So they're already set up with the file structure and everything as if they were a composer, create project style projects. And those still work because if you haven't further modified the lock files and things like that from there, that's still fine and that's still going to work. But it's just once you start requiring new new packages and new modules that we run into these limitations. But in principle I think there's a few things and I think doing a sort of componetize architecture will make sense.
- Tim Lehnen:
- So we obviously need the capability to say, "Hey, here's a site with some current code base, here is a set of updates and here is a way to run the composer dependency resolution for that set." And maybe in certain enterprise scenarios that's something that you don't plug into using the production site core module but you some might plug into your CI pipeline for either your sort of ongoing regular testing or deployment process. Maybe it's something where we provide some extensibility that can plug into a whole platform, Pantheon or Acquia or platform as H, maybe their hosting platforms can have a system that uses some of the tools we're building as part of maybe their staging systems rather than just running in place on production environments. But those are all ideas that I think we're still exploring.
- Lucas Hedding:
- Early, early on at mid camp this past year in 2019, I had some conversations with Matt Glam from commerce and they're looking at doing a service offer. I think they've already launched their service offering in this space and we all knew about Pantheon because they're in the booths across the way. And a lot of what we built this pluggable using plugins in Drupal 8 especially where you can swap out the implementations or extend and add your own additional plugins, say, "Hey, let's do a backup before we apply database updates using Drush because I know I got Drush dressed in my site."
- Lucas Hedding:
- Well, we can't assume that for every once we don't have a out of the box plugin for that. But if on your site you've got Drush or an S three bucket or something where you want to dump a database backup, there's lots of room for folks to add features and functionality and secret sauce for their providers, as a provider to their clients. I don't know if people even have dug into the code to know that this is possible.
- David Strauss:
- Yeah. One thing I wanted to touch back on that sort of opened up this topic is the question of contrib updates as well. And one thing that I've at least held as, I can't speak on behalf of everyone but my perspective is that much of accomplishing contrib updates is the same scope as accomplishing the composer integration because we have so many contrib modules that now rely on composer derived dependencies.
- Tim Lehnen:
- Yeah. From a practical point of view, there's definitely technical work. There's definitely some brainstorming, mind-bending thinking that needs to go on to really land on exactly the right kind of architecture for this. But there are some good ideas together thanks to the people we've already had in the room. And then from a more brass tax point of view, as someone who comes from the sort of project management side of things are myself, on the Drupal association side, we're spending the next couple of months letting the initial release soak, letting it prove out a little bit and getting some feedback. We're then at the same time, I'm working on approaching other folks who might be interested in sponsoring. When we can secure some more funding and resources, I think we'll do a phase of discovery again about these particular problems and then move into that problem solving. And yeah, it would be great to keep up the momentum. So I'm hoping I'll be hearing from folks out there listening to this who are interested in supporting the initiative.
- Matt Kleve:
- Yeah. So how can people support the initiative? What can people do to help?
- Tim Lehnen:
- There's a few different things that folks can do. So one is people who are interested in supporting this but are coming from a technical perspective and not ability to get funding can certainly come in, start using this on sites, make bug reports, could also start collaborating on issues or could offer to help co maintain. Lucas is the primary maintainer of this control module and we've had a certain amount of funding that we've been able to devote to support things like this. But at a certain point, this is open source, this is Drupal so we're going to need to have ongoing maintenance and so that would be a great way to get involved. And then for those folks who are at organizations who might be interested in funding another round of work, you can contact me directly at the Drupal association, my contact information is on the staff page. It's just tim@association.drupal.org and I'm happy to chat with anybody who's interested either in just learning more about this or potentially helping us move it forward.
- Matt Kleve:
- So we've been talking about this automatic update initiative and putting it into Drupal core and we haven't said a whole lot about putting it in a Drupal core. What's the plan? What do we have to do?
- Tim Lehnen:
- Yeah. So it's going to go together I think with solving some of these problems. I think that part of the concern from speaking to core maintainers or at least managers, framework managers, things like that is they do want to make sure that we have all the safety nets that what's supported by the automatic updates module when it becomes a core module maps well to all of the environments and places we say are supported for Drupal itself, right? So we say you can use Drupal on a shared host of this size with these PHP versions and things like that. We want to make sure all those are in place.
- Tim Lehnen:
- So that said, over the course of the next month, there might also be value in deciding, hey, maybe for the D7 version where this composer stuff is not such a concern, maybe that's something we would want to do sooner rather than later for all the folks who are still on that or maybe we can do some more incremental things. So those are sort of ongoing discussions. But short version is we want to truly handle the scope of a Drupal site and not say, "Hey, you can update a Drupal site except all of these and these and these and these. We want to get a better coverage of all the use cases.
- Matt Kleve:
- Is it possible to get this into Drupal 7 core earlier than Drupal 8 since you don't have to worry about composer?
- Tim Lehnen:
- I am not totally sure really. That's a core maintainer question more than me. In principle I would think so.
- Lucas Hedding:
- In principle yeah, but just, I'm taking off my hat of maintainer ship of any of this stuff. I haven't seen a lot of velocity on Drupal 7 in the last two years. I doubt it's going to land for the next six months because we're looking at about six.
- Tim Lehnen:
- Everybody's so focused on nine.
- Lucas Hedding:
- Yeah. But as far as eight and Drupal 9, let me quote something from extra jammer Jess from five days ago on a Slack meeting. She says, "Oh, it's absolutely belonging in the core product once it's done. It's the most strategically important future for Drupal's future."
- Matt Kleve:
- Wow.
- Lucas Hedding:
- And that's from the core maintainer ship in that community. There's a lot of support. A lot of hours were given by the Drupal association, funding from European commission. Lots of people are interested. People are just giving up their own free time. Thanks David and others that worked tirelessly over the last year or two to make this happen, and we're starting to see some fruition. We need more bug reports.
- Mike Herchel:
- What's the definition of done?
- Lucas Hedding:
- What's the definition of done for anything?
- Mike Herchel:
- Somebody has to name it.
- Lucas Hedding:
- So in our case, I think when we can say running the site on a complex composer side is safe and running an update will cover your contrib, I think that'll be done for the purposes of core and collision in eight and nine. I'm pretty sure that those will be the gates.
- Tim Lehnen:
- I think the only question we have around that is really if there's any subset of that that is also going to be satisfactory for core inclusion, not because we would stop working on it to achieve that point but is something somewhat less than that good enough?
- Lucas Hedding:
- Yeah. And it might be but I think until we've really got into the future set of discovery and determine what those subsets even would be in principle, I don't think we'll quite know. The other thing to say though is right, so automatic updates has been talked about probably 10 years or more as a potential thing that should be important, right? And the amount that we accomplished in a single year once we were able to resource it was pretty monumental. And I think as big as handling composer and contribs sounds, I think again, if we can nail down the resources, we can probably actually make progress quite quickly.
- David Strauss:
- Mike, you are a user of automatic updates. I watched you doing it today.
- Mike Herchel:
- Yeah. I did it a little bit earlier as I was talking to Matt and that's when I ran into that one particular issue where I was mentioning the automatic update Slack and Drupal Slack.
- David Strauss:
- Which we should point out exists, right?
- Matt Kleve:
- Yeah. That'd be a good place to follow along anybody out there.
- Mike Herchel:
- Let's see, is it auto updates or automatic updates? I'm looking right now.
- David Strauss:
- Just auto updates.
- Mike Herchel:
- Auto updates. So it's the auto updates and Drupal Slack and you can find out more about Drupal Slack at drupal.org/slack I believe.
- David Strauss:
- Did you get your Drupal 7 site updated?
- Mike Herchel:
- Not yet, no. I'm going to take a little bit of time and take a lot of little bit of screenshots just to try to document the problem that we have. And at that point, open up an issue on drupal.org.
- Matt Kleve:
- Yeah.
- Mike Herchel:
- But something that I have been doing is I've been, like I mentioned my friend's lawn care website and in return I get free lawn care which is probably about the best deal ever. So this is a website, it's a Drupal separate website. I don't use features or anything to manage to configure it, I just update everything live. I use backup migrate to download the database. Everything's a little bit of fast and loose and everything.
- Matt Kleve:
- It is in a code repository. I saw that. You haven't yet.
- Mike Herchel:
- And so it is in the code repository so that's something. It didn't start off that way but it definitely is. So I said, "Well, this is kind of a perfect use case for automatic updates. I'm going to maybe download it and see what breaks." And I've been finding a couple of small little bugs here and there and posting it in the slack channel on drupal.org and I think that this is an important initiative so I'm happy to help out.
- Matt Kleve:
- And I think anyone who does this kind of thing professionally ends up with those little side websites that you end up maintaining for mom or something and I think you're right. It's a really great use case and I'm looking forward to not having her email me and tell me that my site needs updated because Drupal is now emailing me and telling me my site is out of date.
- Lucas Hedding:
- You just got to disable that module Mac. Come on you know that.
- Matt Kleve:
- I try and say, "No mom, that just means that it's still up. If you're getting an email, that's a good thing." But we know that's not true.
- Mike Herchel:
- And with that, I think we're at our time here. So let's see. Tim, anything that you want to add on as we're wrapping up here?
- Tim Lehnen:
- Well, just a big thank you for your individual contributions to test this out, but also for hosting us and having us on to talk about the initiative and help get the word out. I think it would be fabulous if people try this out in the wild. I think just the support of the community will be really instrumental in helping us make this happen. So again, anybody out there listening who might be interested in getting involved can reach out to me, tim@associationdotdrupal.org. And thank you to my colleagues on the call who've helped us make this happen.
- Lucas Hedding:
- Just wanted to thank our hosts as well and all the collaborators on this. It's one of the things that makes automatic updates quite a challenge to even get to the point where we're at today is because it's so interdisciplinary in terms of infrastructure, in terms of procedure, in terms of software implementation, in terms of testing. And we've had a lot of people had to come together to make this a thing.
- Matt Kleve:
- Thank you for coming onto the Lullabot podcast, and I'm looking forward to seeing this progress and hopefully, again in Drupal core I think it'll be a big deal.
- Tim Lehnen:
- Yeah. Awesome. Thanks again.