Matt and Mike talk with the Drupal Association's Tim Lehnen and Neil Drumm about the changes to Drupal.org's tooling.
Episode Guests
Tim Lehnen
Tim is the Engineering Director of the Drupal Association.
Neil Drumm
Neil is a Senior Technologist at the Drupal Association
Mentioned in this Episode
Transcript
Transcript
- Matt Kleve:
- For January 25th, 2018. It's the Lullabot Podcast. Everybody, it's the Lullabot Podcast, episode 222. I'm Matt Kleve, a senior developer at Lullabot. With me is always senior friend and developer, cohost of the show, Mike Herchel. Hi Mike.
- Mike Herchel:
- Hey. How are you doing?
- Matt Kleve:
- Re-roll that patch.
- Mike Herchel:
- Yes. Re-roll that patch. What does that mean?
- Matt Kleve:
- Well it's Drupal speak for, you did it wrong and you need to do it right and turns out that that's not exactly the way people talk anymore. There's a movement going on to change our vocabulary, right? To upgrade it?
- Mike Herchel:
- Yeah. Something more modern.
- Matt Kleve:
- To the 21st Century and we've got some people on the line today, to talk about tooling on Drupal.org.
- Mike Herchel:
- Yeah. First up, we've got Tim Lehnen. He's the director of engineering at the Drupal Association. [inaudible 00:01:04] on Drupal.org. He's been in Drupal.org for almost 12 years or probably over 12 years now. From northeast Portland, welcome Tim.
- Tim Lehnen:
- Hey, thanks very much for having me and I want to a say thank you to Lullabot as well for hosting this podcast. Keeping the community informed and for being a Drupal Association premium supporting partner. You and other organizations are a really big help to us and help fund all the cool work we're going to be talking about today, so we really appreciate it.
- Mike Herchel:
- Thank you.
- Matt Kleve:
- I just get the warm fuzzies when I hear that. Also with us today, we have senior technologist with the Drupal Association. He's Drumm on Drupal.org, where he's been for over 13 years with the user ID with four digits. Starting with a 3, in Brooklyn joining us today, we have Neil Drumm. Hey Neil.
- Neil Drumm:
- Hi. Glad to be here.
- Matt Kleve:
- Awesome. You two were both here back in March, episode 207. We talked about the infrastructure Drupal.org ran on and your role with the DA in doing that, so welcome back. Glad you're here.
- Neil Drumm:
- Thanks. I think last time, we were talking about some retooling of the infrastructure. Some new partners we brought on and things like that. Hopefully, your experience has been like ours and things have been running pretty smooth since then.
- Matt Kleve:
- Well if you do it right, we don't notice, right?
- Mike Herchel:
- If you do it wrong, oh boy.
- Neil Drumm:
- Then Twitter explodes.
- Matt Kleve:
- You know, there have been a series of blog posts that I found on Drupal.org and we'll make sure we'll link these to the show notes. Kind of talking about modernizing the workflow for people who are working on Drupal. What was the impotence behind that and where do we stand now? Somebody want to kind of just give a situation here, like what's going on?
- Tim Lehnen:
- Sure, let me go ahead and give the basic stance here. This is a conversation we had over multiple iterations over close to 10 years really, about how the Drupal community builds Drupal, how we build the software and the way that we've build our processes, it predates when GitHub was a thing.It predated Git for that matter. There's been a long ongoing conversation of, when is the right time to go, "Re-rolling patches is 20th Century and poll requests are 21st Century and how do we get those on Drupal.org? Really, Neil is the one who knows the whole history here and can talk about it going back really to the beginning.
- Mike Herchel:
- Yeah. Let's go ahead and talk about the history. How did Drupal start off? I know we were, I think initially, under a CVS, is that right?
- Neil Drumm:
- Yeah. It used to be two CVS repositories. One for Drupal core and one that all of the modules held into. If you got access to maintain a module, you could technically go and maintain all of the modules, which there were maybe 50 of them at the time. Occasionally Dries would commit stuff to all of the modules, to keep up with updates. Keep stuff compatible with things like API changes and core. Since then, around 2010, we completed the move to Git. I don't know the exact dates but we've been talking about pull requests in some form another, really during that migration.
- Mike Herchel:
- I remember you reading about the Great Git Migration, which I always have this awesome vision in my head of buffalo going across the plain about Git. I don't know.
- Matt Kleve:
- Is it like an Oregon Trail reference?
- Mike Herchel:
- Yeah, something like that.
- Matt Kleve:
- Like The Great Git migration.
- Tim Lehnen:
- That's pretty much the same thing I get in my head. Yeah.
- Matt Kleve:
- Well if you look at the logs of a contributed module, you'll find there's a commit message in many of them, at least the ones that existed at the time. Talking about The Great Git Migration when the code was brought into the Git repository on Drupal's infrastructure.
- Mike Herchel:
- What does the workflow currently look like? If I want to contribute to a Drupal module or a Drupal core, what do I have to do?
- Tim Lehnen:
- As we talked about at the beginning, it's a patch base workflow and I think Neil, why don't you run us through the beginning to end, from issue to commit?
- Neil Drumm:
- At the start, there is a bunch of finding the issue you want to work on and understanding it and everything. Then once you're to the point where you know what you're doing and writing some code or testing some code or anything like that, you pull down a copy from Drupal from Git and start hacking on it and maybe apply patches from PBS iterations of the-
- Tim Lehnen:
- Sort of test all that locally?
- Neil Drumm:
- Yeah. Test it locally. Get your development environment working. Once it works, there is a Git command, Git diff and it's a patch which you upload to the issue kill.
- Mike Herchel:
- Yeah. A patch is basically just a text file with a dot patch extension. You do your Git diff and a little arrow thing and you output that to a file that's issue number.patch and at that point, you go to the issue, you browse to it and at that point, you can upload it, right?
- Neil Drumm:
- Yeah. It's uploaded as a regular file, [inaudible 00:06:39] of the screenshots and everything else, if it's something visual.
- Mike Herchel:
- I remember a couple of years ago, there was a little bit of changes to that where, now all the patches show up at the very top of the issue queue, as opposed to in line with the comments.
- Neil Drumm:
- Yeah. That would have been with the Drupal.org we designed, which I think was 2008-ish. It was a couple of years before the Git migration.
- Mike Herchel:
- Got you. I thought it was a lot more recent but it doesn't matter.
- Matt Kleve:
- Regardless, developers are still passing around these patch files.
- Tim Lehnen:
- Yeah. Exactly. The process today is still, someone uploads a patch, someone downloads it, manually applies it locally, test it, decide something's wrong with it. Changes it, uploads the new version, mark the old one obsolete. Rinse and repeat until they feel like they've got something worth doing. Then we set the issue to needs review and people will use a third party tool, like the browser extension Dreditor that the community's been maintaining for a long time. To go in, do code review. Leave comments and if they verify that all that's working, we get an RTBC and then our core maintainers or our project maintainers go in and they're the ones who would go through and commit that code. It's completely familiar to people who were around in the list serve era of open source contribution but it's not what folks are used to these days.
- Matt Kleve:
- As a developer on a project, I'm definitely in the GitHub repo and Wikipedia tells me that GitHub's been around since 2008. Their pull request process, has really improved workflows for a lot of people as far as, I have a branch and I push that branch and these are my changes.Then I create a pull request and somebody can come in and do that review, look at my code, see what I could have done differently and then merge it back in, with a nice point and click interface too.
- Mike Herchel:
- What is a merge request or a pull request, Matt Kleve, just for those who might not know?
- Matt Kleve:
- Sure. It's essentially that diff file, presented in a pretty way and since it's on a separate branch usually, Mike you can check out my branch locally, within your repository and test those changes locally or just look at my merge request, using the pretty stuff on the screen and perhaps create inline comments saying, "Matt, on line nine, you really screwed up there," and review it.
- Mike Herchel:
- It makes sense. It takes a away a lot of the manual drudgery of uploading an actual text file, keeping track of your text files and going through that whole workflow.
- Matt Kleve:
- We want to be better on Drupal.org, right? We want to actually get into the modern era and kind of go that direction, is what it sounds like.
- Mike Herchel:
- Yeah and I think, talking about the goals just on a high level is, we want to have that sort of familiar style of merge request workflow, where you can open a merge request from your, we'll just use the generic term, development work space, back to the canonical repository for the module or for core or for something like that. We want you to have that inline editing capability, code review capability. All those features that you come to expect from using tools like, GitHub, like GitLab, like Bitbucket, whatever kind of code contribution tools we use these days. At the same time, there's a few things about the Drupal workflow that are kind of unique to the way we do things at Drupal. As we were doing our evaluation we were like, "Okay, which of these tools can preserve that notion of a single threaded contribution flow, where one person sort of starts with the ball, gets the work to a certain point and then someone else picks up the ball.""Keeps going, keeps going, keeps going and they work on a solution." One of the challenges in the past is that, the standard GitHub model, is a forks, a forks, a forks model and you usually wind up with just a single person working on a fork and then that one being ... The maintainers picking from five different forks, which one actually goes back and belongs in place.
- Neil Drumm:
- Yeah. Drupal's a bit unusual in that, it's pretty common for multiple people to work on a single issue, at least in Core. You know Contrib, those are smaller projects. They're more close to the average open source project on the internet.
- Tim Lehnen:
- About 18 months ago, we decided ... Well let me step back further. Probably three years ago, there was another push to say, "Hey, GitHub's come a long way since 2008, is that now a viable option?" There was a lot of analysis and things. I believe Angie Byron, web chick, did a lot of the vetting an organization and community feedback around that time. It sort of stalled out. People kind of decided the tools were quite there yet, weren't quite ready for Drupal's needs. Really now 18 months ago, we've decided to pick it up again, just because GitHub has been evolving even more. GitLab is now on the scene and becoming very popular and rapidly iterating on their tools. Bitbucket has been evolving quite a bit and there's been a big investment in their tools. We decided the time was right to do another round of evaluation. Putting that together, we had ourselves on the Drupal Association team. We had a couple key volunteers. I want to call out, Michael [Hess 00:12:17] is one, who's been extremely helpful. Then we had a committee put together by the board, the technical advisory committee and that had [Mosh 00:12:27] Angie and Steve [Francier 00:12:28], who were all there to kind of help guide our thinking. Vet kind of the research that we had done and let us know, hey are we off base or are we on base, is this going in the right direction?
- Matt Kleve:
- What are the requirements, what do we need?
- Tim Lehnen:
- That's a great question. As simply put as I can make it is, we need a developer workflow that's familiar to everybody outside our community. Merge requests, pull requests, whatever we call it, it has to be that thing, that that person who had learned how to code on GitHub, knows how to use. Then we need in addition to that, the ability to preserve the many to one collaboration. The idea of many developers collaborating on a single solution to a problem. We also want to preserve things like, you know, we have special taxonomies in our issues, for things like the notion of being RTBC, reviewed and tested by the community. We want to try and preserve that, if we can. We want to try and preserve our credit system because Drupal's kind of actually a leading open source eco-system, in terms of the fact that we have a crediting system for individuals and organizations, when it comes to code contributions.From a workflow requirements perspective, those are the main things and there's some technical requirements as well but those are not quite as critical in the initial evaluation.
- Matt Kleve:
- Does it matter where the software lives?
- Tim Lehnen:
- It does to a certain degree and it depends also on the vendor. At the moment, we would still like to have a retention of our data. There's a few reasons for that. One is just, an open source project, we want to have control. If we partner with someone who's venture funded and they disappear, we want to make sure that we've already been able to get that data where we needed it.There's all sorts of reasons and risk to using hosted solutions and there's also risk to hosting it ourselves, in terms of maintainability, scalability and all of that. That's a balance and it was part of the decision making process for sure.
- Matt Kleve:
- You know, Neil's really smart in Git's open source, can't we roll it ourselves?
- Tim Lehnen:
- That's a good question and that was definitely one of the options on the table.
- Matt Kleve:
- Was it seriously an option? I mean, that seems like a lot of work?
- Tim Lehnen:
- Well Neil, why don't you talk about that?
- Neil Drumm:
- For actual work, it's not that much different, depending on [inaudible 00:14:53] or work with a tool. Tool certainly bring in some nice things that we'll get sort of for free but the migration into it and maintaining those integrations, will still be something to maintain.The real danger is, we're a pretty small staff on the engineering team and we have careers. At some point, one of us might get another job and if it's this kind of one off contraption that only one person understands, that's a bit of a risk.
- Mike Herchel:
- That makes a lot of sense. You talked a little bit about the evaluation criteria with the pull requests and kind of many to one. Have we considered changing that entire workflow, maybe doing the multiple pull request thing or just kind of going to something that's more standard across the industry?
- Tim Lehnen:
- That's a really good question and it's something we've discussed a lot throughout this process. The first thing I will say is, what we're going to wind up with, even in the initial transition, is going to feel pretty close to those standard processes. It's going to have a few things that sort of encourage us to or facilitate us doing things in the traditional Drupal way better.It's still going to be possible to have multiple pull requests from multiple kinds of proposed code solutions on a single issue. I think there's kind of a workflow separation that's not very well understood, between the Git workflow and even the pull merge request workflow and kind of the issue management workflow. In terms of understanding proposed solutions, understanding the current problem and threading the conversation.There's a couple of issues there. When it comes to Contrib, for example, we could pretty much stand up a GitLab instance or create a dedicated GitHub account and Contrib could totally live there. We'd lose our concept of contribution credits and a few other nice to have features and things like that but it would totally work for the Contrib space. It gets a lot more complicated for Core. Core's our use case, where we want to meet their needs, while meeting Contrib's needs, so that we're not maintaining two wholly separate workflows. That was one of our really big challenges, is figuring out how to accommodate Core's needs, together with Contrib's needs. That lead to, I expand upon this in the blog post but that lead to this notion of, "Okay, let's deploy a system that uses you know, GitHub, GitLab or Bitbucket.""One of those three options, on the back end for the Git repo management, for the merge request management, for the inline editing, for the code review but let's retain our current issue queues, so that those processes that are in place, primarily for Core but also for Contrib and also for our current CI and things like that, won't be interrupted. We have a six months release cycle and when we do a big tooling transition, we need to be able to do most of the back end in tooling part first and then we need to start thinking about, are there other pieces that we can use? You know, could we use a different CI system and hook that up to the existing Drupal CI system in some way? Could we enable other features, that we're not going to start out using? Could we use the [Camben 00:18:28] boards or other features that are provided by some of these tools?Part of it is, we don't want to interrupt our release cycle, by pulling out the rug from under everybody and part of it is, we want to kind of progressively transition the community over to these tools.
- Mike Herchel:
- Just to make sure I understand this properly. Drupal.org would continue to hold the actual issues, have the buttons to actually do all the actions but the back end of it, would be hosted by GitLab, Bitbucket, GitHub, etc. is that correct?
- Neil Drumm:
- Yeah. Another reason for us wanting a kind of the canonical copy of each repository is stuff like, the left pad thing a few months ago or I think one of them had another instance of packages being substituted for different packages. I don't know how much we want to call them out. Yeah, we have more restrictive rules for what could be deleted or renamed on Drupal.org, than third party hosted things.
- Tim Lehnen:
- That's not to say that you would never see the user interface of some of these third party hosted tools. Again, when you were doing the merge request flow, you'd be seeing the Bitbucket UI or the GitLab UI. If you were doing code review, you would be doing that with those code review tools. You'd have a mash up between the issue queues and these party tools in this first iteration.
- Matt Kleve:
- The list, the GitHub, the GitLab, the Bitbucket, those are the three last standing options? I read the blog post, where it talks about each of them but that was kind of what it came down to?
- Tim Lehnen:
- That's what it came down to and to be totally honest, where we actually started was, just keep going with home brew custom tools and GitHub and GitLab, Bitbucket wasn't even on that list at the beginning of this 18 month long process. Yeah, those were the things that started and it's been an interesting sort of series of conundrums.There's all the technical requirements, there's all the workflow requirements and then there's some philosophical requirements with each for example. GitHub is used for all these open sourced projects but it's not itself open source. GitLab is open source for its community edition, although its enterprise edition also exists and is only source available. Bitbucket has a more than a decade of providing licensing for their software to open source projects but they're also not open source, just to sort of source available projects. We had those concerns to think about as well, looking at these three options.
- Matt Kleve:
- I didn't realize I guess, GitLab was an open source project itself.
- Tim Lehnen:
- It is. GitLab's community edition is available open source and hosted on GitLab. Their EE, their enterprise edition has some additional features and support, that are not open source.
- Mike Herchel:
- I want to back up a little bit. Going back to when you mentioned Contrib could fit on something like GitHub pretty easy. Why not keep Contrib separate from Core and the reason I'm asking is, you know, one of the main drivers behind this initiative is, just the onboard new developers, new Drupal developers in general.New Drupal developers coming in, are not going to be doing Core patches. They're going to be doing Contrib. Has that been considered?
- Tim Lehnen:
- Yeah. We talked about that actually and there's a few reasons we don't think we want to go down that path. Reason number one is, maintainability of having two different code contribution landscapes. That notion that, okay ... For example, a lot of times issues are opened against a contrib module that are actually Core issues and need to be moved from one to the other.That's the kind of thing that would become a manual process or we build a tool for, [crosstalk 00:22:49] two different processes. There's also the point Neil raised earlier, about the level of security. The notion of things being moved out from underneath people. The notion of people suddenly relying on and depending on modules that are in somebody's personal name space fork, rather than the canonical one. They receive things there. Lastly, we actually think it's not necessary. We think we can do it to provide Contrib. It's going to take some work but I think we can reach a point, where we can provide Contrib with that much easier, more familiar workflow and accommodate Core. It's going to take this progressive process to get there. After 18 months of looking at this, we think we can get it done.
- Neil Drumm:
- Yeah and I'd say, this is not something to attract new developers. If you're working with Drupal, hopefully you're doing it because of the platform and community and something you want to work on. This will let people hit the ground running, when they do decide to ... There's something they want to fix with Drupal.
- Tim Lehnen:
- Yeah. It will increase philosophy and it will lower barriers to entry. I think would be a good way to put it.
- Matt Kleve:
- We've mentioned the big three, the GitHub, the GitLab, the Bitbucket. Let's talk about each of those, coming up right after this and talk about kind of the way forward and where we're moving.
- Speaker 5:
- Your local Drupal community is the place to get involved and hone your chops. Coming up in February, the community has the US covered coast to coast. February 3rd and 4th, you have your choice between Drupal Camp New Jersey and Princeton or the Pacific Northwest Drupal Summit in Portland, Oregon in President Stay Weekend, February 16, 17 and 18. Slide on down to Florida Drupal Camp in Orlando. There's plenty more on the horizon across the globe, so head on over to [Drupalhelp 00:24:43].com. Find your nearest local event or meet up or if you're ready to level up but can't make the trip, point your browser to Drupalize.me and stuff your brain full of carefully crafted videos and tutorials.
- Mike Herchel:
- Welcome back. We're talking about Drupal.org tooling with Tim and Neil from the Drupal Association.
- Matt Kleve:
- You know, for about seven years now Mike, I've heard, "Why don't we just put it all in GitHub?"
- Mike Herchel:
- Yeah, why not? GitHub sounds great. Everybody knows how to use it. Didn't they invent Git?
- Matt Kleve:
- No. Linus invented Git.
- Mike Herchel:
- Yeah. I know. That was just a horrible joke. It's kind of a misperception.
- Tim Lehnen:
- No but the truth is, that's a question we get a lot and it's definitely a question the community has struggled with for years. There's a few reasons for that. I would say, five years ago, GitHub was kind of lacking in features. Three years ago, GitHub thought they were the only game in town. Two years ago, GitLab showed up on the scene and started adding all sorts of new cool stuff to their product. GitHub sat up and said, "Oh wait, we've got to start improving this." Now they've made a bunch of great improvements but they're still at an interesting position, in terms of their model. We had a lot of internal conversations with the GitHub team. For example, we talked about the fork workflow, verus Drupal's workflow. They talked about some kind of just interesting regrets they had, about how they had internally built up their system and some of the problems we talked about in the first half, where it comes to forks to forks to forks and things moving around and things getting deleted that are dependencies and all sorts of stuff they didn't predict, when they were first building their tool. When it comes down to it, the reason we can't use GitHub is, we want to be able to maintain control of data for ... Because the Drupal project has a tighter control over module management because of our security team practices, because of a few other reasons. There's only two ways to solve that really. One would be, if GitHub allowed us to run our own instance, as a public instance. That's currently, you know, some of their tech guys are like, "That would be cool. We'd love to do that," but their legal team doesn't do that. They want the GitHub experience to only live on GitHub itself. Living on GitHub itself would be okay but we'd just, we'd totally lose that control. A fork of Drupal Core would be one click away into some random name space.Community conduct enforcement, all this other kind of stuff, would just be very, very difficult, if we simply move things just to a straight hosted solution. Okay. For all those reasons, we can't use GitHub right now. We don't have that control of data. We can't protect our community from dependencies on forks, that unmaintained modules that we can't reassign. Security concerns, in terms of the security team being able to do review and important things like that.There's these other options in the landscape and GitLab i think, is the one that has the most sizzle, the most [curb 00:28:00] appeal. I think people are really excited about it. It's being used by a lot of freelancers and a lot of enterprise organizations. It feels very familiar to people how have used GitHub. They've got a ton of VC funding recently and they're moving really, really, really fast. So fast, that that can occasionally lead to some problems, which I'll get into in a bit. Then the other one is a kind of a old standby, that doesn't quite have that same curb appeal but is a really sophisticated, well maintained, well documented tool, which I Bitbucket.
- Matt Kleve:
- Let's talk about GitLab real quick, if you don't mind. Let's jump; in there. I was on a project, where they ran their own instance of GitLab. It seemed to work as expected. It had the features that I came to expect, out of GitHub and a couple other niceties.I remember, before GitHub came out with a similar feature, I was able to fix a merge conflict directly in my browser and edit the files and push that commit and it was really slick.
- Tim Lehnen:
- Yeah. It has several things like that. The inline editing tools are quite good. Code reviews are quite good. Their pipelines features are really pretty excellent, in terms of whether you're using that for CI purposes or other kinds of workflow pipelines. They added issue boards and a number of other things. They're trying to be a comprehensive tool suite, a monolithic tool suite for code collaboration. That's great and they're moving really quickly in that direction but that kind of monolithic attitude is where we ran into some problems. Again, we worked with their team for most of that 18 months. We had direct contacts with them, all that kind of stuff, to talk about some of these issues. For us, there's a few places where we ran into some trouble. One is, for the same reasons I mentioned for GitHub, we wanted to self host, rather than use their cloud version. We did things like standing up prototypes and creating prototype integrations with Drupal.org. We got a lot of things working reasonably well but we ran into some problem. Like for example, the SSH key management and controlling whether that was related to your [D.O 00:30:20] profile, so we would keep it related to other commit history and issue credits. Being able to only control it, be it API, rather than let people edit their user profiles and wind up with different keys in different places. At the moment, their system isn't architected for pluggable componentized use of the] software, which is really what we would need. Turns out, that as I was drafting my blog post, they were drafting ... By they, I mean Sid, the CEO and some of this team over at GitLab, were drafting a new vision for a plugin architecture for GitLab.My blog post goes up and Sid, the CEO of GitLab shows up in the comments and says, "Hey, wait, wait, wait, wait. I know you said not yet for us but let's talk. We've got some new stuff coming." That's going to be really interesting and we've been talking to them a little bit more since then. Because with a plugin architecture, that would let us do that phase deployment we talked about in the first half. We can pick and choose, we want to start using their merge request, we want to start using their code review tools. We want to keep our issues for now but maybe later, we want to turn on some of their CI tools or we want to turn on one of their other features and it lets us do it in that progressive way. It would also, something like a plugin architecture would let us, if we eventually went to using their issues, have a contribution credit system, right? That's something we would lose if we use anybody else's issues. That's really important.
- Mike Herchel:
- Is that something that GitLab is definitely doing? Are they currently working on it, do they have a timetable?
- Tim Lehnen:
- They are currently working on creating a plugin architecture and we've talked about what the scope of their plugin architecture will include. How much of the UI you can effect, by writing your own plugins. What you can turn on, what you can turn off. What's accessible, you know, what you can extend in the API. We've had some initial conversations there. I can't give you an exact timeline and I don't want to speak for them because I was talking on some internal calls, so I don't want to go into any details, that they might not want to make public. They're moving on that very aggressive. As far as I can tell, it's pretty much their next product priority. They seem to do major leases almost every two months or so. We'll see.
- Matt Kleve:
- It sounds like they want to work with us, that's a good sign.
- Tim Lehnen:
- They do. They seem very motivated to work with us. It was kind of a first for me to have the experience of the CEO of a major venture capital funded organization, show up in a comment thread on my blog post and email me and be like, "Hey, can we just get on the phone like right now?"
- Mike Herchel:
- That says a lot though.
- Tim Lehnen:
- It does.
- Mike Herchel:
- The fact that they want to work with us, means that they're going to be able to facilitate some of this work. They're going to be able to make things easier on us and that we're valued.
- Tim Lehnen:
- Yeah. Now that has made me take a little step back, after our blog post series and say, "Okay, we need to seriously think about this again." Let me give a sort of comparison though, of the things that we're considering right now. Like I said, we were really bullish on GitLab earlier in this evaluation process. It was kind of our leading contender for quite a while.We were going into full prototyping, again with ML Hess's help in the community. We were trying to get a full dev site integration and get as far as we could with that and see what we would run into. For example, because GitLab is this really fast moving organization, going very, very quickly, officially they support MySQL as the database backend of their tool.However half way through our prototype, they pushed an update and their latest update broke about four different features and new [Catar 00:34:22] installation that we were in the middle of, of prototyping with. They're one of these organizations that's in a position where, sometimes they're moving a little too fast. We've also had issues with APIs that we needed, turned out they were there but they were undocumented or some of them were missing.We had some concerns about project stability and product maturity, just because of the rate of change at which they move. We were balancing that against, you know, they're really committed. They want Drupal to be on their platform. It's been an interesting series of conversations and it's actually still kind of an ongoing conversation, at this point.
- Matt Kleve:
- In your blog post, you were high on Bitbucket, it sounded like, which is the third option from Atlassian. They're the folks that make JIRA and I know that Bitbucket is a similar project to all of these three and it has really tight JIRA integration, which you use JIRA for your ticketing side of things. If your code integrates with that, it works great and that's all I know about Bitbucket. Why is that good?
- Tim Lehnen:
- That's a really interesting question. Kind of like I said at the beginning, we weren't considering Bitbucket from the get go. Again, we have the same perception that I think a lot of people who have worked in development in a corporate environment do of Atlassian products, which is there's this suit and it's a different tool for every element that you want to use. It can be a pain and it's Java based and sometimes it's slow.We have these preconceived notions about what it is. Then we were at Midwest Drupal Summit in August and we were having this heart to heart conversation. By we, I mean myself, Neil, Michael [Hess 00:36:16], Angie, some of the Core folks, like XJM. Just sitting around a table, Peter [Bolanin 00:36:21]. Saying, "Okay, we've stood up this GitLab prototype and it's like 90% but it's not 100% and there's some hard blockers. It can't handle the duplication of Git repositories on the infrastructure side.We'd have 200 terabytes just to cover the Core issues, if there was one fork for each Core issue." We had all these things. It's like, "It almost works but it doesn't quite. What the hell are we going to do? We don't want to come back to the community and say, "Hey, we've done another 18 months and we just have nothing to show for it." Sitting around the table, we said, "Well, there is another tool out there. Bitbucket's out there. People use it.""We've used it internally for a few things, like a make file repository basically, private make file rebuild for [D dot O 00:37:15] builds. Why don't we try and stand that up and take a look at it?" It turned out, that they've done quite a bit with it. It's a pretty remarkable tool, in terms of being modular, in ways that these other tools aren't. Having a kind of API first architecture.Their UI just simply consumes an API that's available to everything else. It's very, very ... Well, the word easy is not right but it's very possible to use API hooks to connect into any part of that system. The code review tools are great. The inline editing tools are great. I would say they're just as good as GitLab's and it has some neat features when it comes to merging pull requests.Like automatic re-basing of a branch against its parent, so that it doesn't go stale. You can kind of configure interesting options like that. It has some things that we could integrate it, without having to just pick up and throw something away, without any compromises and without any, "Oh gosh, we wish that wasn't broken but we're going to do it anyway." That's where we wound up at the time of that blog post at the end of last year.
- Matt Kleve:
- Yeah. API first pluggable, those sound like good things for somebody who wants to extend something.
- Mike Herchel:
- Did you end up having any conversations with the Bitbucket team, similarly to what you do with GitHub and GitLab?
- Tim Lehnen:
- We had some conversations. They weren't quite as in depth. Michael Hess is one of the volunteers we work closely with, has kept come connections over there. We sort of telephone through him a little bit but also, we had less questions.It was so well documented and we were just able to stand with the prototype so quickly that, there was less of us having to go and sway, "Hey, can you change your product to fit our needs or allow licensing kind of things." Neil, do you have anything to add to that?
- Neil Drumm:
- Yeah. There was someone from Atlassian. I forget what their title was. They showed up in the comment section of the blog post too. I think they were a developer evangelist or something like that.
- Tim Lehnen:
- Yeah or even a product director or something like that. They've certainly reached out as well. They're I would say, also hungry to support a community like ours. That puts us in an interesting place, right? I'm sure before we started this conversation, we had the blog post to go on and it was like, "Well we're probably going to move to Bitbucket, as this interim solution but maybe we'll look at GitLab again, sometime in the future," was kind of the conversation. Now, we're not ready to say, "We've changed our minds," or anything. Now we're having a few more conversations and we might take another look at some things, depending on what happens.
- Matt Kleve:
- If I were Monty Hall and not a statistics professor and I told you, "Now, you have four doors to choose from," at some point, you have to choose the door to be able to move forward, right?
- Tim Lehnen:
- This is true. This is absolutely true.
- Matt Kleve:
- How do we make sure we can make that decisions safely?
- Tim Lehnen:
- Right now, we know that if we moved forward with Bitbucket, it would be a safe decision. This is the rock at the hard place. GitLab has more like I said, more of a sizzle, more of a curb appeal, more that kind of cultural [inaudible 00:40:46] of the developer community right now and it's the open source option of the two, which is not a small thing. People care about being able to use an open source option. It's also the one that doesn't quite work yet but that they're committed to making work. From my point of view, by Nashville, we should be absolutely certain and where we are now is, Bitbucket's the default but we want to give GitLab the chance to prove that maybe they really can be the right choice for us.
- Matt Kleve:
- Is money a part of the equation on any of these?
- Tim Lehnen:
- Not in a significant way, in terms of the decision making process. Some of the teams have said, that they'd be able to offer developer support or offer financial support for us, in terms of sponsoring the integration process and things like that. Just because of the nature of what it is, it's really more about hitting those requirements.
- Matt Kleve:
- Stay tuned huh? We'll hear more by Nashville?
- Tim Lehnen:
- Yeah. We'll hear more by Nashville or sooner, for sure.
- Mike Herchel:
- What is the timeline for moving over everything to the new workflow?
- Tim Lehnen:
- That's going to depend on this sort of unexpected Hail Mary, that's going on. I would say, one way or another, we're going to get up to our elbows by the Nashville timeframe. There's already work in progress and that can be done right now. Neil, do you want to talk about some of those things?
- Neil Drumm:
- Yeah. Really, the way to actually test these things, is to build integrations. We're kind of doing some of the work as like a version zero tested out sort of thing. We'll be able to salvage some code out of that and make it production ready, once we're at that point. Some of the first things we'll see is, just replacing the Git servers themselves, what's hosting the repositories. They will be pretty transparent. If we do it right, you won't notice that what's hosting the Git repositories has changed and that certainly can start doing things like, having the ... Turning on the code browsing. Some text highlighting, replacing cgit.drupal.org and turning on pull requests, that's the supplement issues.To get to that point, there are some things that we need to make sure Drupal.org is thinking about the same way, as these tools. Like will all of our user names fit into whatever restrictions the tools have? Do we need to ask some people to change their Git username? How's password authentication going to work? We don't really like people using password authentication over SSH keys but it turns out, passwords are easy and people use them. Making sure that that can work in some way or we can ... If we have to turn that off, it's as gently as possible. A lot of making sure things keep working.
- Matt Kleve:
- Exiting stuff. I know that as a developer, sometimes I am definitely adverse to the current workflow and I'm looking forward to anything that can be done, to move this along in the right direction.
- Tim Lehnen:
- I think another way to put it is, there's still going to be a human element and an educational element to onboarding new people into the community, even with new more familiar tools.
- Neil Drumm:
- Yeah. Computers aren't writing the software for us, yet.
- Matt Kleve:
- Thanks for coming on and telling us all about it guys.
- Tim Lehnen:
- Yeah. Really happy to be here. Thanks for the invite and thanks again to Lullabot for being a supporting partner and funding the work that Neil and I and the rest of our team do. It's the only thing that's going to make this possible.
- Mike Herchel:
- Yeah. Right back at you. Thanks to the Drupal Association for keeping the infrastructure running and keeping Drupal humming as a community. It takes a lot of work. You guys are under a lot of fire a lot, so I really, really appreciate it.