Always be shippable

Drupal will soon be 15 years old, and 5 of that will be spent on building Drupal 8 -- a third of Drupal's life. We started work on Drupal early in 2011 and targeted December 1, 2012 as the original code freeze date. Now almost three years later, we still haven't released Drupal 8. While we are close to the release of Drupal 8, I'm sure many many of you are wondering why it took 3 years to stabilize. It is not like we didn't work hard or that we aren't smart people. Quite the contrary, the Drupal community has some of the most dedicated, hardest working and smartest people I know. Many spent evenings and weekends pushing to get Drupal 8 across the finish line. No one individual or group is to blame for the delay -- except maybe me as the project lead for not having learned fast enough from previous release cycles.

Trunk-based development

The past 15 years we used "trunk-based development"; we built new features in incremental steps, maintained in a single branch called "trunk". We'd receive the feature's first patch, commit it to our main branch, put it behind us, and move on to the next patch. Trunk-based development requires a lot of discipline and as a community we have mostly mastered this style of development. We invested heavily in our testing infrastructure and established a lot of processes. For all patches, we had various people reviewing the work to make sure it was solid. We also had different people figure out and review all the required follow-up work to complete the feature. The next steps are carefully planned and laid out for everyone to see in what we call "meta" issues. The idea of splitting one large feature into smaller tasks is not a bad idea; it helps to make things manageable for everyone involved.

Given all this rigor, how do you explain the delays then? The problem is that once these features and plans meet reality, they fall apart. Some features such as Drupal 8's configuration management system had to be rewritten multiple times based on our experience using it despite passing a rigorous review process. Other features such as our work on URL routing, entity base fields and Twig templating required much more follow-up work compared to what was initially estimated. It turns out that breaking up a large task into smaller ones requires a lot of knowledge and vision. It's often impossible to estimate the total impact of a larger feature on other subsystems, overall performance, etc. In other cases, the people working on the feature lacked time or interest to do the follow-up work, leaving it to others to complete. We should realize is that this is how things work in a complex world and not something we are likely to change.

The real problem is the fact that our main branch isn't kept in a shippable state. A lot of patches get committed that require follow-up work, and until that follow-up work is completed, we can't release a new version of Drupal. We can only release as fast as the slowest feature, and this is the key reason why the Drupal 8 release is delayed by years.

Shippable trunk based development
Trunk-based development; all development is done on a single main branch and as a result we can only release as fast as the slowest feature.

We need a better way of working -- one that conforms to the realities of the world we live in -- and we need to start using it the day Drupal 8.0.0 is released. Instead of ignoring reality and killing ourselves trying to meet unrealistic release goals, we need to change the process.

Feature branch workflow

The most important thing we have to do is keep our main branch in a shippable state. In an ideal world, each commit or merge into the main branch gives birth to a release candidate — it should be safe to release after each commit. This means we have to stop committing patches that put our main branch in an unshippable state.

While this can be achieved using a trunk-based workflow, a newer and better workflow called "feature branch workflows" has become popular. The idea is that (1) each new feature is developed in its own branch instead of the main branch and that (2) the main branch only contains shippable code.

Keeping the main branch shippable at all times enables us to do frequent date-based releases. If a specific feature takes too long, development can continue in the feature branch, and we can release without it. Or when we are uncertain about a feature's robustness or performance, rather than delaying the release, it will simply have to wait until the next release. The maintainers decide to merge in a feature branch based on objective and subjective criteria. Objectively, the test suite must pass, the git history must be clean, etc. Subjectively, the feature must deliver value to the users while maintaining desirable characteristics like consistency (code, API, UX), high performance, etc.

Shippable feature branching
Feature branching; each feature is developed in a dedicated branch. A feature branch is only merged into the main branch when it is "shippable". We no longer have to wait for the slowest feature before we can create a new release.

Date-based releases are widely adopted in the Open Source community (Ubuntu, OpenStack, Android) and are healthy for Open Source projects; they reduce the time it takes for a given feature to become available to the public. This encourages contribution and is in line with the "release early, release often" mantra. We agreed on the benefits and committed to date-based releases following 8.0.0, so this simply aligns the tooling to make it happen.

Feature branch workflows have challenges. Reviewing a feature branch late in its development cycle can be challenging. There is a lot of change and discussion already incorporated. When a feature does finally integrate into main, a lot of change hits all at once. This can be psychologically uncomfortable. In addition, this can be disruptive to the other feature branches in progress. There is no way to avoid this disruption - someone has to integrate first. Release managers minimize the disruption by prioritizing high priority or low disruption feature branches over others.

Here is a workflow that could give us the best of both worlds. We create a feature branch for each major feature and only core committers can commit to feature branches. A team working on a feature would work in a sandbox or submit patches like we do today. Instead of committing patches to the main branch, core committers would commit patches to the corresponding feature branch. This ensures that we maintain our code review process with smaller changes that might not be shippable in isolation. Once we believe a feature branch to be in a shippable state, and it has received sufficient testing, we merge the feature branch into the main branch. A merge like this wouldn't require detailed code review.

Feature branches are also not the silver bullet to all problems we encountered with the Drupal 8 release cycle. We should keep looking for improvements and build them into our workflows to make life easier for ourselves and those we are implementing Drupal for. More on those in future posts.


Jose (not verified):

Feature branches sound good. However it doesn't matter which strategy we use if at the end we are not serious about deadlines, code freezes, etc..

Really, 5 years is a lot of time, and all this started when we missed the first code freeze. The thing is once you start missing the small deadlines, you are going to miss by large the big one. And still worse, people loses faith on deadlines, they don't mean anything anymore ... And once you lose faith on the product being released anytime soon then some people may decide -- well, I did -- it is better to devote their energy to the current working version instead, say Drupal 7, because that is the one that is going to pay your / your company's bills next year.

So feature branches sound nice, at least I guess that will mean no more @todo's in the main branch anymore. :-) But a release date === serious deadlines === "if it's not ready we move on without it" would sound better. We do have modules for filling the gaps if something is not into core yet.

Btw, feature branches sound a lot like "old Drupal patches" that never got committed till the whole thing was -really- ready, call me nostalgic ...

Paul Lomax (not verified):

Agree re scope creep - and also endless refactoring. In the 'real world' you'd have product managers who can decide what's in and what's out and say 'no' as required. They shouldn't also be developers, as developers like to write code too much. :)

dawehner (not verified):

In general Its a great idea, which solves a lot of the problem of classical release management.

One crucial part of the continues deployment idea, which is kinda the underlying concept, is that things are actually used often and just out there in the wild. This is kinda hard in Drupal, 6 months are a short time. You know, working on features is nice, but in reality the hard bugs aren't discovered until someone actually uses it. We need to not forget about it, so we would need even better test coverage so finding architectural/underlying issues, so people can actually trust us in our rapid releases

webchick (not verified):

One idea to mitigate that which came from Alex Pott during today's discussions was to ask the question initially: "Can this happen in a contributed module first?". Because if it can, we have a way for Drupal 8 sites to try in production, have a way to monitor how much demand there is for the feature (based on usage stats), have a way to monitor technical debt we'd be inheriting (via the issue queue), and have an idea of the activity/engagement of the team behind it (based on commit history). I don't think we would necessarily never commit a feature that didn't happen in contrib first, but for many features this can be a good option. It also forces developers to think how to introduce it to core in the least invasive way possible.

Jose (not verified):

Great idea.

While we are doing one side of it, which is "Getting into core things that have worked in the past as contrib modules", we are completely missing the other side of it, "Do try it in contrib first, come back if it works nicely and people really use it.". And as a result we are inventing APIs and UIs on the fly without really knowing how they are going to be used in real life.

Many of those future feature branches could very well start being a 9.x branch of current D8 contrib modules and if they really need core API changes, this will be the case for it.

I think this should be something to seriously enforce for D9.

cosmicdreams (not verified):

It should be noted that this is how Wordpress handles major feature additions. Features exist as a addon even if the work was crated and led by internal Wordpress developers. If the feature is well received and if it has lived as a fully 1.0, finished, complete feature for at least one minor release of core then it can be consider for inclusion.

I think this idea is a decent one as it limits the field of features that the core committers need to be focused on. Not every feature can land in core. It has to be something that can live on it's own. Something that provides enough value that it attracts a following. And even in that case, it could be rejected because of various concerns.

And if Drupal 8's architecture does not allow for this kind of module-first development, we should look at where the impediments are and fix the platform.

In the perfect world I'm imagining, platform developer enables application development. Application development informs platform development's priorities. Which establishes a world where blue-sky / crazy fringe development happens in contributed modules and platform development focuses on how best it can serve applications.

Kristofer Tengström (not verified):

Yeah, feature branches are pretty much the standard these days. All feature branches should pull from the main branch continually during development to make merge conflicts as small and contained as possible, thus minimizing the disruption from committing new features.

Kristoffer Wiklund (not verified):

The idea of a feature branches sounds very nice. And when listening to the keynote it felt like this is the solution to get new releases faster and more functionality. But when I was thinking of it, it might not always be possible.

Let’s look at the Drupal 8 improvements. We have
- Twig
- Symphony

Let’s play with the thought that OOP and Twig will be developed in different feature branches. Any commit to trunk will need some heavy rewrite for OOP. And if Twig was committed to trunk lots of the Twig work needs to be redone in the OOP branch. And when OOP is pushed to trunk, every other feature branch will be invalid (and more or less needs a totally rewrite).

So feature branches works nice when the different features is quite separated and the do not depends on each other. We have the similar problem with Drupal 7 site development with features. Let's say we have a D7 site with two teams working on it. Team A is refactoring the code from template-files to a panels/ds solution. And Team B is redoing the front page layout.

If Team A commits its code first, team B needs to redo all of their code (the design idea can be kept).
If Team B commits its code first, team A will need to redo the work that team B has done.

So the best solution here is to have one team make its feature first. And when it's done the other feature could be developed. So it's not possible to have parallel because the first to commit to trunk will invalid most of the work for the other. So if Drupal 8 would be developed in feature branches. The OOP branch would block coding for all other branch.

One other problem with feature branches is the contrib module compatibility. Let's say that the D8 features would have been committed in different releases. OOP will make all modules invalid, CMI will make most of the modules invalid, Twig will force most module to update, etc. It's increasing the burden on contrib modules.

moshe weitzman (not verified):

Right. If a feature can live in a standalone module then we barely need to discuss it here. It's a pretty trivial problem to usher it into core. The cross-cutting features are way more challenging. And these are the ones that delay releases, not a standalone module that can be removed mainline at any time.

rszrama (not verified):

I think in the case of the items you list, it's fair to say that rearchitecture is not the normal use case for feature branches. But at least in a semantic versioning world, rearchitecting can be deferred for a major release while feature branches are still maintained for minor release improvements.

cosmicdreams (not verified):

That's the wrong scenerio to test. When Drupal 8 is out it will have OOP and Twig and all of these things. The question isn't could the "always be ship-able" approach support the addition of these features. The question is can we finish new feature additions in a timely manner so that when they are added to core, core can remain ship-able.


It's true that feature branching could lead to us working on fewer disruptive features in a parallel but I don't agree that the best solution is to work on disruptive features sequentially. While working on multiple disruptive features in parallel can be costly in terms of merges, it can still be faster than implementing the same features sequentially. The reason is that we have different people working on different feature branches. Implementing disruptive features 100% sequentially, would only make sense if there was no parallelism whatsoever. We need to find a healthy balance between between throughput and workload (time spent merging/updating branches).

Tom Metcalfe (not verified):

With feature branches I've found the issue is when the trunk moves on a considerable amount while your feature is left behind.

Something that could help would be a standard workflow for when an update is made to trunk, that change is then pushed out to the feature branches.

Gitflow (…) does it quite nicely, but I've never gotten on with it personally.

Would we need to create something like this along the lines of the test suite?

Kevin Kaland (not verified):

In terms of that, we just have to rebase feature branches often. Rebasing is a process that automates the following steps:

1) Check out the branch from which we want to rebase
2) Rewind our feature branch to the same point as from which we branched
3) Re-commit our commits against the new codebase and let us fix conflicts at any point in the process, then continue

This will show if there are any conflicts, and a combination of IDEs and code sniffing could tell us if we're using now-deprecated APIs.

Doing this many times throughout the life of a feature branch will keep the difficulty of merging to a minimum, and it will (hopefully) allow the branch maintainers — the ones with domain knowledge — to resolve the conflicts, rather than the trunk maintainers on the other end, all at once.

The downside to rebasing is that it changes history, and's Git doesn't allow this right now. I'm not sure how we'd solve that technically, but it's certainly possible. One way is to create a new feature branch and switch development to that (since then one isn't changing history that others may already have cloned).

The thing we should avoid is merging trunk INTO feature branches, since that results in a lot of white noise.

IMO, potentially using several feature branches per issue would be fine, similar to how we use several patches per issue now when the trunk changes and breaks patches.

David Corbacho (not verified):

Although the pace of releases of Chromium is not comparable to Drupal core, I found this article interesting "How Chromium works" https:[email protected]/in-march-2011-i-drafted-an-article-explain…

Quote from article:

No branches.

On many projects, it's common to branch the source code to work on major new features. The idea is that temporary destabilization from the new code won't affect other developers or users. Once the feature is complete, its branch is merged back into trunk, and there is usually a period of instability while integration issues are ironed out.

This wouldn't work in Chrome because we release every day. We can't tolerate huge chunks of new code suddenly showing up in trunk because it would have a high chance of taking down the canary or dev channels for an extended period. Also, the trunk of Chrome moves so fast that it isn't practical for developers to be isolated on a branch for very long. By the time they merged, trunk would look so different that integration would be difficult and error-prone.

Simon C (not verified):

I'm an outsider here, but I hope you'll allow me to say a few things.

My current project is being written using Joomla instead of Drupal. There were a whole load of factors that went into that decision, but ultimately a lot of them boil down to the slow progress of releasing Drupal 8.

A release cycle of five years between major versions simply isn't acceptable. The only other piece of software I can think of that waited that long between major versions is Internet Explorer, and we all know what the world thinks about IE6.

This blog post is a hopeful sign that things might change now after D8 is released. They need to.

Paul (not verified):

I endorse Simon's comments 100%. I got tired of waiting and switched to Joomla. It wasn't just the delay, it was the hassle of finding repeatedly that extensions were incompatible, unfinished or not yet available for the latest version of Drupal. As ecosystems go the whole thing seemed Kafkaesque.

Adrian Pintilie (not verified):

With feature branching, we can also drop the public versioning: eg. Drupal 8, Drupal 9 and focus on just promoting Drupal. Drupal will always be in a shippable state and that will matter to end-user. We can keep the version "internal" for devs/etc, but on d.o. and to the general public we should focus on promoting "Get Drupal".

Philip Van Hoof (not verified):

Consider taking it one step further and implementing gitflow as branch strategy. Here your feature branches branch off of a branch called develop and get merged into it. Your developers will never work on develop directly, but always in feature branches or in release branches. Then when you release you branch develop to a release/x.y.z branch, you do final changes for the release, and you merge those to both develop and master. Then you tag x.y.z on master. Then finally when you have a bug to fix in a old release, you checkout the tag x.y.z on master and you branch it as support/x.y.z+1 and you create a new release as x.y.z+1 with the bug fixed. And again you merge that to both develop. Now when you make a new feature release, the fix will be rippled down. So your release/x.y+1.0 will have the fix for x.y.z+1 too. Since you branch release/x.y+1.o off of develop (where you rippled support/x.y.z+1's fix to).

You actually have tools that do it all for you. They work like this: git flow release start x.y.z; git flow release finish x.y.z. And in modern continuous integration tools like TeamCity, Jenkins but also GitLab and gitg or gitk you get fancy graphs with bullets and arrows and stuff that flow from left and right into your master and develop branches. Just like the pictures on google-images when you search for gitflow. Management will love it. I mean ... pictures, with lines. And bullets for commits! You won't believe it ;).

You also have user interfaces for this, so you can press buttons like "Start feature", "Finish feature", "Start release", etc. So even the people who are totally new to version-control can do it.