Posted 2 years 7 months 3 weeks ago.

Did Config Management Initiative in Drupal do the exact opposite of what it intended to do? Is the lack of a documented Config Management Strategy creating a barrier for teams to adopt Drupal? Can team be successful with using Drupal without investing heavily to figure out their own Config Management Recipe?

Let us explore with a simulation of what would be an experience of an engineer trying to implement a basic config management strategy on their project, and how it ends up with a chicken-egg problem, ultimately leading to a vortex of config management modules.

Configuration Management in Drupal has been ever changing. Drupal 7 had features as a go-to standard to isolate configuration from database and to version control it. Drupal Configuration Management Initiative resulted in a configuration management interface in core, which was supposed to be the holy grail of all config management problems in Drupal 7. Which it failed to be. Features, which was supposed to be extinct by now due to CME, is now becoming popular again, due to the limitations in Core CMI. And there is a plethora of exploding set of Config Management modules in Drupal 8/9, that make it challenging to decide “What is a good config management strategy for my Drupal project?”

There is no one-size-fits-all documented way of configuration management in Drupal. Which adds to its learning curve. Sticking to core CMI includes challenges of having to depend on a database copy to setup new environments, or overwriting the UUID of the site in some ways, to get the configuration to be imported in a newly setup environment. While this is addressed in Drupal 8.6 to some level, but it still has a chicken-egg problem that we will discuss below.  Further, you can’t pull to your environment config changes that your team member just pushed in, unless you export your configuration before you pull your team mate’s changes.

Trying to simulate here the journey of what would be that of a developer or a tech lead, trying to accomplish basic configuration management on their project, with a very basic objective, looks like.

Objective:

  1. Be able to setup an environment from scratch, with all required configuration, without any dependency of an old import of a database dumps flying around in Slack, but just using the codebase
  2. Ensure changes to configuration are version controlled
  3. Allow incremental changes to configuration to be added, without having to rely on complete dump of configuration, that often have unrequired changes from local environment. 
  4. Allow team members to work in parallel, often on same configuration, allowing them to merge their incremental changes

Initialization of the project:

  • Clone from a composer-based template and not in the legacy way
  • Stack to use : Lando or DDev. Both look great. I picked Lando. But DDev should be equally good. 
  • Do a composer install to get all dependencies
  • Add Config Ignore module, and drush as well (drush may not be required if you used the first template above, will be needed if you used the second one)
    composer require drupal/config_ignore
    composer require drush/drush
  • Proceed with composer install
  • Power up your local server - lando start - if you are using Lando
  • Install the site using drush si command
  • And your local site should be up and running now!
    (If you are using lando, you can use check the url via lando info)
    (If you are using lando, and you see a 404 instead of your Drupal homepage, run these 3 magic commands)
  • Change config sync directory in settings.php to below
    $settings["config_sync_directory"] = "../config/sync";
  • Ensure you have a proper .gitignore file. If you used the first template above, create one manually by copying this from template 2. 
  • Enable config ignore module
  • Add the following to config ignore settings via UI (We will discuss later why we are doing this)
    shortcut.*
    image 8
  • Remove the remnant sites/default/files/config_* directory you may have from the initial setup
  • Make any initial configuration changes that you see required, content types, or other configuration. 
  • Do an export of initial configuration via drush cex. This will populate your initial site config into config/sync directory
  • Note the UUID of the installation -
    drush config-get "system.site" uuid
    Communicate this to your team, or add it to the readmefile. This is required for the rest of the team to setup their local environment later. 
  • Check in your code to the repo that the team would use. 

Setting up local environment by other team members

  • Clone from the repo
  • If using lando - lando init (if lando.yml was checked into the repo during earlier project setup, this is not required. Checking in that file might be required, only if you are trying to freeze specific versions of php, or some custom tooling to lando). Use options - Current Working Directory, Drupal 9, web, any-name for lando init
  • Run composer install
  • Up your servers (lando up if you are using lando)
  • Now, if you run a site install, and try to import the config from the config sync folder you have on your site, the site would not know the existence of config/sync directory. 
  • But, there is a deadlock. settings.php would not exist until we ran install. But if we run install, it would create a site with a different UUID and not honoring the existing configuration. We could run drush si --existing-config , but how would it know about the config/sync folder as we haven’t updated it in the settings.php which doesn’t exist yet.
    image 10
  • We stumbled into the popular chicken-egg problem with Drupal CMI that is discussed here and here in detail.
  • Let us work around this, by doing a regular installation of the site (without using existing config, then manually change the UUID of the site, then update config sync directory in settings.php, and then doing a drush CIM. 
  • Run a simple drush si (for site install)
  • Change the UUID of the site to the UUID mentioned in initial setup
    drush config-set "system.site" uuid  5a5ccb99-8411-42e8-a8fe-3c4b2d827910
  • Now you think you could run configuration import (drush cim). That still won’t work because you there are some configuration related to shortcut module etc, that would throw an error like this
    image 12
  • We added this to config ignore in the project setup, but that config is not imported yet, so it doesn’t know that it should ignore that shortcut config. 
  • So let us enable config ignore manually, and add shortcut.* to config ignore, and try the import again.
  • Now run a config import (drush cmi), and you have your local instance ready, which is a clone of the instance that was setup initially, without any transfer of database dumps, across slack.

So far, so good!! It wasn't straight forward. Had to plug in config ignore to get the config sync to work. Had to work around the chicken-egg problem for getting the site install to work with existing configuration. But we got there!

Workflow for feature development story

  • Team member (say Person A, which is you) working on the ticket ensures they have the latest of the config/sync folder before starting. 
  • Say Person A had made some configuration changes now - say added field B to a content type. (Note there could be some additional config added by a different team member Person B to the repo by now). 
  • Person A now wants to raise a pull request. Before doing so, he checks if he has all the latest and greatest from the repo.
    image 13
  • Person A realises there is a field C added by person B and pushed to the repo
  • Now if Person A does a config import to get the field C, he will lose the field B that he added in his local machine.
    image 14
  • So Person A (you) would export your configuration, and then merge the changes from the master into your feature branch. But guess what, if both person A and person B are working on same config files, like in this example each one added a different field to an existing content type, then the automerge would actually replace one’s changes with the other. This is detailed here. A pull request raised would look like this.
    image 15
  • Enter Config Merge, which requires Config Sync, which requires a dozen other modules like Config Distro, Config Filter, Configuration Merge, Configuration Normalizer, Configuration Provider, 8.x-2.x branch, Configuration Snapshot and Configuration Update Manager..

    Making you enter the black hole of config management modules. The exact opposite of what CMI tried to solve in the first place!

Looks like what was a very robust configuration management system that Drupal had with features in Drupal 7, has expanded into a large system of scattered and often abandoned maintained modules, to accomplish a simple objective of having configuration in code, allowing teammates to work in parallel. CMI looks to have accomplished the exact opposite of what it started to do in the first place.

This creates a barrier of entry for organizations to start with Drupal, as there is no documented way of handling configuration management that allows a few team members to work in parallel, unless they are working on completely unrelated items not touching configuration that other team members are working on.

Unless a team have their own recipe figured out, with a combination of techniques and combination of modules that work for them and their customers, they can hardly expect any success with having a sustainable configuration management strategy.

Why is it so hard to have one prescribed configuration management strategy that works for most (if not all) of the Drupal projects, allowing parallel collaboration between team members?

Most teams have their recipes figured out with their own combination of tools and techniques using some part of core CMI, and Features module (yeah, Features again, which comes with its own set of caveats and challenges, some of which you can see in this video), and a set of Config Ignore, Confit Split, and Config Merge, Config Sync, Config Normalizer and other modules.

But there is definitely a need to remove the barrier to entry, allowing more teams to adopt Drupal, without having to invest heavily upfront in discovering a secret sauce combination of modules and techniques to get basic configuration working for a simple team that just a few people working in parallel on configuration.

There are multiple configuration management strategies that I have been exposed to, from the teams that I have worked with in the past few years. Not just calling out the problem. And I acknowledge the tonnes of work that went into CMI from many volunteers. Although, as with anything, we should look at things from the value it adds, and not just the intent and effort. 

As with any opensource community, you have as much access to solve the problem, as you have access to talk about it. As my part to solving this problem, I would try to analyze them against how they fit into the problem statement defined in the objective above, and try to document one such simple strategy either here on somewhere on drupal.org.

Meanwhile, may be, there is already a popular one that I failed to notice. May be, this is not even a problem, and I am just missing some pieces of the puzzle that are not very obvious.

If you know of one documented config management strategy that qualifies the below criteria, drop a link in the comments.

  • Spinning up local instances from repo, without depending on a database dump
  • Version control config
  • Allow 3-way merging of config from local and upstream
  • Is documented with a recipe of modules and techniques to use.

Comments

Wed, 08/18/2021 - 06:17

tanay

As i did mention in the note:

As my part to solving this problem, I would try to analyze them against how they fit into the problem statement defined in the objective above, and try to document one such simple strategy either here on somewhere on drupal.org.

Thu, 08/19/2021 - 05:18

Anonymous

I think you are missing a simple trick because it is so much easier than what you do. So most of this blog post is not helpful at all.

The first half is unnecessary because there is no chicken-egg problem, if you do the simple thing:
echo '$settings["config_sync_directory"] = "../config/sync";' >> web/sites/default/default.settings.php
Then you can do drush si --existing-config without config ignore or dealing with uuids etc, the whole first half is solved with one line.

The second half is easy if you follow the best practice I learned at devdays in 2017 https://www.slideshare.net/nuvoleweb/advanced-configuration-management-… (slides 33 and following). You are right about merging with git, but in my experience the examples where it doesn't work are contrived and working in a team it is easy to just communicate about working on the same content type. So in practice this has never been a problem in many years in both big and small teams for me.

And finally you are wrong about config merge. It doesn't depend on anything, if you want it to be useful you have to enable the sub-module which depends on config filter which in turn doesn't depend on anything else. So no black hole of config modules even if you want to use config merge.

Thu, 08/19/2021 - 14:38

tanay

Thank you for your note

because there is no chicken-egg problem, if you do the simple thing:

A large part of my note is around the fact that nuggets like this are no where documented, even in the d.o issue queue for that issue here and here.Without which, of course, it is a chicken egg problem.

The second half is easy if you follow the best practice I learned at devdays in 2017

That is exactly my point. Unless a team spends the time and effort to dig into a 5 year old presentation behind a paywall/popup, after being pointed in that direction, there is no one way to discover it on any CMI documentation like this one.

in my experience the examples where it doesn't work are contrived

This is a very basic example of two developers trying to add a different field to an existing content type. A very common scenario often seen in applications that revolve around a single large content type, with a complex workflow.

And finally you are wrong about config merge. It doesn't depend on anything, So no black hole of config modules even if you want to use config merge

It doesn't. But it doesn't do anything by itself for the merge, unless you have config_sync, which inturn depends on all the modules that i mentioned.

Further, even with this vortex of modules, three way sync works only for config provided by modules, and not the config from sync directory via Drupal CMI.

Add new comment

About text formats
Submitted by tanay on Wed, 08/18/2021 - 02:35