How to manage/maintain multiple decoupled repositories

Background

With the Composer Initiative we have decoupled all of the plugins and themes from the core Mautic repositories, in addition to the /app directory now being a separate repo of mautic/core-lib.

At the moment the repos only contain a copy of the code at a point in time - they do not have the git history which we need to retain.

We also need to find a way so that we can break out all of these into their own repo, but keep all of them updated (one way only from mautic/mautic → core-lib/plugin/theme repo) so that when these are updated when a PR is merged, the code is automatically synchronised (maybe with a GitHub Action?) to the plugin/theme repo.

Some related things we need to consider:

• Governance for all the repositories

• All plugins will need to use the same branching strategy so that they can be aligned with PR’s against different versions that may be in flight at the same time

• The build will need to make sure that the plugin repos are tagged as well as the core repo

Research

Splitting the repos while maintaining the GitHub commit history

• There is a built in, GitHub recommended way to do this: https://docs.github.com/en/github/getting-started-with-github/splitting-a-subfolder-out-into-a-new-repository - we should follow this process for the plugins and ensure that we end up with the correct branching strategy

• Try this with one of the plugins and see how this works - then maybe roll that out more widely

Maintaining the sync between mautic/mautic and the core-lib, plugins and theme repos

• Once we have split out the app directory, plugins and themes we then need to figure out how to keep the app directory, plugin and theme folders in sync

• There are a few tools out there that we could potentially consider:

  • https://github.com/dflydev/git-subsplit

    • Not sure if this is what we are after - it mentions a subtree split. Need to read more.

    • Allows you to do the subsplit and maintain thereafter

    • One-way read only sync

    • Allows you to specify the heads/tags to be updated if you don’t want to auto-detect

    • Seems to be abandonware - 4+ years since last update and multiple open issues/PR’s

  • https://github.com/ryanwinchester/subsplit-service

    • Allows you to do the subsplit and maintain thereafter

    • Composer-based install

    • Also supports webhooks if we need to use them

    • Seems to be abandonware - 5+ years since the last update

  • https://github.com/splitsh/lite Symfony uses this

  • Github Actions

    • https://github.com/drud/action-cross-commit this may work as a GitHub Action - we would need to work out what directories are being updated in the merged PR and push those directories to the correct repos using some kind of dynamic folder magic 🧙‍♀️

    • https://github.com/s0/git-publish-subdir-action may also be an option (we would have to set up each folder and match it with the correct repo) - maybe run an ‘if files have changed’ check beforehand to only push changes if there are changes?

Governance of decoupled repos

• We need to make sure that all of the plugins and themes are appropriately licensed and that a readme & license file plus all the security and community health files are present

• We can set up a centralised .github repo that allows us to orchestrate all of our organisation repos centrally

  • We would need to reach out to the maintainers of repos ahead of time to make sure that they are aware of any changes (eg changes to branch names) that we may require / suggest

  • We can apply rules to groups of repos, to all repos, or manage them individually

• There is a tool that we can use to manage our repos centrally: https://organizer.gitconsensus.com

  • Open source project https://github.com/gitconsensus/GithubOrganizer

  • Has a Github App https://github.com/apps/organization-manager

Useful resources

• https://m.dotdev.co/splitting-out-a-directory-to-its-own-git-repository-3e5dc410871b

• https://www.atlassian.com/blog/git/tear-apart-repository-git-way

• https://www.atlassian.com/git/tutorials/big-repositories

• https://www.atlassian.com/git/tutorials/git-subtree (not planning to use this but read through it to discount subtrees)

• https://blog.developer.atlassian.com/the-power-of-git-subtree (As above)

Options considered

Action items

Outcome