How to manage/maintain multiple decoupled repositories

Owned by Ruth Cheesley, created
Last updated: Apr 05, 2021
4 min read

Status

In progress

Impact

High

Driver

@Ruth Cheesley 

Approver

 

Contributors

@Nick Veenhof (Deactivated) @Dennis Ameling @Alan Hartless @Jan Linhart 

Informed

@Norman Pracht (Unlicensed) @Mohit Aghera @Jozsef Keller

Due date

Apr 16, 2021

Outcome

Background

With the Composer Initiative we have decoupled all of the plugins and themes from the core Mautic repositories, in addition to the /app directory now being a separate repo of mautic/core-lib.

At the moment the repos only contain a copy of the code at a point in time - they do not have the git history which we need to retain.

We also need to find a way so that we can break out all of these into their own repo, but keep all of them updated (one way only from mautic/mautic → core-lib/plugin/theme repo) so that when these are updated when a PR is merged, the code is automatically synchronised (maybe with a GitHub Action?) to the plugin/theme repo.

Some related things we need to consider:

  • Governance for all the repositories

  • All plugins will need to use the same branching strategy so that they can be aligned with PR’s against different versions that may be in flight at the same time

  • The build will need to make sure that the plugin repos are tagged as well as the core repo

This MUST be in place before the Mautic 4 Beta Sprint (23-25 April)

Research

Splitting the repos while maintaining the GitHub commit history (this does not take into account maintaining the sync)

Maintaining the sync between mautic/mautic and the core-lib, plugins and theme repos

  • Once we have split out the app directory, plugins and themes we then need to figure out how to keep the app directory, plugin and theme folders in sync

  • If we physically remove the files from the main repo, we then need to figure out how to manage PR’s for those repos:

  • There are a few tools out there that we could potentially consider:

    • https://github.com/dflydev/git-subsplit [Recommend to not consider due to no recent updates]

      • Not sure if this is what we are after - it mentions a subtree split. Need to read more.

      • Allows you to do the subsplit and maintain thereafter

      • One-way read only sync

      • Allows you to specify the heads/tags to be updated if you don’t want to auto-detect

      • Seems to be abandonware - 4+ years since last update and multiple open issues/PR’s

    • https://github.com/ryanwinchester/subsplit-service [Recommend to not consider due to no recent updates]

      • Allows you to do the subsplit and maintain thereafter

      • Composer-based install

      • Also supports webhooks if we need to use them

      • Seems to be abandonware - 5+ years since the last update

    • https://github.com/jderusse/docker-gitsplit was mentioned by Tobias Nyholm (maintainer of subtreesplit.com/symfony core team)

      • Allows multiple branches

      • Allows you to use Github Actions

      • Splits are all maintained in a workflow file

      • Has to clone the entire repo - may have performance issues?

      • Triggered on push to main branch and on release, based on example - so should meet our needs

    • https://github.com/thephpleague/flysystem/blob/2.x/.github/workflows/publish-subsplits.yml was mentioned by Tobias Nyholm (maintainer of subtreesplit.com/symfony core team)

      • Github Action used by Flysystem to push subsplits automatically - could be something we could look into but may need some assistance from @Dennis Ameling to understand how we might use it on PR merge / release and how to bring it into our current workflows.

      • This supports actions running on multiple branches

      • We would need to set up the splits first, I think - once they are set up, then the action would run and push the changes when triggered.

    • https://github.com/splitsh/lite Symfony uses this tool but the docs are not all that great!

      • There is a hosted ‘service’ which uses this tool - created by one of the core Symfony team at https://www.subtreesplit.com/

        • Played around with it and was able to split the app/ folder from https://github.com/RCheesley/a-mautic-playground  to https://github.com/RCheesley/core-lib-playground  very easily.

        • It only pushes the default branch at the moment but the author may be able to dig into making it push other branches

        • It has the option to comment on PR’s that are made on the mirror repos in order to push people back to the main repo - nice feature that would hopefully save us a bit of a headache (I don’t think that you can disable PR’s on a repo)

        • Key point: the underlying technology seems to work!

           

    • Github Actions

      • https://github.com/drud/action-cross-commit this may work as a GitHub Action - we would need to work out what directories are being updated in the merged PR and push those directories to the correct repos using some kind of dynamic folder magic

      • https://github.com/s0/git-publish-subdir-action may also be an option (we would have to set up each folder and match it with the correct repo) - maybe run an ‘if files have changed’ check beforehand to only push changes if there are changes?

Governance of decoupled repos

  • We need to make sure that all of the plugins and themes are appropriately licensed and that a readme & license file plus all the security and community health files are present

  • We can set up a centralised .github repo that allows us to orchestrate all of our organisation repos centrally

    • We would need to reach out to the maintainers of repos ahead of time to make sure that they are aware of any changes (eg changes to branch names) that we may require / suggest

    • We can apply rules to groups of repos, to all repos, or manage them individually

  • There is a tool that we can use to manage our repos centrally: https://organizer.gitconsensus.com

Useful resources

Options considered

 

Option 1:

Option 2:

 

Option 1:

Option 2:

Description

 

 

Pros and cons

Estimated cost

Large

Medium

Action items

Outcome