How to manage/maintain multiple decoupled repositories

Background

With the Composer Initiative we have decoupled all of the plugins and themes from the core Mautic repositories, in addition to the /app directory now being a separate repo of mautic/core-lib.

At the moment the repos only contain a copy of the code at a point in time - they do not have the git history which we need to retain.

We also need to find a way so that we can break out all of these into their own repo, but keep all of them updated (one way only from mautic/mautic → core-lib/plugin/theme repo) so that when these are updated when a PR is merged, the code is automatically synchronised (maybe with a GitHub Action?) to the plugin/theme repo.

Some related things we need to consider:

• Governance for all the repositories

• All plugins will need to use the same branching strategy so that they can be aligned with PR’s against different versions that may be in flight at the same time

• The build will need to make sure that the plugin repos are tagged as well as the core repo

Research

Splitting the repos while maintaining the GitHub commit history (this does not take into account maintaining the sync)

• There is a built in, GitHub recommended way to do this: https://docs.github.com/en/github/getting-started-with-github/splitting-a-subfolder-out-into-a-new-repository - we could follow this process for the plugins and ensure that we end up with the correct branching strategy

• Try this with one of the plugins and see how this works - then maybe roll that out more widely

  • @Rahul Shinde is working on a POC for this in TPROD-204: Using the git split repositories, the CI pipeline does not function as expected as the app folder will be overwritten each time.Done

Maintaining the sync between mautic/mautic and the core-lib, plugins and theme repos

• Once we have split out the app directory, plugins and themes we then need to figure out how to keep the app directory, plugin and theme folders in sync
  
  

• If we physically remove the files from the main repo, we then need to figure out how to manage PR’s for those repos:
  
  

  • Do we accept them to core?

    • How, if the folders do not exist in GitHub - mautic/mautic: Mautic: Open Source Marketing Automation Software. ? I think this might not be possible if we remove the folders entirely.
      
      

  • Do we accept PR’s to plugin repos?

    • This would be really clunky if a bug fix needs to apply to several different plugins or to core and to the plugin - you would have to make multiple PR’s, know how to find the right repo, etc. Pretty poor/confusing developer experience?
      
      

• There are a few tools out there that we could potentially consider:

  • GitHub - dflydev/git-subsplit: Automate and simplify the process of managing one-way read-only subtree splits. [Recommend to not consider due to no recent updates]

    • Not sure if this is what we are after - it mentions a subtree split. Need to read more.

    • Allows you to do the subsplit and maintain thereafter

    • One-way read only sync

    • Allows you to specify the heads/tags to be updated if you don’t want to auto-detect

    • Seems to be abandonware - 4+ years since last update and multiple open issues/PR’s
      
      

  • GitHub - ryanwinchester/subsplit-service: automate subtree splits with webhooks and scheduled commands [Recommend to not consider due to no recent updates]

    • Allows you to do the subsplit and maintain thereafter

    • Composer-based install

    • Also supports webhooks if we need to use them

    • Seems to be abandonware - 5+ years since the last update
      
      

  • GitHub - jderusse/docker-gitsplit was mentioned by Tobias Nyholm (maintainer of subtreesplit.com/symfony core team)

    • Allows multiple branches

    • Allows you to use Github Actions

    • Splits are all maintained in a workflow file

    • Has to clone the entire repo - may have performance issues?

    • Triggered on push to main branch and on release, based on example - so should meet our needs
      
      

  • https://github.com/thephpleague/flysystem/blob/2.x/.github/workflows/publish-subsplits.yml was mentioned by Tobias Nyholm (maintainer of subtreesplit.com/symfony core team)

    • Github Action used by Flysystem to push subsplits automatically - could be something we could look into but may need some assistance from @Dennis Ameling to understand how we might use it on PR merge / release and how to bring it into our current workflows.

    • This supports actions running on multiple branches

    • We would need to set up the splits first, I think - once they are set up, then the action would run and push the changes when triggered.
      
      

  • https://github.com/splitsh/lite Symfony uses this tool but the docs are not all that great!

    • There is a hosted ‘service’ which uses this tool - created by one of the core Symfony team at https://www.subtreesplit.com/

      • Played around with it and was able to split the app/ folder from https://github.com/RCheesley/a-mautic-playground  to https://github.com/RCheesley/core-lib-playground  very easily.

      • It only pushes the default branch at the moment but the author may be able to dig into making it push other branches

      • It has the option to comment on PR’s that are made on the mirror repos in order to push people back to the main repo - nice feature that would hopefully save us a bit of a headache (I don’t think that you can disable PR’s on a repo)

      • Key point: the underlying technology seems to work!

         

  • Github Actions

    • https://github.com/drud/action-cross-commit this may work as a GitHub Action - we would need to work out what directories are being updated in the merged PR and push those directories to the correct repos using some kind of dynamic folder magic
      
      

    • https://github.com/s0/git-publish-subdir-action may also be an option (we would have to set up each folder and match it with the correct repo) - maybe run an ‘if files have changed’ check beforehand to only push changes if there are changes?

Governance of decoupled repos

• We need to make sure that all of the plugins and themes are appropriately licensed and that a readme & license file plus all the security and community health files are present
  
  

• We can set up a centralised .github repo that allows us to orchestrate all of our organisation repos centrally

  • We would need to reach out to the maintainers of repos ahead of time to make sure that they are aware of any changes (eg changes to branch names) that we may require / suggest

  • We can apply rules to groups of repos, to all repos, or manage them individually
    
    

• There is a tool that we can use to manage our repos centrally: https://organizer.gitconsensus.com

  • Open source project https://github.com/gitconsensus/GithubOrganizer

  • Has a Github App https://github.com/apps/organization-manager

Useful resources

• https://m.dotdev.co/splitting-out-a-directory-to-its-own-git-repository-3e5dc410871b

• Git - Atlassian Blog

• How to handle big repositories with Git | Atlassian Git Tutorial

• https://www.atlassian.com/git/tutorials/git-subtree (not planning to use this but read through it to discount subtrees)

• https://blog.developer.atlassian.com/the-power-of-git-subtree (As above)

Options considered

Action items

Outcome