Developing nested php composer projects using git subtree

When I started to develop a huge php project consisting of several composer (= php dependency manager) modules which I wanted to develop along with a core project I spent quite a while researching on how to master that. To hopefully save you some time I would like to share my approach here.
Although the following example focuses on organizing composer specific projects the idea is valid and suitable for all kinds of projects consisting of sub modules which need to be developed in parallel and to be organized in their own git repository.
Just one quick example: A content management system with plugin support. You might want to be able to develop a bunch of standard plugins along with your cms core application so that you can test them instantly before publishing them. You might not want the plugins to be bundled with your cms core application but to have them separated in a dedicated git repository so that everybody can use it independently from the core application.

My requirements

  • Changes in a module are instantly available to other modules without pushing the according changes of one module and pulling it back to the top level project or any dependent sub module (ensures testability before committing and pushing changes)
  • A clean and uninterrupted git history for each component as well as the top level project itself
  • Development of top level project including a set of independent sub modules organized in a single IDE project

Since composer requires each module (the top level project as each nested composer project) to have its own git repository I tried to find a suitable way of nesting git repositories. I don’t want to open the “why you do not want to use git submodules” topic again since there are plenty of resources on the internet explaining the disadvantages very well. However git submodules was not what I wanted to use. I found git subtrees but they were not very popular at that time when the project started so it was hard to find some information on how I could benefit from using them. But I figured out that git subtrees was exactly what I needed.

git subtrees

  • Split up existing repositories into smaller repositories while keeping the entire and specific history for that tree
  • Add existing subtrees to an existing project or start developing new components from scratch inside your top level project
  • Push and pull dedicated changes to/from any subtree from/into your top level repository

I had to get used to the usage of git subtrees a bit but in the end it works like a charm and the gain in flexility is priceless.

The project structure

tworabbits/
    - doc/
    - bin/
    - [...]
    - src/
        - Component1
            - [...]
            - composer.json
        - Component2
            - [...]
            - composer.json
        - [...]
        - ComponentN
        
    - composer.json

It is very important to keep the single components separated like you would do when developing the component in its own project scope. That does not mean they cannot use other components but they need to use it in the way your dependency manager defines. In terms of PHP and composer that means:

  • Keep the components inside their own namespace
  • Add interdependencies to other components to the according ComponentX/composer.json

Adding/pushing/pulling component sources

I actually doesn’t matter whether you start from scratch, you already have the top level project and only need to add sub modules or whether you already have an existing top level project and some sub modules organized in git repositories. You can apply the described mechanism in any state of your development process.

Starting from sratch (no existing sources)

1) Initialize a top level project repository in your project root directory: git init
2) Develop your core application as well as the single components inside your top level project
3) Push your changes to the core repository and each component into its own component repository:

git remote add origin git@bitbucket.org:tworabbits/core.git

# Add a remote for every single component you would like to keep separately
# git remote add [component_name] [url_to_repository]
git remote add plugin-one git@bitbucket.org:tworabbits/plugin-one.git

# Push your changes to the core repository
git push origin master

# Push each component to the dedicated component repository
# git subtree push --prefix=[path/to/component] [component_name] [branch]
git subtree push --prefix=src/Component1 plugin-one master

When your are working in a team and need to pull changes back into your core application:

# git subtree pull --prefix=[path/to/component] [component_name] [branch]
git subtree pull --prefix=src/Component1 plugin-one master

Splitting an existing project

# Create new bare repositories for each component you would like to split from your core application somewhere in your filesystem
git init --bare

# Add a remote component repository to your bare component repository
# git remote add origin [url_to_remote_component_repository]
git remote add origin git@bitbucket.org:tworabbits/plugin-one.git

# Inside your main repository split the component into a branch 'split'
# git subtree split --prefix=[path/to/component] -b split
git subtree split --prefix=src/Component1 -b split

# Push the 'split' branch to your bare component bare repository
# git push [/path/to/local/component/repository] split:master
git push /path/to/component/bare/repo split:master

# Push the bare repository to the component remote repository
git push origin master

# In your main repository: Remove the branch 'split' created before
git branch -D split

# In your main repository: Remove the component sources and commit your changes 
# (no worries, they are already save in your component remote at this point)
# git rm -r [path/to/component]
git rm -r src/Component1
git commit -am "Splitted Component1"

# Add the new component repository to the main repository
# git remote add [component_name] [url_to_remote_component_repository]
git remote add plugin-one git@bitbucket.org:tworabbits/plugin-one.git

# And pull the current component state into your main repository:
# git subtree pull --prefix=[path/to/component] [component_name] [branch]
git subtree pull --prefix=src/Component1 plugin-one master

Adding a new component

1) Start developing the component inside the according component sub directory of your core project
2) Add a component remote repository to your main repository
git add plugin-two git@bitbucket.org:tworabbits/plugin-two.git
3) Push your changes
git subtree push --prefix=src/Component2 plugin-two master

Branching

Branching just doesn’t change.
1) Checkout a new branch on your main repository
git checkout -b v1.5
2) Push main and component repositories to the according remote repositories
git push remote origin v1.5
git subtree push --prefix=src/Component1 plugin-one v1.5
git subtree push --prefix=src/Component2 plugin-two v1.5

Tagging components

To be honest tagging is the only thing which is not very handy since you have to split the components first into its own branch, tag it accordingly, push the tags and remove the splitted component branch:

# Split the component into a temporary branch
# git subtree split --prefix=[path/to/component] -b component-split
git subtree split --prefix=src/Component1 -b component-split

# Tag the branch accordingly
# git tag -a [tag_name] -m [tag_message] component-split
git tag -a v1.5.1 -m "Version tag v1.5.1" component-split

# And push the tags to the dedicated component repository
# git push [component_name] --tags
git push plugin-one --tags

# Don't forget to tidy up afterwards
git branch -D component-split
git tag -d v1.5.1

Automating the daily work

In order to automate my daily work I wrote the following little PHP cli scripts which push changes into a specific branch and create a tag if needed. Feel free to use it and let me know if it works for you.

1) Create a component.json containing a component <-> directory mapping
2) Create a push.php (code given below) and save it along with your components.json (i.e. in bin/git/push.php)
3) Call the script from your main repository, i.e.:
cd /home/dev/core
php ./bin/git/push.php [branch_name] [tag_name]
default of [branch_name] is master; no tag given means do not tag

components.json

[
    {
        "name": "plugin-one",
        "subtree": "src/Component1"
    },
    {
        "name": "plugin-two",
        "subtree": "src/Component2"
    }
]

push.php

<?php

if (php_sapi_name() !== 'cli') {
    echo "ERROR: This script should be executed from the command line.\n";
    return;
}

// Check if a certain branch is given
if (!isset($argv[1])) {

    if (!confirm('No specific branch selected, continue with master branch?')) {
        return;
    }
    $branch = 'master';

} else {
    $branch = $argv[1];
}

// Check if the components should be tagged or not
if (!isset($argv[2])) {
    $tag = null;
} else {
    $tag = $argv[2];
}

$components = json_decode(file_get_contents(__DIR__  . DIRECTORY_SEPARATOR . 'components.json'));
$gitConfigFile = getcwd() . DIRECTORY_SEPARATOR . '.git' . DIRECTORY_SEPARATOR . 'config';

if (!file_exists($gitConfigFile)) {
    echo "ERROR: Cannot find git config file for repository validation. Aborting.\n";
    return;
}

$gitConfig = parse_ini_file($gitConfigFile, true);

// Push main project first
echo "> Pushing parent repository ...\n";
echo 'git push origin ' . $branch . "\n\n";
exec('git push origin ' . $branch);

// Evaluate tag
if (null !== $tag) {

    echo "> Creating tag " . $tag . " for main repository\n";
    echo 'git tag -a ' . $tag . ' -m "Version ' . $tag . '"' . "\n\n";
    exec('git tag -a ' . $tag . ' -m "Version ' . $tag . '"');

    echo "> Pushing tag to main repository\n";
    echo "git push origin --tags\n\n";
    exec('git push origin --tags');

    echo "> Deleting tag\n";
    echo 'git tag -d ' . $tag . "\n\n";
    exec('git tag -d ' . $tag);

}

// Push each component to its component repository
foreach ($components as $component) {

    if (!is_dir(getcwd() . DIRECTORY_SEPARATOR . $component->subtree)) {
        echo "> Subtree '" . $component->subtree . "' not existing for component " . $component->name . ". Please check the component mappings.\n";
        continue;
    }

    echo '> Pushing subtree ' . $component->subtree . ' to ' . $gitConfig['remote ' . $component->name]['url'] . ' (branch ' . $branch . ")\n";
    echo 'git subtree push --prefix=' . $component->subtree . ' ' . $component->name . ' ' . $branch . "\n\n";
    exec('git subtree push --prefix=' . $component->subtree . ' --squash ' . $component->name . ' ' . $branch);

        // Evaluate tag
        if (null !== $tag) {

            $temporaryBranch = 'component-split';

            echo "> Splitting component into a temporary branch '" . $temporaryBranch . "'\n";
            echo 'git subtree split --prefix=' . $component->subtree . ' -b ' . $temporaryBranch . "\n\n";
            exec('git subtree split --prefix=' . $component->subtree . ' -b ' . $temporaryBranch);

            echo "> Creating tag " . $tag . " for component " . $component->name . " in branch '" . $temporaryBranch . "'\n";
            echo 'git tag -a ' . $tag . ' -m "Version ' . $tag . '" ' . $temporaryBranch . "\n\n";
            exec('git tag -a ' . $tag . ' -m "Version ' . $tag . '" ' . $temporaryBranch);

            echo "> Pushing tag to component repository\n";
            echo 'git push ' . $component->name . " --tags\n\n";
            exec('git push ' . $component->name . ' --tags');

            echo "> Removing temporary branch '" . $temporaryBranch . "'\n";
            echo 'git branch -D ' . $temporaryBranch . "\n\n";
            exec('git branch -D ' . $temporaryBranch);

            echo "> Removing tag\n";
            echo 'git tag -d ' . $tag . "\n\n";
            exec('git tag -d ' . $tag);

        }

}

function confirm($question) {

    while(true) {

        if (PHP_OS == 'WINNT') {
            echo '$> ' . $question . ' [Y/n] ';
            $line = strtolower(trim(stream_get_line(STDIN, 1024, PHP_EOL)));
        } else {
            $line = strtolower(trim(readline('$> ' . $question.' [Y/n] ')));
        }

        if(!$line || $line == 'y') {
            return true;
        } elseif($line == 'n') {
            return false;
        }

    }

}