A common recommendation is to have a reliable deployment for your software. At first sight, this sounds very hard to learn and to achieve. But with the right tool and a good introduction, this becomes quite easy to manage, even for freelancers or small agencies. In this post, I will introduce you to TYPO3 Surf.

TYPO3 Surf originated from the TYPO3 Neos project. Main goal was to deploy (TYPO3) Flow and (TYPO3) Neos applications and distributions. Shortly after the start of TYPO3 Surf, first steps were taken to use this tool also for TYPO3 CMS. Back then, it was a (TYPO3) Flow application. After the split of the projects steps were taken to remove this dependency and to rely on Symfony components instead.

The main maintainer currently is Helmut Hummel (@helhum). Thank you, Helmut for your time and dedication to bring this project forward.

TYPO3 Surf is (of the time of writing) in beta-state. This week the beta7 of version 2.0.0 was tagged. Source and official documentation are available at Github: https://github.com/TYPO3/Surf/. TYPO3 Surf also uses the bug tracker there.

But this should be enough background info … Let’s get our hands dirty now!

Project Setup and Prerequisites for TYPO3 Surf

There are just a few prerequisites to be able to run a deployment successfully:

Git Repository

For this tutorial, we need a git repository on a server, which is available from your console. Any service can be used for this like GitHub, Bitbucket or GitLab. It must be publicly available or you must add a deployment key to this repository.

Directory structure

In your project root there must be a “Web”- and a “Build”-folder. The DocumentRoot of the webserver points to the “web” folder. The “Build” folder contains everything, which will be necessary to deploy the website (and probably more, if your project contains also a klickdummy and / or styleguide). The “Web” folder contains everything which will be deployed to the target instance.

• Project-Root
  • Build
  • Web

Use composer

Your TYPO3 installation should be composer based, as it eases the package management a lot and helps deploying a website. It is possible to use TYPO3 Surf without composer, but I will not cover it in this article. If you want a composer based setup, please follow the tutorial of Daniel Goerz (@ervaude) on his blog at https://usetypo3.com/typo3-and-composer.html.

SSH-Account

For a (successful) deployment you need a ssh account on the target site. This must be accessible via public key authentication.

Surf Basics

In this section I introduce you to some definition and wordings, that help to understand the whole TYPO3 Surf deployment.

Deployment

This is the aggregate of all things, defined in the following components. Usually I have four deployments at hand, because my default workflow follows the Git Flow Branching model: preview, develop, release and live.

A deployment consists of one workflow and one or more applications. The applications share the workflow, but the tasks are be defined for each application.

Application

An application is the piece of software which is deployed. In our case it is solely TYPO3 CMS. But basically any other web software can be deployed with TYPO3 Surf. Currently TYPO3 Surf brings some pre-configuration for TYPO3 CMS, Flow and Neos. For all other applications you must take care for all things yourself, especially for the workflows and tasks.

Node

A node is a server instance to which the application is deployed. Depending on your infrastructure, there may be one or more nodes.

Tasks

Tasks are single steps which are executed in defined stages. For the already mentioned software there are already several predefined tasks available.

Workflow

The workflow defines which tasks are executed in which order. To make the grouping of tasks easier, nine stages are defined in the workflow. The “Simple Workflow” is the default workflow in TYPO3 Surf. In this workflow the following stages are available. Not all of them may be used in every workflow.

initialize

This is normally used only for an initial deployment to an instance. At this stage you may prefill certain directories for example.

package

This stage is where you normally package all files and assets, which will be transferred to the next stage.

transfer

Here all tasks are located which serve to transfer the assets from your local computer to the node, where the application runs.

update

If necessary, the transferred assets can be updated at this stage on the foreign instance.

migrate

Spoken in words of the InstallTool: This is the database compare. If you want to update contents of certain tables, this is also the stage to use. But this is definitely not the stage to delete unused columns and tables in the database, because the old code, relying on these, is still live.

finalize

This stage is meant for tasks, that should be done short before going live, like cache warmups and so on.

test

In the test stage you can make tests, which work before switching the releases.

switch

This is “the” crucial stage. Here the old live instance is switched with the new prepared instance. Normally the new instance is symlinked.

cleanup

At this stage old releases are cleaned up. Normally you will keep three to five releases, in case something goes totally wrong. Then you have still the possibility to switch back to an older release by just changing the symlink.

This was the big picture of a TYPO3 Surf deployment. In the next section, I will show you how to set it up.

Get TYPO3 Surf up and running

Installing TYPO3 Surf

First thing is to set up the directory structure. Therefore a folder below the “Build” directory must be created. I usually call it “Deployment”. Then switch into this folder

cd <project-root>/Build
mkdir Deployment
cd Deployment

I recommend to install TYPO3 Surf via composer. That’s means, we need a small composer.json here. My file looks like this

{
   "require": {
      "typo3/surf": "dev-master"
   },
   "autoload": {
      "psr-4": {
         "MarcusSchwemer\\MyCustomer\\Deployment\\": "src"
      }
   }
}

The “require” – section installs TYPO3 Surf in the current directory. The autoload section is needed for later customization of the deployment.

Now run a “composer install” in this directory. This will fetch all necessary dependencies and TYPO3 Surf itself in a subdirectory “vendor”.

Create local folders for the deployment

In the next step you must create the rest of the required folder structure.
We are still in “<project-root>/Build/Deployment”.

mkdir .surf
mkdir -p src/Application
mkdir -p src/Node
mkdir -p src/Task

The subfolder “.surf” will contain the deployments. The other subfolder are self-explaining as they contain exactly the things how they are named.

Create the node definition

In the next step we must define a node in “src/Node”. The code of the class is this:

<?php
  namespace MarcusSchwemer\MyCustomer\Deployment\Node;

  class MyCustomerNode extends \TYPO3\Surf\Domain\Model\Node
  {
     /**
      * @var array
      */
      protected $options = [
         'username' => 'my-ssh-user',
         'hostname' => 'hostname-to-deploy-to',
      ];

      public function __construct()
      {
         parent::__construct('Webserver for my customer');
      }
   }

Basically the only thing we do here is to set the ssh username and the hostname, we are deploying to.

Define the application

Now the application must be defined. This is also just a php class. As the complete application with all tasks is defined, it is a little bit longer. I add some notes about the function below the code.

<?php
namespace MarcusSchwemer\MyCustomer\Deployment\Application;

use MarcusSchwemer\MyCustomer\Deployment\Task\LocalInstallTask;
use TYPO3\Surf\Domain\Model\Deployment;
use TYPO3\Surf\Domain\Model\Workflow;
use TYPO3\Surf\Task\Package\GitTask;
use TYPO3\Surf\Task\Php\WebOpcacheResetCreateScriptTask;
use TYPO3\Surf\Task\Php\WebOpcacheResetExecuteTask;
use TYPO3\Surf\Task\TYPO3\CMS\CompareDatabaseTask;
use TYPO3\Surf\Task\TYPO3\CMS\FlushCachesTask;
use TYPO3\Surf\Task\TYPO3\CMS\SetUpExtensionsTask;

class MyCustomerApplication extends \TYPO3\Surf\Application\TYPO3\CMS
{
   /**
    * @param string $deploymentPath
    * @param string $baseUrl Used as an required option in the WebOpcacheResetExecuteTask
    * @throws \Exception
    */
    public function __construct($deploymentPath, $baseUrl)
    {
        if (!$deploymentPath) {
            throw new \Exception(
                'Deployment path was not set in Surf deplomyent configuration file',
                1479394104502
            );
        }

        if (!$baseUrl) {
            throw new \Exception(
                'Base URL was not set in Surf deplomyent configuration file',
                1479882363615
            );
        }

        parent::__construct('MyCustomersWeb');

        $this->setOptionDeploymentSource();
        $this->setOption('keepReleases', 5);
        $this->setOption('repositoryUrl', 'git@bitbucket.org:mschwemer/mycustomer.git');
        $this->setOption('composerCommandPath', 'composer');
        $this->setOption('applicationRootDirectory', 'Web');
        // "baseUrl" is a required option of WebOpcacheResetExecuteTask
        $this->setOption('baseUrl', $baseUrl);
        // "databaseCompareMode" is used by the CompareDatabaseTask
        $this->setOption('databaseCompareMode', '*.add');
        $this->setOption(
            'rsyncExcludes',
            [
                '.DS_Store',
                '/.editorconfig',
                '/.git',
                '/.gitignore',
                '/Build',
                '/build.xml',
                '/Web/composer.json',
                '/Web/composer.lock',
            ]
        );
        // The folders "fileadmin", "uploads" and "AdditionalConfiguration" must not be part
        // of your repository as they contain user generated content or contain node specific
        // settings in the "AdditionalConfiguration". 
        $this->setSymlinks(
            [
                'Web/fileadmin' => '../../../shared/Data/fileadmin',
                'Web/uploads' => '../../../shared/Data/uploads',
                'Web/typo3conf/AdditionalConfiguration' => '../../../../shared/Configuration/AdditionalConfiguration/',
            ]
        );

        $this->setDeploymentPath($deploymentPath);
    }

    /**
     * @param Workflow $workflow
     * @param Deployment $deployment
     */
    public function registerTasks(Workflow $workflow, Deployment $deployment)
    {
        parent::registerTasks($workflow, $deployment);
        $this->defineTasks($workflow, $deployment);
        $workflow->addTask(WebOpcacheResetCreateScriptTask::class, 'package', $this);
        $workflow->addTask('MarcusSchwemer\\MyCustomer\\DefinedTask\\FixFolderStructure', 'migrate', $this);
        $workflow->addTask(WebOpcacheResetExecuteTask::class, 'switch', $this);
        $workflow->addTask('MarcusSchwemer\\MyCustomer\\DefinedTask\\LanguageUpdate', 'finalize', $this);
    }

    /**
     * Define the individual tasks
     *
     * All these tasks require the typo3_console. Please install it with your projects composer json.
     *
     * @param Workflow $workflow
     * @return void
     */

    private function defineTasks(Workflow $workflow)
    {
        $workflow->defineTask(
            'MarcusSchwemer\\MyCustomer\\DefinedTask\\LanguageUpdate',
            \TYPO3\Surf\Task\ShellTask::class,
            ['command' => 'php {releasePath}/Web/vendor/bin/typo3cms language:update',]
        );

        $workflow->defineTask(
            'MarcusSchwemer\\MyCustomer\\DefinedTask\\FixFolderStructure',
            \TYPO3\Surf\Task\ShellTask::class,
            ['command' => 'php {releasePath}/Web/vendor/bin/typo3cms install:fixfolderstructure',]
        );
    }

    /**
     * This method checks whether there is a correct deployment source specified. If not, it throws an exception
     * TODO: This method is not project specific and
     * may be put into something like a Library of Surf deployment related
     * classes in the future.
     *
     * @throws \Exception
     * @return void
     */
     protected function setOptionDeploymentSource()
     {
         $source = getenv('DEPLOYMENT_SOURCE');

         if (!is_string($source)) {
             throw new \Exception('DEPLOYMENT_SOURCE environment variable is missing. Pattern: "DEPLOYMENT_SOURCE=branch|tag|sha1:foobar"', 1479391741322);
         }

         $sourceArray = explode(':', $source);

         if (
             count($sourceArray) === 2
             &&
             in_array(
                 $sourceArray[0],
                 ['sha1', 'branch', 'tag']
             )
         ) {
             $this->setOption($sourceArray[0], $sourceArray[1]);
         } else {
             throw new \InvalidArgumentException(
                 'DEPLOYMENT_SOURCE environment variable does not meet the mandatory pattern. Pattern: "DEPLOYMENT_SOURCE=branch|tag|sha1:foobar", 1479391747337',
                 1455797642
             );
         }
     }
}

This “application” extends the default TYPO3 CMS application. The TYPO3 CMS application defines five default tasks, to which additional tasks can be added. The code for this application is located at “Build/Deployment/vendor/typo3/surf/src/Application/TYPO3/CMS.php”. Take a look at this code to understand in detail what’s happening there. 

The constructor of the class contains all main definitions and options for this application. The function “registerTasks” is there to add new tasks to the workflow and to place them into the right order. TYPO3 Surf brings already a couple of predefined tasks. You can find all predefined tasks in “Build/Deployment/vendor/typo3/surf/src/Task” and in the subfolders below. The names of the subsequent folders reflect the purpose of the tasks.

I grouped the individual tasks in the function “defineTasks”. As these are quite short, I think, it is ok to do this here. If you need tasks with more lines of code, these should be put in the directory “Build/Deployment/src/Task” of your deployment. 

The function “checkDeploymentSource” checks whether the necessary environment variable “DEPLOYMENT_SOURCE” is available. It must set on the command line right in front of the rest of the cli call to TYPO3 Surf.

Define the deployment

The final step before you can run the deployment is to define the deployment. The deployment glues the above described components together. The deployment scripts are created in the folder “Build/Deployment/.surf”. If you want to deploy to a development instance create a php script called “develop.php” in this folder. If you have more deployment, you must create a file for each deployment. The following lines are sufficient to create a deployment:

<?php
use MarcusSchwemer\MyCustomer\Deployment\Application;
use MarcusSchwemer\MyCustomer\Deployment\Node;

$application = new MyCustomerApplication(
    '/var/www/vhosts/mycustomer/develop',
    'http://develop.mycustomer.de'
);

$node = new MyCustomerNode();

$application->addNode($node);

/** @var \TYPO3\Surf\Domain\Model\Deployment $deployment */
$deployment->addApplication($application);

The first option for the constructor of the application is the path, where the application is installed. This must be available on the target system.

Prepare the target system

The first step is to provide the deployment directory on the target node. It is the base directory where TYPO3 Surf will act. It must be writeable for the web server and the ssh user, you defined in the node definition. In order to have a running dummy, you must create two more folder levels below: “releases/<releasetimestamp>”. At this time “<releasetimestamp>” can be named as you like. But you must link it with “current”, as TYPO3 Surf will need this symlink while deploying:

cd <deployment-path>/releases/
mkdir <releasetimestamp>
ln -s <releasetimestamp> current

In the folder “current” aka “<releasetimestamp>” will reside the complete deployed software with both folders “Build” and “Web”.  This means the “DocumentRoot” on the target node must be “<DeploymentPath>/releases/current/Web/”. 

Furthermore the “shared” folder for “fileadmin” and “uploads” must be placed in parallel to the “releases” folder.  The symlinks to these folders are recreated in every deployment. If you decide for another location, you must adapt the paths in the application for “$this->setSymlinks”.

cd <deployment-path>
mkdir -p shared/Data/fileadmin
mkdir -p shared/Data/uploads
mkdir -p shared/Configuration/AdditionalConfiguration

Run the deployment via TYPO3 Surf

Now you are able to run the deployment with the following cli call:

cd Build/Deployment/
DEPLOYMENT_SOURCE=tag:1.0.0 ./vendor/bin/surf deploy develop

This call runs the deployment defined “develop”. It deploys the code of the tag “1.0.0”. The deployment source can be one of “sha1”, “branch” or “tag”. “sha1” means that you use a specific commit for deployment. “branch” uses the head of the branch for the deployment. “tag” is a tag which must be available in the remote repository.

Conclusion

It is not that hard to create a reliable deployment and to roll out defined versions of your web application / TYPO3 CMS at any time. It has the advantage, that you cannot forget any step, as it might happen if you do it manually.