Team Development + Continuous Integration in Kentico
How do we get the the changes we’ve been working on for months integrated smoothly into our live site without breaking it?
Content Management Systems (CMS) such as Kentico make updating your homepage, adding new pages, writing blogs or publishing news articles as simple as sending an email. But what about more significant changes, such as a whole new section, a complete redesign, or introducing mobile device support or responsive design? These sorts of changes require design and development work, just like the initial build of your site.
Typically the changes are made to a development copy of the site, or “dev site”, so your live site isn’t out of action throughout the entire process. But once a team of developers has built and tested a whole range of enhancements, you’re left with a dev site that has new features but no new content, and a live site that is technologically obsolete, but full of new content. How do we get the the changes we’ve been working on for months integrated smoothly into the live site without breaking it?
Firstly, let’s get clear on the key terms I’m using in this post.
By team development, I’m simply referring to having more than one developer working on a website at the same time. Ideally they should be able to work confidently and productively with each other, without holding each other up or interfering with (or overwriting) each other’s work.
Continuous Integration (CI) and Continuous Delivery (CD) are very exciting buzz-words in software engineering, and are very often misused! So let’s be clear on what I’m talking about:
Continuous Integration is merging your dev changes into source control very frequently, rather than making a large number of changes to a dev environment in isolation. This helps to avoid merge conflicts, sometimes referred to as “development hell”, and the inevitable issues associated with “big bang testing”.
Continuous Delivery refers to more frequent, smaller releases - or deployments - to your production environment. Incredibly, we’re seeing real-world examples where enterprise-level web applications are being deployed hourly. Employing CD makes Quality Assurance much easier, minimises the impact of any newly introduced issues, and enables critical issues to be addressed more quickly. It is in turn enabled by the proper use of source control and CI.
Sporadic Integration: The Wrong Way
The alternative to using CI and CD is to make a whole bunch of changes on a development environment, then get to the end and scratch your head over how to deploy a huge number of potentially breaking changes to your live site.
There are plenty of terrible ways to deploy changes to a website, and they are not specific to Kentico. They’re usually just a result of bad planning on the part of the person managing or building the site! Some (unfortunately) common methods I’ve seen employed:
The developer painstakingly reapplies every change made to the dev site, to the live site. This has the benefit of allowing you to keep all the content changes made to your live site while dev was happening, but unfortunately it’s extremely slow and prone to mistakes!
Blow away the live site
The safest and possibly quickest/cheapest option is to simply replace the live site with the latest copy of your dev site. Unfortunately you will lose any content changes that have been made to your live site since development began.
Content staging in reverse
By switching on content staging on the live site at the beginning of development, it is sometimes possible to push any content changes from the live site back down to your dev site during and after development, before initiating operation “blow away the live site” as described above.
If you’re interested enough in this subject to have read this far, admit it: you’ve employed one of the above methods before. :)
The Kentico Deliver Now! Methodology
Kentico have their own recommended best practice for team development including deployments, as explained in section 3.2 of their Deliver Now! methodology.
This post is not aiming to suggest you move away from Kentico’s best practices! The methods outlined in my post build upon Kentico’s established processes, mostly in the area of deployments.
In summary, the Deliver Now! methodology suggests the following for Team Development:
- Developers run their own local dev copy of the website (on their own machine)
- Developers use Source Control (such as TFS, SVN or Git) to stay in sync with files changes
- All developers’ dev sites connect to a single, shared dev database (over the network) so database change are shared immediately
- QA/Test/UAT/Production environments consist of their own web site and separate database
- Deployments between environments (such as pushing changes from dev to test) are done using the import/export functionality with Kentico for code changes (such as web parts), or the content staging module for database changes (such as document/page types).
If you haven’t read through the Deliver Now! methodology, I strongly encourage you to look now! The area I’m referring to is only a small part of what is a huge and comprehensive guide to Kentico development that I’d recommend any serious Kentico developer study.
Challenges Kentico 8 presents for CI and CD
Kentico is not like other web applications. In its current version at time of writing (8.2) many of Kentico’s strengths as a CMS that make it extremely powerful for development - such as the portal method and virtual objects - can make implementing team development, CI and CD a little tricky using traditional methods.
Objects stored in the database
Kentico 8 stores much of its architecture in its database, rather than serialised into physical files on disk, which prevents developers from pushing some changes through source control. An example of this is adding a field to a page type. It can also present difficulties when developers are sharing a database for development, as any database changes immediately affect all developers, even if they have not yet shared their code changes via source control. Proper use of CI (regular commits to source control) can alleviate a lot of this pain.
The Staging module as a deployment tool
Kentico deals with changes to its database objects between environments using its powerful Staging module. However the module in the current version of Kentico does not cater for multiple devs to be making changes and merging them into a single target, as is the case with traditional source control systems.
In the case where more than one dev makes a change and pushes it through staging, they will overwrite each other’s changes in many cases, as a full copy of the new object will be created, rather than an appropriately modified version (or “delta”).
With traditional source control, dev changes will be merged into the test environment. Using Kentico 8’s staging module, each dev environment will simply push a new version to test that is an exact copy of its own dev version.
This is one of the reasons it’s currently highly recommended by Kentico (and by me) that developers connect to a shared database.
If staging is used to push database changes in a team dev environment, as of version 8.2, there is currently no way to reverse, or “back out” a staging push. It is possible to revert to an earlier version of an object on the target server, however this is a manual process and no longer part of a deployment chain. A key benefit of source control systems is being able to quickly move backward or forward to any point in a repository’s history.
Of course, most problems have a solution, or at least a workaround!
The following approach is a work in progress, but it has been working fairly well for us on recent projects. It satisfies our main requirements:
- Cleanly separated environments for dev, QA, test, staging, production
- Clearly-defined deployments including database and file changes
- Version history for (close to) all files and virtual objects
- Reliable QA processes, and predictable (safe) deployments
- Team development with minimal disruption caused by individual devs
We store as many physical files on disk as possible. This is supported by many Kentico settings, so it includes:
- custom code files
- web parts
- custom modules
- images, both content managed and design assets
- Kentico Objects/Resources
Under Kentico’s recommended Portal development method, it stores “Virtual Objects” directly in the database, allowing them to be edited and version-controlled through the web interface.
By enabling Kentico’s “Source Control Mode” we store these objects on disk in CMSVirtualFiles. Developers can then edit Virtual Objects such as CSS, transformations, templates layouts, rapidly in Visual Studio, or continue to do so through the GUI (CMSDesk or Admin).
The entire Kentico website project is then tracked in source control. We use git and Bitbucket.
Per Kentico’s best practices, we use a single shared dev database, often configured as a web farm if development is proceeding at a rapid pace!
Our developers are careful to make any database changes in as non-destructive a manner as possible. For example, if fields are to be added to and removed from a Page Type, the new fields should be added first, without immediately removing the unneeded fields until a later time when you’re confident you won’t be breaking another developer’s website.
For the remaining items that are not tracked in Source Control, such as Page Types, we continue to use object staging via the Staging module for deployments.
Our deployments are strictly controlled and managed by a combination of tools.
Bitbucket, git, and git flow
We have a separate branch within the git repo for each deployment target, e.g. dev, and test, with master representing the production environment (live).
Feature branches are then created by developers for independent tasks, to minimise code changes outside of the scope of their task.
dploy.io is an extremely powerful tool that allows us to deploy any commit from any branch of a git repo, to any deployment target, with the click of a button.
Generally, we have it configured like so:
- the dev branch is automatically deployed to an internal dev site (build server) on each commit
- other branches (test, qa, master) are deployed manually, on demand
This system means that once testing on an environment is complete, the process for an entire deployment from qa to live can be as simple as:
- Press the “sync” button in the Staging module
- Press the “deploy” button in dploy.io
So far our results are extremely promising.
At a minimum, we’ve seen that the effect of strictly enforcing use of source control and managed deployments has made it much easier for one developer to continue work that was begun by another developer, or deploy another developer’s changes in their absence.
A number of our teams have successfully moved to a regular deployment cycle where changes are planned out, developed, tested and deployed monthly, fortnightly, or even weekly.
On the most positive end of the scale, one of our dev team leads on a particularly large project with lots of customisation and a distributed dev team, reports that we’ve gone from generally dedicating a full day to a significant deployment, to the whole process only taking around fifteen minutes!
The system is still not perfect, but we’re working on it.
For a smaller project, or one with very little ongoing maintenance, it can be difficult to justify the time taken to set up the systems, processes and environments required for a full CI/CD setup. We’re working on getting the setup time down as low as possible.
Shared database and source control not always in sync
Significant changes to the database, such as creating/registering a new Page Template or Web Part, can cause other developers in the team to see errors in their website. This is currently handled by strict dev processes such as always committing and pushing your code files immediately following the creating of any significant object such as this, so any other developer can resolve errors by simply pulling the latest changes from the repository.
The current version of the staging module won’t let developers have separate databases
The staging module rules with an iron fist! It’s very powerful, but with great power comes great responsibility. If the staging module were able to push changes to an object rather than an entirely new version of an object, we could potentially move to a completely distributed dev model.
Staging doesn’t work between sites of different versions, so a major version upgrade simply can’t happen through a process such as this. Currently it’s a requirement to run the upgrade process against each environment separately. However, never fear! We have nearly perfected our upgrade process to work within this methodology, and I’ll post about that soon!
Latency affects shared database performance for remote developers
This is not an issue specifically with Kentico - it’s an issue with any developer connecting to a remote database. Other competing CMSs also recommend shared databases for team development.
On any one page in edit or preview mode (with caching obviously not in action), a huge number of queries will be executed against the database, meaning tens or potentially hundreds of round trips between the web server and the database server. Usually this isn’t a problem with a local database on your gigabit LAN, however if you increase the latency on the connection from a couple of milliseconds to a couple of hundred milliseconds, suddenly you’re waiting a full minute for a page to load.
Some Other Ideas
I’ve spoken to a bunch of Kentico partners (and staff!) about this, and I’m aware that there are a number of other ways people are tackling these situations, for example:
- Third-party tools (such as the excellent products from www.red-gate.com) can be used for intelligently synchronising changes to a database’s structure and data.
- Enable versioning everywhere, then use staging to push all changes up the dev seniority chain. This can deal with the risk of overwriting changes as part of a peer review by a senior, and means the most senior dev is the only person pushing all changes further beyond the dev environments.
How are you handling team development and deployments for websites? Let me know if you have other ideas (or if you think mine are rubbish) in the comments below!
Want more? Here are some other blog posts you might be interested in.
A content calendar can be an extremely powerful tool – if well set-up and maintained. Content Strategist Tami Iseli outlines some of the factors that can reduce the chances of abandonment, along with a downloadable content calendar spreadsheet that automatically syncs your content ideas to a Google calendar.
Our CTO, Andy Thompson, has been named as a Kentico MVP for the fifth consecutive year.