OSS Build and Release with VSTS

Over the past few weeks I have been moving the build system for the OSS projects I maintain over to use VSTS. Up until recently I was using AppVeyor for builds, as they have provided a generous free offering for OSS for years. A huge thank you goes out to them for their past and ongoing support for OSS. So why move to VSTS? There’s three reasons for me:

  1. Support for public projects. This is key since there’s no point in using their builds if users can’t see the results.
  2. Release Management. The existing build systems like AppVeyor, Jenkins, TeamCity, and Travis, can all build a project. Sure, they have different strengths and weaknesses, and some offer free OSS builds as well, but none of them really has a Release Management story. That is, they can build artifacts….but then what? How do the bits get where you want, like NuGet, MyGet, a Store, etc. This is where release management fits in as a central part of CI/CD. More on this later.
  3. Windows, Linux, and Mac build host support in one system. It’s possible to run a single build on all three at the same time (fan out/in), like how VS Code does. No other host can do this easily today using a hosted build pool. I should note that using a hosted build pool is critical for security if you want to build pull requests from public forks. You don’t want to be running arbitrary code on a private build agent. Hosted agent VMs are destroyed after each use, making them far safer.

Life without Release Management

Many projects strive to achieve a continuous deployment pipeline without using Release Management (RM from now on). This is often achieved by using the build script or configuration to do some deployment steps given certain conditions, like if building a certain branch like master. In some ways, the GitFlow branching strategy encourages this, making it easy to decide that builds from the develop branch are pre-release and should go to a dev environment, while builds from master are production and should thus be deployed to a production environment. To me, this is conflating the real purpose of branches, which should be isolation of code churn, from deployment target. I believe any artifact should be able to be deployed to any environment, releasing is a different process than build and should have no bearing on which branch it comes from. For the vast majority of projects, I believe that a GitHub Flow or Release Flow (video) is a better, simpler, option.

Without RM, a pipeline to deploy a library might look something like this:

  1. Builds on the develop branch get deployed to MyGet by the build system. To me, it doesn’t much matter if it’s in the build script directly or if it’s build server configuration (like the deployment option in AppVeyor).
  2. To create a stable release, code is merged to master and then tagged. Often, tags are the mechanism that determines if there should be a release — effectively, tag a commit, that triggers a build which gets released to NuGet.org.

In this model, it’s usually a different build that gets deployed to release than to dev. I believe that mixing build and release like this ultimately leads to less flexibility and more coupling. The source system has to know about the deployment targets. If you need to change the deployment target, or add another one, you have to commit to the source and rebuild.

Following the single responsibility principle, a build should produce artifacts, that’s it. Deployment is something else, don’t conflate the two concepts. Repeat after me: Build is just build. I think projects have tended to mix the two, in part because there wasn’t a good, free, RM tool. It was easy and pragmatic to do both from the build tool. That changes now with VSTS public projects.

CD Nirvana with Release Management

VSTS has a full featured RM tool (deep dive on docs here) that is part of the platform. It is explicitly designed around the concept of artifacts, environments, and releases. In short, a build that contains artifacts can trigger a release. A release defines one or more environments with specific steps that should execute for each. Environments can be chained after another one, enabling a deployment promotion flow. There are many ways to gate each environment, automated and manual. A configuration I use frequently is to have two environments: MyGet and NuGet (dev and prod, respectively). The NuGet environment has an approval step so that releases don’t automatically flow from dev to prod; rather, I can decide to release to production at any time. Any build is a potential release.

Release steps can do anything and there are many existing tasks built-in for common things (like NuGet push, Azure blob copy, Azure App Services, and Docker) and a rich marketplace for things that aren’t (like creating a GitHub release, tagging the commit, and uploading artifacts to the GitHub release). In addition, you can run any custom script.

I think it’s easier to show by example, and that’s what follows in the next sections.

Versioning

Having an automatic version baked into your build artifacts is a crucial element. I use Andrew Arnott‘s Nerdbank.GitVersioning package to handle that for me. I set the Major.Minor in a version.json file and it increments the Patch based on the Git commit height since the last minor change. Add a prerelease tag to the version, if desired. You can control where the git height goes if you don’t want it in the patch (like 1.2.0-build.{height}). The default is the patch, and I think it’s completely okay to have a release like 1.2.42 if there were 42 commits since the version bump. I believe too much time is wasted on “clean” versions; it’s just a number :). Nerdbank.GitVersioning can also can set the build number in the agent, which is really handy knowing what version was just built.

Structuring your branches without overkill

There are many theories around how to structure your branches in Git. I tend to go with simplicity, aiming for a protected master branch with topic/feature branches for work. In my view, the sole reason for branches should be around code churn and isolation.

When it comes to delivery, there are two main schools of thought: releases and continuous. Releases are the traditional way shipping software. A group of features is batched together and shipped out once someone decides “it’s ready.” Continuous Delivery (CD) takes the thought out of releases: every build gets deployed. Note that doesn’t mean every build gets deployed to all environments, but every build is treated as if it could be.

I bring this up because I choose different tagging/branching strategies based on whether I’m doing releases or full CD.

If you’re doing continuous delivery, I would suggest using a single master branch with a stable version in it. Every build triggers a release, at least to a CI environment. At some point, could be every release, a set schedule, etc, that build gets promoted to the production environment. The key here is that it’s a promotion process; builds are fixed and flow through the environments.

If you’re doing release-based delivery, I would suggest using the master with a prerelease version tagged in it (like 1.2-preview). When you’re ready to stabilize your release, cut a rel/1.2 branch for it. In that branch, remove the prerelease tag and continue your stabilization process. Fixes should target master via a PR and then cherry-picked to the release branch if applicable. The release branch never merges back to master in this model.

In my view, using rel/* is perfect for stabilization of a release, enabling master to proceed to the next release. I’ll come back to my earlier point about branches: they should be for isolation of code churn, not environments. A rel branch isn’t always necessary; I’d only create it if there is parallel development happening.

Examples

I have two examples that illustrate how I implement the strategy above.

First, a library author creating a package that gets deployed to two feeds: a CI feed and a stable feed. The tree uses a preview prerelease tag in master and branches underneath rel for a stable release build. My example shows a .NET library with MyGet and NuGet, but the concepts apply to anything.

Second, I have an application that does continuous deployment to an automatically updating CI feed and controlled releases to the Microsoft Store, Chocolatey, NuGet, and GitHub. All releases move forward in master, and will hotfix under rel only if necessary.

Basic Library

Build

For this first scenario, I’ll talk about Rx.NET. It has a a build definition defined in yaml that has these essential parts (non-relevant parts omitted for brevity):

trigger:
- master
- rel/*

queue: Hosted VS2017

variables: 
  BuildConfiguration: Release
  BuildPlatform: Any CPU

steps:
- task: BatchScript@1
  inputs:
    filename: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Enterprise\\Common7\\Tools\\VsDevCmd.bat"
    arguments: -no_logo
    modifyEnvironment: true
  displayName: Setup Environment Variables

- task: PowerShell@1
  inputs:
    scriptName: 'Rx.NET/Source/build-new.ps1'
    workingFolder: 'Rx.NET/Source'
  env:
    VSTS_ACCESS_TOKEN: $(System.AccessToken)
  displayName: Build

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: 'Rx.NET/Source/artifacts'
    ArtifactName: artifacts
    publishLocation: Container
  condition: always()

I’m not going to dive too deep in the YAML itself; instead I’ll call your attention to the documentation and examples. As of now, there may be more up-to-date docs in their GitHub repo.

In this case, I have a build script (build-new.ps1) that calls msbuild and does all of the work. In order to ensure the right things are in the path, I call out to the VsDevCmd.bat first. xUnit, as of 2.4.0 beta 2, has direct support for publishing test results to VSTS if you supply the VSTS_ACCESS_TOKEN variable. Other frameworks are supported by the VSTest task. After running the main build script, I use a task to publish the binaries (NuGet packages) that were generated ty the build.

Another approach is to use the tasks for all of this directly, similar to this. We all have preferences between using scripts like PowerShell, Cake, PSake, etc, and the tasks. Doesn’t matter what you pick, use what works for you.

Release

The previous section was about build. The end result is a versioned set of artifacts that can be used as input to a release process. Rx.NET has a release definition here:

Release Management Pipeline

One tip on release naming that’s easily overlooked: it can be customized. I like to put the build number in it so I can associate a release with a version, and I concat it with the instance number (since it’s possible to have multiple releases for a particular version). In the definition options, I use the string Release v$(Build.BuildNumber).$(rev:r). That uses the build number from the primary artifact as the name.

The release defines two environments, MyGet and NuGet with an auto release trigger for builds on the master or rel/* branches. RM lets you put branch filters at any point, so you can enforce that releases only come from a specified branch, if desired. In this case, I tell it to create a release after builds from those branches. Then, in the MyGet environment, I’ve configured it to deploy to that environment automatically upon a release creation. That gets me Build -> Release -> MyGet in a CD pipeline. I do want to control releases to NuGet in two ways: 1) I want to ensure they are in MyGet, and 2) I want to manually approve it. I don’t want every build to go to NuGet. I have configured the NuGet environment to do just that, as well as only allowing the latest release to be deployed (I’m not looking to deploy older releases after-the-fact).

The MyGet environment has one step: a NuGet push to a configured endpoint. The NuGet environment has two steps: create a GitHub release (which will tag the commit for me), and a NuGet push. Releases don’t have to be complicated to benefit from using an RM flow. My release process is simple: when it’s time for a release, I take the selected build and push the “approve” button on the NuGet environment. There’s many other ways to gate releases to environments and you can do almost anything by calling out to an Azure Function as a gate.

It’s a bit hard to see how the release pipelines are configured on the site, so here are some screenshots showing the configuration:

Deployment Trigger:
Deployment Trigger

MyGet Environment:
MyGet Environment

NuGet Environment:
NuGet Environment

Environment Triggers for NuGet
Approvers
Queue

The actual release process to NuGet goes like this:

  • If I want to release a prerelease package, I can just press the approve button. It’ll do the rest.
  • If I want to release a stable package, I create a branch called rel/4.0 (for example) and make one edit to the version.json to remove the prerelease tag. That branch will never merge back to master. I can do as much stabilization in that branch as needed, and when I’m ready, I can approve that release to the NuGet environment. If there are hotfix releases I need to make, I will always make the changes to master (via a PR), then cherry-pick to the rel branch. This ensures that the next release always contains all of the fixes.

A Desktop Application

NuGet Package Explorer (NPE) is a WPF desktop application that is released to the Microsoft Store, Chocolatey, and GitHub as a zip. It also has a CI feed that auto-updates by using AppInstaller. NPE is delivered via a full CD process. There aren’t any prerelease versions; every build is a potential release and goes through an environment promotion pipeline.

Build

As a Desktop Bridge application, it contains a manifest file that must be updated with a version. Likewise, the Chocolatey package must be versioned as well. While there may be better options, I’m currently using a PowerShell script at build to replace a fixed version with one generated from Nerdbank.GitVersioning. I also update a build badge for use as an deployment artifact later.

# version    
nuget install NerdBank.GitVersioning -SolutionDir $(Build.SourcesDirectory) -Verbosity quiet -ExcludeVersion

$vers = & $(Build.SourcesDirectory)\packages\nerdbank.gitversioning\tools\Get-Version.ps1
$ver = $vers.SimpleVersion

# Update appxmanifests. These must be done before build.
$doc = Get-Content ".\PackageExplorer.Package\package.appxmanifest"    
$doc | % { $_.Replace("3.25.0", "$ver") } | Set-Content ".\PackageExplorer.Package\package.appxmanifest"

$doc = Get-Content ".\PackageExplorer.Package.Nightly\package.appxmanifest"    
$doc | % { $_.Replace("3.25.0", "$ver") } | Set-Content ".\PackageExplorer.Package.Nightly\package.appxmanifest"

$doc = Get-Content ".\Build\PackageExplorer.Package.Nightly.appinstaller"    
$doc | % { $_.Replace("3.25.0", "$ver") } | Set-Content "$(Build.ArtifactStagingDirectory)\Nightly\PackageExplorer.Package.Nightly.appinstaller"

# Build PackageExplorer
msbuild .\PackageExplorer\NuGetPackageExplorer.csproj /m /p:Configuration=$(BuildConfiguration) /bl:$(Build.ArtifactStagingDirectory)\Logs\Build-PackageExplorer.binlog
msbuild .\PackageExplorer.Package.Nightly\PackageExplorer.Package.Nightly.wapproj /m /p:Configuration=$(BuildConfiguration) /p:AppxPackageDir="$(Build.ArtifactStagingDirectory)\Nightly\" /bl:$(Build.ArtifactStagingDirectory)\Logs\Build-NightlyPackage.binlog
msbuild .\PackageExplorer.Package\PackageExplorer.Package.wapproj /m /p:Configuration=$(BuildConfiguration) /p:AppxPackageDir="$(Build.ArtifactStagingDirectory)\Store\" /p:UapAppxPackageBuildMode=StoreUpload /bl:$(Build.ArtifactStagingDirectory)\Logs\Build-Package.binlog

# Update versions
$doc = Get-Content ".\Build\ci_badge.svg"    
$doc | % { $_.Replace("ver_number", "$ver.0") } | Set-Content "$(Build.ArtifactStagingDirectory)\Nightly\version_badge.svg"

$doc = Get-Content ".\Build\store_badge.svg"    
$doc | % { $_.Replace("ver_number", "$ver.0") } | Set-Content "$(Build.ArtifactStagingDirectory)\Store\version_badge.svg"

# Choco and NuGet 
# Get choco

$nugetVer = $vers.NuGetPackageVersion

nuget install chocolatey -SolutionDir $(Build.SourcesDirectory) -Verbosity quiet -ExcludeVersion 
$choco = "$(Build.SourcesDirectory)\packages\chocolatey\tools\chocolateyInstall\choco.exe"

mkdir $(Build.ArtifactStagingDirectory)\Nightly\Choco

& $choco pack .\PackageExplorer\NuGetPackageExplorer.nuspec --version $nugetVer --OutputDirectory $(Build.ArtifactStagingDirectory)\Nightly\Choco
msbuild /t:pack .\Types\Types.csproj /p:Configuration=$(BuildConfiguration) /p:PackageOutputPath=$(Build.ArtifactStagingDirectory)\Nightly\NuGet

You can find the full build definition here.

Release

The release definition for NPE is more complicated than the previous example because it contains more environments: CI, Prod - Store, Prod - Chocolatey, Prod - NuGet, and Prod - GitHub. Most of the time releases go out to all production environments, but if there’s a fix that’s applicable to a specific environment, it only goes out to that one. All fixes go to CI first.

Release Management Pipeline for NPE

The triggers for the release and environments are the same as the previous example, so I won’t repeat the pictures. The steps vary per environment, performing the steps needed to take the artifacts and copy to Azure Blob, upload to the Microsoft Store, push to NuGet or Chocolatey, or create a GitHub release with artifacts, as the case may be.

Conclusion

For me, Release Management is a huge differentiator and fits into my way of thinking very well. I like the separation of responsibilities between build and release it provides. Now that public projects are available in VSTS, contributors to the projects can get build feedback and the community can check-in on the deployment status.

I’d love to hear feedback or suggestions and getting me on Twitter is usually the fastest way.