VSTS

OSS Build and Release with VSTS

May 15, 2018 Coding 1 comment , ,

OSS Build and Release with VSTS

Over the past few weeks I have been moving the build system for the OSS projects I maintain over to use VSTS. Up until recently I was using AppVeyor for builds, as they have provided a generous free offering for OSS for years. A huge thank you goes out to them for their past and ongoing support for OSS. So why move to VSTS? There’s three reasons for me:

  1. Support for public projects. This is key since there’s no point in using their builds if users can’t see the results.
  2. Release Management. The existing build systems like AppVeyor, Jenkins, TeamCity, and Travis, can all build a project. Sure, they have different strengths and weaknesses, and some offer free OSS builds as well, but none of them really has a Release Management story. That is, they can build artifacts….but then what? How do the bits get where you want, like NuGet, MyGet, a Store, etc. This is where release management fits in as a central part of CI/CD. More on this later.
  3. Windows, Linux, and Mac build host support in one system. It’s possible to run a single build on all three at the same time (fan out/in), like how VS Code does. No other host can do this easily today using a hosted build pool. I should note that using a hosted build pool is critical for security if you want to build pull requests from public forks. You don’t want to be running arbitrary code on a private build agent. Hosted agent VMs are destroyed after each use, making them far safer.

Life without Release Management

Many projects strive to achieve a continuous deployment pipeline without using Release Management (RM from now on). This is often achieved by using the build script or configuration to do some deployment steps given certain conditions, like if building a certain branch like master. In some ways, the GitFlow branching strategy encourages this, making it easy to decide that builds from the develop branch are pre-release and should go to a dev environment, while builds from master are production and should thus be deployed to a production environment. To me, this is conflating the real purpose of branches, which should be isolation of code churn, from deployment target. I believe any artifact should be able to be deployed to any environment, releasing is a different process than build and should have no bearing on which branch it comes from. For the vast majority of projects, I believe that a GitHub Flow or Release Flow (video) is a better, simpler, option.

Without RM, a pipeline to deploy a library might look something like this:

  1. Builds on the develop branch get deployed to MyGet by the build system. To me, it doesn’t much matter if it’s in the build script directly or if it’s build server configuration (like the deployment option in AppVeyor).
  2. To create a stable release, code is merged to master and then tagged. Often, tags are the mechanism that determines if there should be a release — effectively, tag a commit, that triggers a build which gets released to NuGet.org.

In this model, it’s usually a different build that gets deployed to release than to dev. I believe that mixing build and release like this ultimately leads to less flexibility and more coupling. The source system has to know about the deployment targets. If you need to change the deployment target, or add another one, you have to commit to the source and rebuild.

Following the single responsibility principle, a build should produce artifacts, that’s it. Deployment is something else, don’t conflate the two concepts. Repeat after me: Build is just build. I think projects have tended to mix the two, in part because there wasn’t a good, free, RM tool. It was easy and pragmatic to do both from the build tool. That changes now with VSTS public projects.

CD Nirvana with Release Management

VSTS has a full featured RM tool (deep dive on docs here) that is part of the platform. It is explicitly designed around the concept of artifacts, environments, and releases. In short, a build that contains artifacts can trigger a release. A release defines one or more environments with specific steps that should execute for each. Environments can be chained after another one, enabling a deployment promotion flow. There are many ways to gate each environment, automated and manual. A configuration I use frequently is to have two environments: MyGet and NuGet (dev and prod, respectively). The NuGet environment has an approval step so that releases don’t automatically flow from dev to prod; rather, I can decide to release to production at any time. Any build is a potential release.

Release steps can do anything and there are many existing tasks built-in for common things (like NuGet push, Azure blob copy, Azure App Services, and Docker) and a rich marketplace for things that aren’t (like creating a GitHub release, tagging the commit, and uploading artifacts to the GitHub release). In addition, you can run any custom script.

I think it’s easier to show by example, and that’s what follows in the next sections.

Versioning

Having an automatic version baked into your build artifacts is a crucial element. I use Andrew Arnott‘s Nerdbank.GitVersioning package to handle that for me. I set the Major.Minor in a version.json file and it increments the Patch based on the Git commit height since the last minor change. Add a prerelease tag to the version, if desired. You can control where the git height goes if you don’t want it in the patch (like 1.2.0-build.{height}). The default is the patch, and I think it’s completely okay to have a release like 1.2.42 if there were 42 commits since the version bump. I believe too much time is wasted on “clean” versions; it’s just a number :). Nerdbank.GitVersioning can also can set the build number in the agent, which is really handy knowing what version was just built.

Structuring your branches without overkill

There are many theories around how to structure your branches in Git. I tend to go with simplicity, aiming for a protected master branch with topic/feature branches for work. In my view, the sole reason for branches should be around code churn and isolation.

When it comes to delivery, there are two main schools of thought: releases and continuous. Releases are the traditional way shipping software. A group of features is batched together and shipped out once someone decides “it’s ready.” Continuous Delivery (CD) takes the thought out of releases: every build gets deployed. Note that doesn’t mean every build gets deployed to all environments, but every build is treated as if it could be.

I bring this up because I choose different tagging/branching strategies based on whether I’m doing releases or full CD.

If you’re doing continuous delivery, I would suggest using a single master branch with a stable version in it. Every build triggers a release, at least to a CI environment. At some point, could be every release, a set schedule, etc, that build gets promoted to the production environment. The key here is that it’s a promotion process; builds are fixed and flow through the environments.

If you’re doing release-based delivery, I would suggest using the master with a prerelease version tagged in it (like 1.2-preview). When you’re ready to stabilize your release, cut a rel/1.2 branch for it. In that branch, remove the prerelease tag and continue your stabilization process. Fixes should target master via a PR and then cherry-picked to the release branch if applicable. The release branch never merges back to master in this model.

In my view, using rel/* is perfect for stabilization of a release, enabling master to proceed to the next release. I’ll come back to my earlier point about branches: they should be for isolation of code churn, not environments. A rel branch isn’t always necessary; I’d only create it if there is parallel development happening.

Examples

I have two examples that illustrate how I implement the strategy above.

First, a library author creating a package that gets deployed to two feeds: a CI feed and a stable feed. The tree uses a preview prerelease tag in master and branches underneath rel for a stable release build. My example shows a .NET library with MyGet and NuGet, but the concepts apply to anything.

Second, I have an application that does continuous deployment to an automatically updating CI feed and controlled releases to the Microsoft Store, Chocolatey, NuGet, and GitHub. All releases move forward in master, and will hotfix under rel only if necessary.

Basic Library

Build

For this first scenario, I’ll talk about Rx.NET. It has a a build definition defined in yaml that has these essential parts (non-relevant parts omitted for brevity):

trigger:
- master
- rel/*

queue: Hosted VS2017

variables: 
  BuildConfiguration: Release
  BuildPlatform: Any CPU

steps:
- task: BatchScript@1
  inputs:
    filename: "C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Enterprise\\Common7\\Tools\\VsDevCmd.bat"
    arguments: -no_logo
    modifyEnvironment: true
  displayName: Setup Environment Variables

- task: PowerShell@1
  inputs:
    scriptName: 'Rx.NET/Source/build-new.ps1'
    workingFolder: 'Rx.NET/Source'
  env:
    VSTS_ACCESS_TOKEN: $(System.AccessToken)
  displayName: Build

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: 'Rx.NET/Source/artifacts'
    ArtifactName: artifacts
    publishLocation: Container
  condition: always()

I’m not going to dive too deep in the YAML itself; instead I’ll call your attention to the documentation and examples. As of now, there may be more up-to-date docs in their GitHub repo.

In this case, I have a build script (build-new.ps1) that calls msbuild and does all of the work. In order to ensure the right things are in the path, I call out to the VsDevCmd.bat first. xUnit, as of 2.4.0 beta 2, has direct support for publishing test results to VSTS if you supply the VSTS_ACCESS_TOKEN variable. Other frameworks are supported by the VSTest task. After running the main build script, I use a task to publish the binaries (NuGet packages) that were generated ty the build.

Another approach is to use the tasks for all of this directly, similar to this. We all have preferences between using scripts like PowerShell, Cake, PSake, etc, and the tasks. Doesn’t matter what you pick, use what works for you.

Release

The previous section was about build. The end result is a versioned set of artifacts that can be used as input to a release process. Rx.NET has a release definition here:

Release Management Pipeline

One tip on release naming that’s easily overlooked: it can be customized. I like to put the build number in it so I can associate a release with a version, and I concat it with the instance number (since it’s possible to have multiple releases for a particular version). In the definition options, I use the string Release v$(Build.BuildNumber).$(rev:r). That uses the build number from the primary artifact as the name.

The release defines two environments, MyGet and NuGet with an auto release trigger for builds on the master or rel/* branches. RM lets you put branch filters at any point, so you can enforce that releases only come from a specified branch, if desired. In this case, I tell it to create a release after builds from those branches. Then, in the MyGet environment, I’ve configured it to deploy to that environment automatically upon a release creation. That gets me Build -> Release -> MyGet in a CD pipeline. I do want to control releases to NuGet in two ways: 1) I want to ensure they are in MyGet, and 2) I want to manually approve it. I don’t want every build to go to NuGet. I have configured the NuGet environment to do just that, as well as only allowing the latest release to be deployed (I’m not looking to deploy older releases after-the-fact).

The MyGet environment has one step: a NuGet push to a configured endpoint. The NuGet environment has two steps: create a GitHub release (which will tag the commit for me), and a NuGet push. Releases don’t have to be complicated to benefit from using an RM flow. My release process is simple: when it’s time for a release, I take the selected build and push the “approve” button on the NuGet environment. There’s many other ways to gate releases to environments and you can do almost anything by calling out to an Azure Function as a gate.

It’s a bit hard to see how the release pipelines are configured on the site, so here are some screenshots showing the configuration:

Deployment Trigger:
Deployment Trigger

MyGet Environment:
MyGet Environment

NuGet Environment:
NuGet Environment

Environment Triggers for NuGet
Approvers
Queue

The actual release process to NuGet goes like this:

  • If I want to release a prerelease package, I can just press the approve button. It’ll do the rest.
  • If I want to release a stable package, I create a branch called rel/4.0 (for example) and make one edit to the version.json to remove the prerelease tag. That branch will never merge back to master. I can do as much stabilization in that branch as needed, and when I’m ready, I can approve that release to the NuGet environment. If there are hotfix releases I need to make, I will always make the changes to master (via a PR), then cherry-pick to the rel branch. This ensures that the next release always contains all of the fixes.

A Desktop Application

NuGet Package Explorer (NPE) is a WPF desktop application that is released to the Microsoft Store, Chocolatey, and GitHub as a zip. It also has a CI feed that auto-updates by using AppInstaller. NPE is delivered via a full CD process. There aren’t any prerelease versions; every build is a potential release and goes through an environment promotion pipeline.

Build

As a Desktop Bridge application, it contains a manifest file that must be updated with a version. Likewise, the Chocolatey package must be versioned as well. While there may be better options, I’m currently using a PowerShell script at build to replace a fixed version with one generated from Nerdbank.GitVersioning. I also update a build badge for use as an deployment artifact later.

# version    
nuget install NerdBank.GitVersioning -SolutionDir $(Build.SourcesDirectory) -Verbosity quiet -ExcludeVersion

$vers = & $(Build.SourcesDirectory)\packages\nerdbank.gitversioning\tools\Get-Version.ps1
$ver = $vers.SimpleVersion

# Update appxmanifests. These must be done before build.
$doc = Get-Content ".\PackageExplorer.Package\package.appxmanifest"    
$doc | % { $_.Replace("3.25.0", "$ver") } | Set-Content ".\PackageExplorer.Package\package.appxmanifest"

$doc = Get-Content ".\PackageExplorer.Package.Nightly\package.appxmanifest"    
$doc | % { $_.Replace("3.25.0", "$ver") } | Set-Content ".\PackageExplorer.Package.Nightly\package.appxmanifest"

$doc = Get-Content ".\Build\PackageExplorer.Package.Nightly.appinstaller"    
$doc | % { $_.Replace("3.25.0", "$ver") } | Set-Content "$(Build.ArtifactStagingDirectory)\Nightly\PackageExplorer.Package.Nightly.appinstaller"

# Build PackageExplorer
msbuild .\PackageExplorer\NuGetPackageExplorer.csproj /m /p:Configuration=$(BuildConfiguration) /bl:$(Build.ArtifactStagingDirectory)\Logs\Build-PackageExplorer.binlog
msbuild .\PackageExplorer.Package.Nightly\PackageExplorer.Package.Nightly.wapproj /m /p:Configuration=$(BuildConfiguration) /p:AppxPackageDir="$(Build.ArtifactStagingDirectory)\Nightly\" /bl:$(Build.ArtifactStagingDirectory)\Logs\Build-NightlyPackage.binlog
msbuild .\PackageExplorer.Package\PackageExplorer.Package.wapproj /m /p:Configuration=$(BuildConfiguration) /p:AppxPackageDir="$(Build.ArtifactStagingDirectory)\Store\" /p:UapAppxPackageBuildMode=StoreUpload /bl:$(Build.ArtifactStagingDirectory)\Logs\Build-Package.binlog

# Update versions
$doc = Get-Content ".\Build\ci_badge.svg"    
$doc | % { $_.Replace("ver_number", "$ver.0") } | Set-Content "$(Build.ArtifactStagingDirectory)\Nightly\version_badge.svg"

$doc = Get-Content ".\Build\store_badge.svg"    
$doc | % { $_.Replace("ver_number", "$ver.0") } | Set-Content "$(Build.ArtifactStagingDirectory)\Store\version_badge.svg"

# Choco and NuGet 
# Get choco

$nugetVer = $vers.NuGetPackageVersion

nuget install chocolatey -SolutionDir $(Build.SourcesDirectory) -Verbosity quiet -ExcludeVersion 
$choco = "$(Build.SourcesDirectory)\packages\chocolatey\tools\chocolateyInstall\choco.exe"

mkdir $(Build.ArtifactStagingDirectory)\Nightly\Choco

& $choco pack .\PackageExplorer\NuGetPackageExplorer.nuspec --version $nugetVer --OutputDirectory $(Build.ArtifactStagingDirectory)\Nightly\Choco
msbuild /t:pack .\Types\Types.csproj /p:Configuration=$(BuildConfiguration) /p:PackageOutputPath=$(Build.ArtifactStagingDirectory)\Nightly\NuGet

You can find the full build definition here.

Release

The release definition for NPE is more complicated than the previous example because it contains more environments: CI, Prod - Store, Prod - Chocolatey, Prod - NuGet, and Prod - GitHub. Most of the time releases go out to all production environments, but if there’s a fix that’s applicable to a specific environment, it only goes out to that one. All fixes go to CI first.

Release Management Pipeline for NPE

The triggers for the release and environments are the same as the previous example, so I won’t repeat the pictures. The steps vary per environment, performing the steps needed to take the artifacts and copy to Azure Blob, upload to the Microsoft Store, push to NuGet or Chocolatey, or create a GitHub release with artifacts, as the case may be.

Conclusion

For me, Release Management is a huge differentiator and fits into my way of thinking very well. I like the separation of responsibilities between build and release it provides. Now that public projects are available in VSTS, contributors to the projects can get build feedback and the community can check-in on the deployment status.

I’d love to hear feedback or suggestions and getting me on Twitter is usually the fastest way.

Continuous Deployment of Cloud Services with VSTS

October 18, 2017 Coding 4 comments , , ,

Continuous Deployment of Cloud Services with VSTS

In my last blog post, I showed how you can use ASP.NET Core with an Azure Cloud Service Web Role. The next step is to enable CI/CD for it, since you really shouldn’t be using “Publish” within Visual Studio for deployment.

As part of this, I wanted to configure the Cloud Service settings per environment in VSTS and not have any configuration checked-in to source control. Cloud Services’ configuration mechanism makes this a bit challenging due to the way it stores configuration, but with a few extra steps, it’s possible to make it work.

What you’ll need

To follow along, you’ll need the following:

  • Cloud Service the code can live in GitHub, VSTS, or many other locations. VSTS can build from any of them.
  • Azure Key Vault we’ll use Azure Key Vault to store the secrets. Creating a Key Vault is easy and the standard tier will work.
  • VSTS this guide is using Visual Studio Team Services, so you’ll need an account there. Those are free for up to five users and any number of users with MSDN licenses.

What we’re going to do

The gist here is that we’ll create a build definition that publishes the output of the Cloud Service project as an artifact. Then, we’ll create a release management process that takes the output of the build and deploys it to the cloud service in Azure. To handle the configuration, we’ll tokenize the checked-in configuration, then use a release management task to read configuration values stored in Key Vault and replace the matching tokenized values before the Azure deployment.

Moving the configuration into Key Vault

Create a new Key Vault to hold your configuration. You should have one Key Vault per environment that you intend to release to, since the secret names will directly translate to variables within VSTS. For each setting you need, create a secret with name like CustomSetting-Setting1 or CustomSetting-Setting2 and set their values. Next, in your ServiceConfiguration.Cloud.cscfg, set the values to be __CustomSetting-Setting1__ and __CustomSetting-Setting2__. The __ is the token start/end, and the value identifies which VSTS variable should be used to replace it.

One tip: If you have Password Encryption certificates or SSL endpoints configured, the .cscfg will have the certificates’ SHA-1 thumbprint’s encoded in them. If you want to configure this per environment, then replace those with token values. The configuration checker will enforce that it looks like a thumbprint, so use values like:

  • ABCDEF01234567ABCDEF01234567ABCDEF012345
  • BACDEF01234567ABCDEF01234567ABCDEF012345

Those sentinel values will be replaced with tokens during the build process and those tokens can be replaced with variable values.

We’ll use these in the build task later on.

The build definition

  1. Start with a new Empty build definition.
  2. On the process tab, choose the Hosted VS2017 Agent queue and give your build definition a name.
  3. Select Get Sources and point to your repository. This could be VSTS, GitHub or virtually any other location.
  4. Add the tasks we’ll need: Visual Studio Build (three times), Publish Build Artifacts (once). It should look something like this:
  5. For the first Visual Studio Build task, set the following values:
    SettingValue
    Display nameRestore solution
    SolutionAspNetCoreCloudService.sln
    Visual Studio VersionVisual Studio 2017
    MSBuild Arguments/t:restore
    Platform$(BuildPlatform)
    Configuration$(BuildConfiguration)
  6. For the second Visual Studio Build task, use the following values:

    SettingValue
    Display nameBuild solution
    SolutionAspNetCoreCloudService.sln
    Visual Studio VersionVisual Studio 2017
    MSBuild Arguments
    Platform$(BuildPlatform)
    Configuration$(BuildConfiguration)
  7. And the third Visual Studio Build task should be set as:

    SettingValue
    Display namePublish Cloud Service
    SolutionTheCloudService\TheCloudService.ccproj
    Visual Studio VersionVisual Studio 2017
    MSBuild Arguments/t:Publish /p:OutputPath=$(Build.ArtifactStagingDirectory)\
    Platform$(BuildPlatform)
    Configuration$(BuildConfiguration)
  8. If you are using sentinel certificate values, add a PowerShell Task. Configure the PowerShell task by selecting “Inline Script”, expand Advanced and set the working folder to the publish directory (like $(Build.ArtifactStagingDirectory)\app.publish) and use the following script:

    $file = "ServiceConfiguration.Cloud.cscfg"
    # Read file
    $content = Get-Content -Path $file
    # substitute values
    $content = $content.Replace("ABCDEF01234567ABCDEF01234567ABCDEF012345", "__SslCertificateSha1__")
    $content = $content.Replace("BACDEF01234567ABCDEF01234567ABCDEF012345", "__PasswordEncryption__")
    # Save
    [System.IO.File]::WriteAllText($file, $content)
    

    This replaces the fake SHA-1 thumbprints with tokens that release management will use. Be sure to define variables in release management that match the names you use.

  9. Finally, set the Publish Artifact step to:

    SettingValue
    Display namePublish Artifact: Cloud Service
    Path to Publish$(Build.ArtifactStagingDirectory)\app.publish
    Artifact NameTheCloudService
    Artifact TypeServer
  10. Go to the Variables tab and add two variables:

    NameValue
    BuildConfigurationRelease
    BuildPlatformAny CPU
  11. Hit Save & Queue to save the definition and start a new build. It should complete successfully. If you go to the build artifacts folder, you should see TheCloudService with the .cspkg file in it.

Deploying the build to Azure

This release process depends on one external extension that handles the tokenization, the Release Management Utility Tasks. Install it from the marketplace into your VSTS account before starting this section.

  1. In VSTS, switch to the Releases tab and create a new release definition using the “Azure Cloud Service Deployment” template.
  2. Give the environment a name, like “Cloud Service – Prod”.
  3. Click the “Add artifact” box and select your build definition. Should look something like this:

    If you want continuous deployment, click the “lightning bolt” icon and enable the CD trigger.
  4. Click on the Tasks tab and specify an Azure subscription, storage account, service name and location. If you need to link your existing Azure subscription, click the “Manage” link. If you need a new storage account to hold the deployment artifacts, you can create that in the portal as well, just make sure to create a “Classic” storage account.
  5. Go to the Variables tab and select “Variable groups”, then “Manage variable groups.” Add a new variable group, give it a name like “AspNetCloudService Production Configuration”, select your subscription (click Manage to link one), and select the Key Vault we created earlier to hold the config. Press the Authorize button if prompted.

    Finally, click Add to select which secrets from Key Vault should be added to this variable group.

    It’s important to note that it does not copy the values at this point. The secret’s values are always read on use, so they’re always current. Save the variable group and return back to the Release Management definition. At this point, you can select “Link variable group” and link the one we just created.
  6. Add a Tokenize with XPath/Regular Expressions task before the Azure Deployment task.
  7. In the Tokenizer task, browse to the ServiceConfiguration.Cloud.cscfg file, something like $(System.DefaultWorkingDirectory)/AspNetCoreCloudService-CI/TheCloudService/ServiceConfiguration.Cloud.cscfg depending on what you call your artifacts.
  8. Ensure that the Azure Deployment task is last, and you should be all set.
  9. Create a new release and it should deploy successfully. If you view your cloud service configuration on Azure Portal, you should see the real values, not the __Tokenized__ values.

Summary

That’s it, you now have an ASP.NET Core Cloud Service deployed to Azure with CI/CD through VSTS. If you want to add additional environments, simply add an additional key vault and linked variable group for each environment, clone the existing environment configuration in the Release Management editor and set the appropriate environmental values. Variable groups are defined at the release definition level, so for multiple-environments you can use a suffix in your variable names and then update the PowerShell script in step 7 to append that per environment (__MyVariable-Prod__), etc.