Month: October 2017

Continuous Deployment of Cloud Services with VSTS

October 18, 2017 Coding 4 comments , , ,

Continuous Deployment of Cloud Services with VSTS

In my last blog post, I showed how you can use ASP.NET Core with an Azure Cloud Service Web Role. The next step is to enable CI/CD for it, since you really shouldn’t be using “Publish” within Visual Studio for deployment.

As part of this, I wanted to configure the Cloud Service settings per environment in VSTS and not have any configuration checked-in to source control. Cloud Services’ configuration mechanism makes this a bit challenging due to the way it stores configuration, but with a few extra steps, it’s possible to make it work.

What you’ll need

To follow along, you’ll need the following:

  • Cloud Service the code can live in GitHub, VSTS, or many other locations. VSTS can build from any of them.
  • Azure Key Vault we’ll use Azure Key Vault to store the secrets. Creating a Key Vault is easy and the standard tier will work.
  • VSTS this guide is using Visual Studio Team Services, so you’ll need an account there. Those are free for up to five users and any number of users with MSDN licenses.

What we’re going to do

The gist here is that we’ll create a build definition that publishes the output of the Cloud Service project as an artifact. Then, we’ll create a release management process that takes the output of the build and deploys it to the cloud service in Azure. To handle the configuration, we’ll tokenize the checked-in configuration, then use a release management task to read configuration values stored in Key Vault and replace the matching tokenized values before the Azure deployment.

Moving the configuration into Key Vault

Create a new Key Vault to hold your configuration. You should have one Key Vault per environment that you intend to release to, since the secret names will directly translate to variables within VSTS. For each setting you need, create a secret with name like CustomSetting-Setting1 or CustomSetting-Setting2 and set their values. Next, in your ServiceConfiguration.Cloud.cscfg, set the values to be __CustomSetting-Setting1__ and __CustomSetting-Setting2__. The __ is the token start/end, and the value identifies which VSTS variable should be used to replace it.

One tip: If you have Password Encryption certificates or SSL endpoints configured, the .cscfg will have the certificates’ SHA-1 thumbprint’s encoded in them. If you want to configure this per environment, then replace those with token values. The configuration checker will enforce that it looks like a thumbprint, so use values like:

  • ABCDEF01234567ABCDEF01234567ABCDEF012345
  • BACDEF01234567ABCDEF01234567ABCDEF012345

Those sentinel values will be replaced with tokens during the build process and those tokens can be replaced with variable values.

We’ll use these in the build task later on.

The build definition

  1. Start with a new Empty build definition.
  2. On the process tab, choose the Hosted VS2017 Agent queue and give your build definition a name.
  3. Select Get Sources and point to your repository. This could be VSTS, GitHub or virtually any other location.
  4. Add the tasks we’ll need: Visual Studio Build (three times), Publish Build Artifacts (once). It should look something like this:
  5. For the first Visual Studio Build task, set the following values:
    SettingValue
    Display nameRestore solution
    SolutionAspNetCoreCloudService.sln
    Visual Studio VersionVisual Studio 2017
    MSBuild Arguments/t:restore
    Platform$(BuildPlatform)
    Configuration$(BuildConfiguration)
  6. For the second Visual Studio Build task, use the following values:

    SettingValue
    Display nameBuild solution
    SolutionAspNetCoreCloudService.sln
    Visual Studio VersionVisual Studio 2017
    MSBuild Arguments
    Platform$(BuildPlatform)
    Configuration$(BuildConfiguration)
  7. And the third Visual Studio Build task should be set as:

    SettingValue
    Display namePublish Cloud Service
    SolutionTheCloudService\TheCloudService.ccproj
    Visual Studio VersionVisual Studio 2017
    MSBuild Arguments/t:Publish /p:OutputPath=$(Build.ArtifactStagingDirectory)\
    Platform$(BuildPlatform)
    Configuration$(BuildConfiguration)
  8. If you are using sentinel certificate values, add a PowerShell Task. Configure the PowerShell task by selecting “Inline Script”, expand Advanced and set the working folder to the publish directory (like $(Build.ArtifactStagingDirectory)\app.publish) and use the following script:

    $file = "ServiceConfiguration.Cloud.cscfg"
    # Read file
    $content = Get-Content -Path $file
    # substitute values
    $content = $content.Replace("ABCDEF01234567ABCDEF01234567ABCDEF012345", "__SslCertificateSha1__")
    $content = $content.Replace("BACDEF01234567ABCDEF01234567ABCDEF012345", "__PasswordEncryption__")
    # Save
    [System.IO.File]::WriteAllText($file, $content)
    

    This replaces the fake SHA-1 thumbprints with tokens that release management will use. Be sure to define variables in release management that match the names you use.

  9. Finally, set the Publish Artifact step to:

    SettingValue
    Display namePublish Artifact: Cloud Service
    Path to Publish$(Build.ArtifactStagingDirectory)\app.publish
    Artifact NameTheCloudService
    Artifact TypeServer
  10. Go to the Variables tab and add two variables:

    NameValue
    BuildConfigurationRelease
    BuildPlatformAny CPU
  11. Hit Save & Queue to save the definition and start a new build. It should complete successfully. If you go to the build artifacts folder, you should see TheCloudService with the .cspkg file in it.

Deploying the build to Azure

This release process depends on one external extension that handles the tokenization, the Release Management Utility Tasks. Install it from the marketplace into your VSTS account before starting this section.

  1. In VSTS, switch to the Releases tab and create a new release definition using the “Azure Cloud Service Deployment” template.
  2. Give the environment a name, like “Cloud Service – Prod”.
  3. Click the “Add artifact” box and select your build definition. Should look something like this:

    If you want continuous deployment, click the “lightning bolt” icon and enable the CD trigger.
  4. Click on the Tasks tab and specify an Azure subscription, storage account, service name and location. If you need to link your existing Azure subscription, click the “Manage” link. If you need a new storage account to hold the deployment artifacts, you can create that in the portal as well, just make sure to create a “Classic” storage account.
  5. Go to the Variables tab and select “Variable groups”, then “Manage variable groups.” Add a new variable group, give it a name like “AspNetCloudService Production Configuration”, select your subscription (click Manage to link one), and select the Key Vault we created earlier to hold the config. Press the Authorize button if prompted.

    Finally, click Add to select which secrets from Key Vault should be added to this variable group.

    It’s important to note that it does not copy the values at this point. The secret’s values are always read on use, so they’re always current. Save the variable group and return back to the Release Management definition. At this point, you can select “Link variable group” and link the one we just created.
  6. Add a Tokenize with XPath/Regular Expressions task before the Azure Deployment task.
  7. In the Tokenizer task, browse to the ServiceConfiguration.Cloud.cscfg file, something like $(System.DefaultWorkingDirectory)/AspNetCoreCloudService-CI/TheCloudService/ServiceConfiguration.Cloud.cscfg depending on what you call your artifacts.
  8. Ensure that the Azure Deployment task is last, and you should be all set.
  9. Create a new release and it should deploy successfully. If you view your cloud service configuration on Azure Portal, you should see the real values, not the __Tokenized__ values.

Summary

That’s it, you now have an ASP.NET Core Cloud Service deployed to Azure with CI/CD through VSTS. If you want to add additional environments, simply add an additional key vault and linked variable group for each environment, clone the existing environment configuration in the Release Management editor and set the appropriate environmental values. Variable groups are defined at the release definition level, so for multiple-environments you can use a suffix in your variable names and then update the PowerShell script in step 7 to append that per environment (__MyVariable-Prod__), etc.

Using ASP.NET Core with Azure Cloud Services

October 16, 2017 Coding 3 comments , ,

Using ASP.NET Core with Azure Cloud Services

Overview

Cloud Services may be the old-timer of Azure’s offerings, but there are still some cases where it is useful. For example, today, it is the only available PaaS way to run a Windows Server 2016 workload in Azure. Sure, you can run a Windows Container with Azure Container Services, but that’s not really PaaS to me. You still have to be fully aware of Kubernetes, DC/OS, or Swarm, and, as with any container, you are responsible for patching the underlying OS image with security updates.

In developing my Code Signing Service, I stumbled upon a hard dependency on Server 2016. The API I needed to Authenticode sign a file using Azure Key Vault’s signing methods only exists in that version of Windows. That meant that using Azure App Services was out, as it uses Server 2012 (based on the version numbers from its command line). That left Cloud Service Web Roles as the sole remaining option if I wanted PaaS. I could have also used a B-Series VM, that’s perfect for this type of workload, but I really don’t want to maintain a VM.

If you have tried to use ASP.NET Core with a Cloud Service Web Role, you’ll probably have come away disappointed as Visual Studio doesn’t let you do this…. until now. Never one to accept no for an answer, I found a way to make this work, and with a few workarounds, you can too.

The solution presented here handles deployment of an MVC & API application that along with config settings and deployment of the ASP.NET Core Windows Hosting Module. VS Cloud Service tooling works for making changes to config and publishing to cloud services (though please use CI/CD in VSTS!)

Many thanks to Scott Hunter‘s team, Jaques Eloff and Catherine Wang in particular, on figuring out a workaround for some gotcha’s when installing the Windows Hosting Module.

Pieces to the puzzle

You can see the sample solution here, and it may be helpful to clone and follow along in VS.

There are a few pieces to making this work:

  1. TheWebsite The ASP.NET Core MVC site. Nothing significantly special here, just an ordinary site.
  2. TheCloudService The Cloud Service project. Contains the configuration files and service definition.
  3. TheWebRole ASP.NET 4.6 project that contains the Web Role startup scripts and “references” the TheWebsite site. This is where the tricks are.

At a high level, the Cloud Service “sees” TheWebRole as the configured website. The cloud service doesn’t know anything about ASP.NET Core. The trick is to get the ASP.NET Core site published and running “in” an ASP.NET site.

Doing this yourself

The Projects

In a new solution, create a new ASP.NET Core 2 project. Doesn’t really matter what template you use. For the descriptions here, I’ll call it TheWebsite. Build and run the site, it should debug and run normally in IISExpress.

Next, create a new Cloud Service (File -> Add -> New Project -> Cloud -> Azure Cloud Service). I’ll call the cloud service TheCloudService, and on the next dialog, add a single Web Site. I called mine TheWebRole.

Finally, on the ASP.NET Template selection, choose “Empty” and continue.

Right now, we have an ASP.NET Core Website and an Azure Cloud Service with a single ASP.NET 4.6 WebRole. Next up is to clear out almost everything from TheWebRole since it won’t actually contain any ASP.NET Code. Delete the packages.config and Web.config files.

Save the project, then select “Unload” from the project’s context menu. Right-click again and select “Edit TheWebRole.csproj”. We need to delete the packages brought in by NuGet along with the imported props and target. There are three areas to delete as noted in the screen shots: Props at the top, all Reference elements with a HintPath pointing to ..\packages\ and the Target at the bottom.



At this point, your project file should look similar to this here. You can also view the complete diff.

Magic

Now comes the special sauce — we need a way to have TheWebRole build TheWebsite and include TheWebsite‘s publish output as Content. Doing this ensures that TheCloudService Package contains the correct folder layout. Add the following snippet to the bottom of TheWebRole‘s project file to call Publish on our website before the main build step.

<Target Name="BeforeBuild">
  <MSBuild Projects="..\TheWebsite\TheWebsite.csproj" Targets="Publish" Properties="Configuration=$(Configuration)" />
</Target>

Then, add the following ItemGroup to include TheWebsite‘s publish output as Content in the TheWebRole project:

<ItemGroup>
  <Content Include="..\TheWebsite\bin\$(Configuration)\netcoreapp2.0\publish\**\*.*" Link="%(RecursiveDir)%(Filename)%(Extension)" />
</ItemGroup>

Save the csproj file, then right-click the TheWebRole and click Reload. You can test that the cloud service package is created correctly by right-clicking TheCloudService and selecting Package. After choosing a build configuration and hitting “Package,” the project should build and the output directory pop up.

The .cspkg is really a zip file, so extract it and you’ll see the guts of cloud service packages. Look for the .cssx file and extract that (again, just a zip file)

Inside there, open the approot folder and that is the root of your website. If the previous steps were done correctly, you should see something like the following

You should see TheWebsite.dll, TheWebsite.PrecompiledViews.dll, wwwroot, and the rest of your files from TheWebsite.

Congratulations, you’ve now created a cloud service that packages up and deploys an ASP.NET Core website! This alone won’t let the site run though since the Cloud Service images don’t include the Windows Hosting Module.

Installing .NET Core 2 onto the Web Role

Installing additional components onto a Web Role typically involves a startup script, and .NET Core 2 is no different. There is one complication though: the installer downloads files into the TEMP folder, and Cloud Services has a 100MB hard limit on that folder. We need to specify an alternate folder to use as TEMP with a higher quota (this is what Jaques and Catherine figured out).

In TheCloudService, expand Roles, right click TheWebRole and hit properties. Go to Local Storage and add a new location called CustomTempPath with a 500MB limit (or whatever else your app might need).

Next, we need the startup script. Go to TheWebRole, add a new folder called Startup and add the following files to it. Ensure that the Build Action is set to Content and that Copy to Output Directory is set to Copy if newer. Finally, we need to configure the cloud service to invoke our startup task. Open the ServiceDefinition.csdef file and add the following xml in the WebRole node to define the startup task:

<Startup>
  <Task commandLine="Startup\startup.cmd" executionContext="elevated" taskType="simple">
    <Environment>
    <Variable name="IsEmulated">
      <RoleInstanceValue xpath="/RoleEnvironment/Deployment/@emulated" />
    </Variable>
    </Environment>
  </Task>
</Startup>

Now we finally have a cloud service that can be deployed, install .NET Core, and run the website. The first time you publish, it will take a few minutes for the role instance to become available since it
has to install the hosting module and restart IIS.

Note: I leave creating a cloud service instance in the Azure Portal as an exercise to the reader

Configuration

There are many ways of getting configuration into an ASP.NET Core application. If you know you’ll only be running in Cloud Services, you may consider taking a direct dependency on the Cloud Services libraries and using the RoleEnvironment types to get populate your configuration. Alternatively, you can likely write a configuration provider that funnels in the RoleEnvironment configuration into the ASP.NET Core configuration system.

In my original case, I didn’t want my ASP.NET Core website to have any awareness of Cloud Services, so I came up with another way—in the startup script, I copy the values from the RoleEnvironment into environment variables that the default configuration settings pick up. The key here to making this transparent is knowing that the double-underscore, __, translates into the : when read from an environment variable. This means you can define a setting like CustomSetting__Setting1, and then you can access it with Configuration["CustomSetting:Setting1"], or similar mechanisms.

To bridge this gap, we can add this to the startup script (complete script):

$keys = @(
  "CustomSetting__Setting1",
  "CustomSetting__Setting2"
)

foreach($key in $keys){
  [Environment]::SetEnvironmentVariable($key, [Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment]::GetConfigurationSettingValue($key), "Machine")
}

This copies the settings from the Cloud Service Role Environment into environment variables on the host, and from there, the default ASP.NET Core configuration adds them into configuration.

Considerations

  • Session affinity If you need session affinity for session state, you’ll need to configure that.
  • Data Protection API Unlike Azure App Services, Cloud Services doesn’t have any default synchronization for the keys. You’ll need a solution for this. If anyone comes up with a reusable solution, I’ll happily mention it here. More info on configuring DPAPI is here.
  • Local Debugging Due to the way local debugging of cloud services works (it directly uses TheWebRole as a startup project in IIS Express), directly debugging the cloud service does not work with the current patterns. Instead, you can set TheWebsite as a startup project and debug that normally. The underlying issue is that TheWebRole includes TheWebsite as Content and does not copy the published files to TheWebRole‘s directory. It may be possible to achieve this, though you’d likely want additional .gitignore rules to prevent those files from being committed. In my case, I did not want my service to have any direct dependency on Cloud Services, so this wasn’t an issue—I simply needed a Server 2016 web host.

CI / CD with VSTS

It is possible to automate build/deploy of these cloud service web role projects using VSTS. My next blog post will show how to set that up.

Update October 18: The post is live