TL;DR: If you ever got this error on Azure App Services, read-on: The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.
This is admittedly an esoteric topic, but useful to know in case you happen to hit it. Azure App Services are wonderful; they make it really easy to host a website or API in Azure, with the platform handling the "hard parts" of HTTPS certificate management, scaling, load balancing, etc. A few months ago, while working on my code signing service, I hit a strange issue when running on Azure that didn't reproduce locally. MakeAppx.exe, a tool that expands and creates AppX packages, was failing. The code signing service bundles a few utility exe's that are executed for certain operations. In this case, I needed to extract Appx/Appxbundle files, modify the contents and repack them.
When I started investigating, I went to the Kudu console and tried executing makeappx.exe, I got the following error: The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.. No combinations of parameters made a difference, it wasn't even getting that far.
After looking at potential differences in that application vs every other application that worked, I noticed that makeappx.exe uses an embedded manifest that declares registration-free COM dependencies. This is because makeappx.exe includes local copies of some files may be newer than the ones included in the operating system. If I replaced the built-in manifest with a default one, I could launch the application, so that was definitely the issue. The caveat is two-fold: 1. I was modifying a Windows SDK binary, and I didn't want to do that, 2. I'd be tied to a particular version of that tool that matched whatever app services was running on.
Given that I was now in uncharted territory, I reached out to the App Service team and they graciously helped. In particular, I need to thank David Ebbo and Petr Podhorsky, without whom I'd still be stuck.
What the investigation found: due to the way App Service maps in the "D" drive (the default one your site runs from). As Petr explains:
Registration-free COM side-by-side is handled by a different process on the box, CSRSS, which runs under a different account and because the path is transferred between processes unchanged, it tries to look at d:\home path, which is a special path and makes sense only for your site process, but nothing else on the box (if there are multiple sites running on the box, they all have their own d:\home and don’t see each other’s content). So it does not find your manifest.
Even if the previous problem was solved, the CSRSS (client server runtime subsystem) process does not have access to the site content anyway.
Fortunately, there is a workaround: use the HOME_EXPANDED environment variable. That points to the real path of the "C drive" equivalent location, something like C:\DWASFiles\Sites\#1mysite1\home. Don't worry about the exact location, just use the variable.
Invoking makeappx.exe from the HOME_EXPANDED location works because CSRSS sees the files that the manifest points to and it loads properly. You can see how I applied the workaround here.
While it's likely rare to have a program using registration-free COM invoked on App Services, it's not entirely impossible. You might have a web job or function that needs to invoke some program, or you might need to invoke programs for some website functionality like I do. If you do hit this, I hope this workaround unblocks you too.
In my last blog post, I showed how you can use ASP.NET Core with an Azure Cloud Service Web Role. The next step is to enable CI/CD for it, since you really shouldn't be using "Publish" within Visual Studio for deployment.
As part of this, I wanted to configure the Cloud Service settings per environment in VSTS and not have any configuration checked-in to source control. Cloud Services' configuration mechanism makes this a bit challenging due to the way it stores configuration, but with a few extra steps, it's possible to make it work.
What you'll need
To follow along, you'll need the following:
Cloud Service the code can live in GitHub, VSTS, or many other locations. VSTS can build from any of them.
Azure Key Vault we'll use Azure Key Vault to store the secrets. Creating a Key Vault is easy and the standard tier will work.
VSTS this guide is using Visual Studio Team Services, so you'll need an account there. Those are free for up to five users and any number of users with MSDN licenses.
What we're going to do
The gist here is that we'll create a build definition that publishes the output of the Cloud Service project as an artifact. Then, we'll create a release management process that takes the output of the build and deploys it to the cloud service in Azure. To handle the configuration, we'll tokenize the checked-in configuration, then use a release management task to read configuration values stored in Key Vault and replace the matching tokenized values before the Azure deployment.
Moving the configuration into Key Vault
Create a new Key Vault to hold your configuration. You should have one Key Vault per environment that you intend to release to, since the secret names will directly translate to variables within VSTS. For each setting you need, create a secret with name like CustomSetting-Setting1 or CustomSetting-Setting2 and set their values. Next, in your ServiceConfiguration.Cloud.cscfg, set the values to be __CustomSetting-Setting1__ and __CustomSetting-Setting2__. The __ is the token start/end, and the value identifies which VSTS variable should be used to replace it.
One tip: If you have Password Encryption certificates or SSL endpoints configured, the .cscfg will have the certificates' SHA-1 thumbprint's encoded in them. If you want to configure this per environment, then replace those with token values. The configuration checker will enforce that it looks like a thumbprint, so use values like:
Those sentinel values will be replaced with tokens during the build process and those tokens can be replaced with variable values.
We'll use these in the build task later on.
The build definition
Start with a new Empty build definition.
On the process tab, choose the Hosted VS2017 Agent queue and give your build definition a name.
Select Get Sources and point to your repository. This could be VSTS, GitHub or virtually any other location.
Add the tasks we'll need: Visual Studio Build (three times), Publish Build Artifacts (once). It should look something like this:
For the first Visual Studio Build task, set the following values:
Visual Studio Version
Visual Studio 2017
For the second Visual Studio Build task, use the following values:
Visual Studio Version
Visual Studio 2017
And the third Visual Studio Build task should be set as:
If you are using sentinel certificate values, add a PowerShell Task. Configure the PowerShell task by selecting "Inline Script", expand Advanced and set the working folder to the publish directory (like $(Build.ArtifactStagingDirectory)\app.publish) and use the following script:
This replaces the fake SHA-1 thumbprints with tokens that release management will use. Be sure to define variables in release management that match the names you use.
Finally, set the Publish Artifact step to:
Publish Artifact: Cloud Service
Path to Publish
Go to the Variables tab and add two variables:
Hit Save & Queue to save the definition and start a new build. It should complete successfully. If you go to the build artifacts folder, you should see TheCloudService with the .cspkg file in it.
Deploying the build to Azure
This release process depends on one external extension that handles the tokenization, the Release Management Utility Tasks. Install it from the marketplace into your VSTS account before starting this section.
In VSTS, switch to the Releases tab and create a new release definition using the "Azure Cloud Service Deployment" template.
Give the environment a name, like "Cloud Service - Prod".
Click the "Add artifact" box and select your build definition. Should look something like this:
If you want continuous deployment, click the "lightning bolt" icon and enable the CD trigger.
Click on the Tasks tab and specify an Azure subscription, storage account, service name and location. If you need to link your existing Azure subscription, click the "Manage" link. If you need a new storage account to hold the deployment artifacts, you can create that in the portal as well, just make sure to create a "Classic" storage account.
Go to the Variables tab and select "Variable groups", then "Manage variable groups." Add a new variable group, give it a name like "AspNetCloudService Production Configuration", select your subscription (click Manage to link one), and select the Key Vault we created earlier to hold the config. Press the Authorize button if prompted.
Finally, click Add to select which secrets from Key Vault should be added to this variable group.
It's important to note that it does not copy the values at this point. The secret's values are always read on use, so they're always current. Save the variable group and return back to the Release Management definition. At this point, you can select "Link variable group" and link the one we just created.
Add a Tokenize with XPath/Regular Expressions task before the Azure Deployment task.
In the Tokenizer task, browse to the ServiceConfiguration.Cloud.cscfg file, something like $(System.DefaultWorkingDirectory)/AspNetCoreCloudService-CI/TheCloudService/ServiceConfiguration.Cloud.cscfg depending on what you call your artifacts.
Ensure that the Azure Deployment task is last, and you should be all set.
Create a new release and it should deploy successfully. If you view your cloud service configuration on Azure Portal, you should see the real values, not the __Tokenized__ values.
That's it, you now have an ASP.NET Core Cloud Service deployed to Azure with CI/CD through VSTS. If you want to add additional environments, simply add an additional key vault and linked variable group for each environment, clone the existing environment configuration in the Release Management editor and set the appropriate environmental values. Variable groups are defined at the release definition level, so for multiple-environments you can use a suffix in your variable names and then update the PowerShell script in step 7 to append that per environment (__MyVariable-Prod__), etc.
Cloud Services may be the old-timer of Azure's offerings, but there are still some cases where it is useful. For example, today, it is the only available PaaS way to run a Windows Server 2016 workload in Azure. Sure, you can run a Windows Container with Azure Container Services, but that's not really PaaS to me. You still have to be fully aware of Kubernetes, DC/OS, or Swarm, and, as with any container, you are responsible for patching the underlying OS image with security updates.
In developing my Code Signing Service, I stumbled upon a hard dependency on Server 2016. The API I needed to Authenticode sign a file using Azure Key Vault's signing methods only exists in that version of Windows. That meant that using Azure App Services was out, as it uses Server 2012 (based on the version numbers from its command line). That left Cloud Service Web Roles as the sole remaining option if I wanted PaaS. I could have also used a B-Series VM, that's perfect for this type of workload, but I really don't want to maintain a VM.
If you have tried to use ASP.NET Core with a Cloud Service Web Role, you'll probably have come away disappointed as Visual Studio doesn't let you do this.... until now. Never one to accept no for an answer, I found a way to make this work, and with a few workarounds, you can too.
The solution presented here handles deployment of an MVC & API application that along with config settings and deployment of the ASP.NET Core Windows Hosting Module. VS Cloud Service tooling works for making changes to config and publishing to cloud services (though please use CI/CD in VSTS!)
You can see the sample solution here, and it may be helpful to clone and follow along in VS.
There are a few pieces to making this work:
TheWebsite The ASP.NET Core MVC site. Nothing significantly special here, just an ordinary site.
TheCloudService The Cloud Service project. Contains the configuration files and service definition.
TheWebRole ASP.NET 4.6 project that contains the Web Role startup scripts and "references" the TheWebsite site. This is where the tricks are.
At a high level, the Cloud Service "sees" TheWebRole as the configured website. The cloud service doesn't know anything about ASP.NET Core. The trick is to get the ASP.NET Core site published and running "in" an ASP.NET site.
Doing this yourself
In a new solution, create a new ASP.NET Core 2 project. Doesn't really matter what template you use. For the descriptions here, I'll call it TheWebsite. Build and run the site, it should debug and run normally in IISExpress.
Next, create a new Cloud Service (File -> Add -> New Project -> Cloud -> Azure Cloud Service). I'll call the cloud service TheCloudService, and on the next dialog, add a single Web Site. I called mine TheWebRole.
Finally, on the ASP.NET Template selection, choose "Empty" and continue.
Right now, we have an ASP.NET Core Website and an Azure Cloud Service with a single ASP.NET 4.6 WebRole. Next up is to clear out almost everything from TheWebRole since it won't actually contain any ASP.NET Code. Delete the packages.config and Web.config files.
Save the project, then select "Unload" from the project's context menu. Right-click again and select "Edit TheWebRole.csproj". We need to delete the packages brought in by NuGet along with the imported props and target. There are three areas to delete as noted in the screen shots: Props at the top, all Reference elements with a HintPath pointing to ..\packages\ and the Target at the bottom.
At this point, your project file should look similar to this here. You can also view the complete diff.
Now comes the special sauce -- we need a way to have TheWebRole build TheWebsite and include TheWebsite's publish output as Content. Doing this ensures that TheCloudService Package contains the correct folder layout. Add the following snippet to the bottom of TheWebRole's project file to call Publish on our website before the main build step.
Save the csproj file, then right-click the TheWebRole and click Reload. You can test that the cloud service package is created correctly by right-clicking TheCloudService and selecting Package. After choosing a build configuration and hitting "Package," the project should build and the output directory pop up.
The .cspkg is really a zip file, so extract it and you'll see the guts of cloud service packages. Look for the .cssx file and extract that (again, just a zip file)
Inside there, open the approot folder and that is the root of your website. If the previous steps were done correctly, you should see something like the following
You should see TheWebsite.dll, TheWebsite.PrecompiledViews.dll, wwwroot, and the rest of your files from TheWebsite.
Congratulations, you've now created a cloud service that packages up and deploys an ASP.NET Core website! This alone won't let the site run though since the Cloud Service images don't include the Windows Hosting Module.
Installing .NET Core 2 onto the Web Role
Installing additional components onto a Web Role typically involves a startup script, and .NET Core 2 is no different. There is one complication though: the installer downloads files into the TEMP folder, and Cloud Services has a 100MB hard limit on that folder. We need to specify an alternate folder to use as TEMP with a higher quota (this is what Jaques and Catherine figured out).
In TheCloudService, expand Roles, right click TheWebRole and hit properties. Go to Local Storage and add a new location called CustomTempPath with a 500MB limit (or whatever else your app might need).
Next, we need the startup script. Go to TheWebRole, add a new folder called Startup and add the following files to it. Ensure that the Build Action is set to Content and that Copy to Output Directory is set to Copy if newer. Finally, we need to configure the cloud service to invoke our startup task. Open the ServiceDefinition.csdef file and add the following xml in the WebRole node to define the startup task:
Now we finally have a cloud service that can be deployed, install .NET Core, and run the website. The first time you publish, it will take a few minutes for the role instance to become available since it has to install the hosting module and restart IIS.
Note: I leave creating a cloud service instance in the Azure Portal as an exercise to the reader
There are many ways of getting configuration into an ASP.NET Core application. If you know you'll only be running in Cloud Services, you may consider taking a direct dependency on the Cloud Services libraries and using the RoleEnvironment types to get populate your configuration. Alternatively, you can likely write a configuration provider that funnels in the RoleEnvironment configuration into the ASP.NET Core configuration system.
In my original case, I didn't want my ASP.NET Core website to have any awareness of Cloud Services, so I came up with another way—in the startup script, I copy the values from the RoleEnvironment into environment variables that the default configuration settings pick up. The key here to making this transparent is knowing that the double-underscore, __, translates into the : when read from an environment variable. This means you can define a setting like CustomSetting__Setting1, and then you can access it with Configuration["CustomSetting:Setting1"], or similar mechanisms.
To bridge this gap, we can add this to the startup script (complete script):
This copies the settings from the Cloud Service Role Environment into environment variables on the host, and from there, the default ASP.NET Core configuration adds them into configuration.
Session affinity If you need session affinity for session state, you'll need to configure that.
Data Protection API Unlike Azure App Services, Cloud Services doesn't have any default synchronization for the keys. You'll need a solution for this. If anyone comes up with a reusable solution, I'll happily mention it here. More info on configuring DPAPI is here.
Local Debugging Due to the way local debugging of cloud services works (it directly uses TheWebRole as a startup project in IIS Express), directly debugging the cloud service does not work with the current patterns. Instead, you can set TheWebsite as a startup project and debug that normally. The underlying issue is that TheWebRole includes TheWebsite as Content and does not copy the published files to TheWebRole's directory. It may be possible to achieve this, though you'd likely want additional .gitignore rules to prevent those files from being committed. In my case, I did not want my service to have any direct dependency on Cloud Services, so this wasn't an issue—I simply needed a Server 2016 web host.
CI / CD with VSTS
It is possible to automate build/deploy of these cloud service web role projects using VSTS. My next blog post will show how to set that up.