In the previous post we have learned how, thanks to WinAppDriver and Appium, we can perform UI testing against our Windows applications. In that post we have created a sample WPF app and then we have tested some basic user interactions, using the local WinAppDriver service running on our machine. However, the most interesting scenario is automating these tests so that, in the context of a CI/CD pipeline, they can be automatically executed every time we commit new code to our repository. The recently announced integration with Azure DevOps makes the task easier to implement, since the WinAppDriver service is already up & running on every Windows hosted agent. However, you may find some challenges in implementing the workflow since UI testing is quite different from unit testing and it has different requirements.
Hopefully, this blog post will help you walking through the implementation of UI Testing for Windows on Azure DevOps. Lets' start!
This post isn't meant to be a comprehensive guide about Azure DevOps for Windows applications. If you want to understand in details some of the concepts that we will implement in this blog post, a good starting point is the workshop that my team delivered at BUILD about moving Windows app forward, which is available here. Specifically, Exercise 5 will guide you through packaging an application with MSIX and implement a CI/CD pipeline on Azure DevOps to automatically build and deploy the MSIX package every time you commit some new code to the repository. Another great resource is the session that Ricardo Minguez Pablos, PM at Microsoft, and Oren Novotny, Microsoft MVP, delivered at BUILD 2019. Here are some important concepts that we will use through the post.
A CI/CD workflow is typically implemented using two different pipelines:
The signing operation is a very delicate one in this process and the separation of concerns between the build and the release pipeline is very important to prevent identity theft and malicious usages of our certificate, especially if we're signing the application with a public or an enterprise certificate. Why is that? Visual Studio supports signing a package as part of the MSIX generation process, so we might be tempted to use this approach to make things simpler and do everything in the build pipeline. However, in order to do that, we would need to make the certificate available to the build process, which means committing it to our repository together with the source code. This is a bad practice, especially if you're working as part of a team. This means that every developer who is working on the application will have the chance to get the certificate and use it to sign other applications, even malicious ones, leading to an identity theft.
When you start to think about implementing UI testing as part of your CI/CD pipeline for MSIX, you face some challenges compared to a traditional testing scenario. Typically unit testing is included as part of the build pipeline, since tests are executed against the code of your application. However, a MSIX package can't be deployed without being signed and, based on what we have just learned, we can't do the signing operation in the build pipeline.
As such, we need to change a bit the architecture we have seen and split the release pipeline into 2 different stages:
In this blog post we're going to build all the components which make this architecture possible. I will assume that you already have an Azure DevOps account. If not, you can create one for free on the official website. Don't worry! Despite the platform contains "Azure" in the name, it isn't directly tight to an Azure subscription. You can create a free account even without owning an Azure subscription or having to provide a credit card.
Let's start to create the build pipeline. Once you're in your project on Azure DevOps, move to the Pipelines → Builds section and choose the option to create a new one. As first step, you will have to choose the repository where your code is hosted. This is the repository which contains our whole solution, which includes both the WPF application and the tests. It's the one we have built on the previous post, which is available here for your reference. Then you will be asked to choose a starting template.
The best one for our scenario is Universal Windows Platform, since it will contain all the steps to create a MSIX package, regardless if it's a pure UWP app or a Win32 app packaged with the Windows Application Packaging Project, like our scenario. The option will create a template based on YAML, which is a markup language used to enable the Infrastructure as code approach, where the pipeline is created through a definition file rather than manual processes. This way, the definition of the pipeline can be treated like any other file of the project, included in the repository and evolve with your code as the requirements change. This is how the template looks like:
# Universal Windows Platform # Build a Universal Windows Platform project using Visual Studio. # Add steps that test and distribute an app, save build artifacts, and more: # https://aka.ms/yaml trigger: - master pool: vmImage: 'windows-latest' variables: solution: '**/*.sln' buildPlatform: 'x86|x64|ARM' buildConfiguration: 'Release' appxPackageDir: '$(build.artifactStagingDirectory)\AppxPackages\\' steps: - task: NuGetToolInstaller@1 - task: NuGetCommand@2 inputs: restoreSolution: '$(solution)' - task: VSBuild@1 inputs: platform: 'x86' solution: '$(solution)' configuration: '$(buildConfiguration)' msbuildArgs: '/p:AppxBundlePlatforms="$(buildPlatform)" /p:AppxPackageDir="$(appxPackageDir)" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=StoreUpload'
This template is a good starting point, but it lacks some important tasks, like publishing the actual output (the MSIX package) so that it can be picked up by the release pipeline.
Let's do a few tweaks:
trigger: - master pool: vmImage: 'windows-latest' variables: buildPlatform: 'x86' buildConfiguration: 'Release' appxPackageDir: '$(build.artifactStagingDirectory)\AppxPackages\\' steps: - task: VSBuild@1 inputs: platform: '$(buildPlatform)' solution: '**/*.sln' configuration: '$(buildConfiguration)' msbuildArgs: '/restore /p:AppxBundlePlatforms="$(buildPlatform)" /p:AppxPackageDir="$(appxPackageDir)" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideloadOnly /p:AppxPackageSigningEnabled=false' - task: PublishBuildArtifacts@1 displayName: 'Publish artifact: drop' inputs: PathtoPublish: '$(appxPackageDir)' condition: succeededOrFailed()
Here is the list of changes we have done:
However, it isn't enough. We need to do a few additional things to properly support the MSIX / UI testing scenario.
When you build a MSIX package using a CI/CD approach, an important requirement is to update the version number in the manifest every time a new package is created. If you rely on a manual approach (eg: changing the version in the manifest before committing the new code) you open the door to many potential problems. It's enough to forget to change the version to trigger the release of a new package with the same version number as the previous one, breaking the update chain. The easiest way to do it is by leveraging a 3rd party extension available on the Azure DevOps Marketplace, called Manifest Versioning Build Tasks by Richard Fennel. Once you have installed it on your Azure DevOps account, you can just add the following step in the YAML before the one which performs the build:
- task: VersionAPPX@2 displayName: 'Version MSIX'
However this isn't enough. This task, in fact, takes care of setting a version number based on the build number generated by Azure DevOps which, by default, has the following syntax:
This expression is translated with a number like 20190205.1. The problem is that this syntax isn't suitable for a MSIX package, since the manifest requires a version number with the expression x.y.z.0 (for example, 18.104.22.168). As such, we need to change the default version number, by adding the following entry before the steps section of the YAML file:
This way, at every build, Azure DevOps will generate a build number like 2019.2.1.0, which is perfect for MSIX versioning.
The changes we have made so far would have been required by any CI/CD implementation for MSIX. The following one, instead, is specific to our scenario, where the solution contains also UI tests. When we move the testing session from the build pipeline to the release pipeline, we hit an issue. In order to run the tests, we need to compile the project which contains the tests, so that we can get a DLL which contains our test class. However, since we are in the release pipeline, we don't have access anymore to the source code of the project, but only to the artifacts published by the build pipeline. As such, other than publishing the MSIX package as an artifact, we need also to publish the DLL which contains our test suite so that the test runner can pick it.
To do this, we can add, as final step, another task to publish a build artifact:
- task: PublishBuildArtifacts@1 inputs: PathtoPublish: 'WpfUiTesting.Tests\bin\Release\' ArtifactName: 'tests' condition: succeededOrFailed()
We're going to create a new artifact, called tests, where we're going to store the compilation output of the test project (WpfUiTesting.Tests).
This is the final YAML file after all our changes:
trigger: - master pool: vmImage: 'windows-latest' variables: buildPlatform: 'x86' buildConfiguration: 'Release' appxPackageDir: '$(build.artifactStagingDirectory)\AppxPackages\\' name: $(date:yyyy).$(Month)$(rev:.r).0 steps: - task: VersionAPPX@2 displayName: 'Version MSIX' - task: VSBuild@1 inputs: platform: '$(buildPlatform)' solution: '**/*.sln' configuration: '$(buildConfiguration)' msbuildArgs: '/restore /p:AppxBundlePlatforms="$(buildPlatform)" /p:AppxPackageDir="$(appxPackageDir)" /p:AppxBundle=Always /p:UapAppxPackageBuildMode=SideloadOnly /p:AppxPackageSigningEnabled=false' - task: PublishBuildArtifacts@1 displayName: 'Publish artifact: drop' inputs: PathtoPublish: '$(appxPackageDir)' condition: succeededOrFailed() - task: PublishBuildArtifacts@1 inputs: PathtoPublish: 'WpfUiTesting.Tests\bin\Release\' ArtifactName: 'tests' condition: succeededOrFailed()
If we now run the build (saving the YAML file will be enough to trigger a new execution) and everything goes well, at the end of the process we should get two artifacts:
Now it's time to build our release pipeline!
As mentioned in the beginning of the post, our pipeline will be made by two different stages. In this section, we're going to focus on the one which performs the UI testing. Only if the tests are all green, we'll move to the next stage to do the actual deployment.
Go back to the Azure DevOps dashboard of your project and move to Pipelines → Releases and create a new one. As first step, you will be asked to choose one of the available templates. Choose Empty job, since we're going to start from scratch:
Give to the stage a meaningful name (for example, UI testing), then close the panel. You should see a screen like the following one:
As first step, let's add an artifact, so that the various stages we're going to build can access to the output of the build pipeline we have previously built. Click on the Add an artifact box and choose, in the Source (build pipeline) field, the name of the build pipeline we have previously built. Feel free to keep all the default options that will be filled.
Now click on the lighting icon on the artifact and set the Continous deloyment trigger switch to Enabled.
Thanks to this option, every time the build pipeline will complete successfully, the release pipeline will be automatically triggered. Once you're back to the pipeline overview, click on the 1 job, 0 task link under the stage name:
We will get access to the deployment process, where we're going to specify all the steps we want to perform. It's similar to what we did with the build pipeline, the only difference is that the release pipeline creation is UI based, so we won't write YAML this time. First click on Agent job and change the Azure Specification dropdown to windows-2019. We want to use the most recent version of Windows and Visual Studio. Now click on the + button near the Agent job to start adding the various tasks we're going to perform. This is the flow we're going to implement:
The easiest way to sign your MSIX package is using a 3rd party extension developed by Stefan Kert, which is available on the Marketplace. Once you have installed it on your account, you can add a Code signing task which, under the hood, will use the signtool utility to sign the package with the certificate you have provided. The bonus of this extension is that it uses Secure Files to store the certificate. This means that you will be able to use it inside a pipeline, but you won't be able to download it or copy somewhere else. Even if you give access to the pipeline to someone else, they won't be able to reuse your certificate to sign other applications.
There are three important fields to edit in the settings:
After you have saved the task, you can click on the Variables tab in the pipeline to store the password. Just add a new item with the same name you have chosen in the task (PfxPassword, in my case) and, as value, the password. For extra security, click on the lock icon near the value so that it will be masked.
This step is required only if you're signing your application with a certificate which hasn't been released by a public certification authority, but it's an internal enterprise or self-signing certificate. In the first scenario, in fact, being released by a public CA, it will be already trusted by Windows and, as such, you can directly proceed to install the application on the hosted agent.
If this isn't your scenario, then, we need to make the certificate trusted by the hosted agent before sideloading the MSIX package, otherwise the operation will fail. The first step is to download the certificate on the machine. For this purpose, we can use the Download secure file task:
There are two important things to configure here:
The next step is to add a PowerShell script that will take care of installing this certificate on the hosted agent. Click again on the + sign near the Agent job and add a task of type PowerShell script.
Choose Inline and copy and paste the following script:
Import-Certificate -Filepath $(certificate.secureFilePath) -CertStoreLocation cert:\LocalMachine\TrustedPeople
We're using the Import-Certificate command to import the certificate we have just uploaded as secure file to the Trusted People store.
To install the MSIX package we need to add another PowerShell Script task, which will use the Add-AppPackage command followed by the file name. However, in our scenario we don't have a fixed name, since Visual Studio generates MSIX packages using, as name, the convention app name_version_CPU architecture (for example, WpfUiTesting.Package_2019.2.1.0_x86). Since it includes the version, the name will change at every build. To workaround this problem we can replace the version number in the file name with the variable $(Build.BuildNumber) , which will be replaced by the current build number.
This is an example of the script to install the package:
Add-AppPackage -Path "WpfUiTesting.Package_$(Build.BuildNumber)_x86.msixbundle"
In order for the task to find the package, you need to expand the Advanced section and set the Working Directory to the folder where the build artifact is copied, like:
$(System.DefaultWorkingDirectory)/_CI - CD/drop/WpfUiTesting.Package_$(Build.BuildNumber)_Test
In my scenario, _CI - CD is the name of the build pipeline, while WpfUiTesting.Package is the name of the packaged version of my WPF app. You can click on the three dots near the field if you want to explore the folder with the artifacts, so that you can find the right one. Also in this case, however, remember to replace the fixed version number with the the build number's variable, since the folder name generated by Visual Studio includes it.
Now that the MSIX application is installed on the hosted agent, we can finally run the UI tests. As first step, we need to run the WinAppDriver service, otherwise the Appium SDK won't be able to launch and interact with the application. As already mentioned, WinAppDriver is already installed on the hosted agent, so we just need to start it using one of the available tasks called WinAppDriver. Once we have added it to our pipeline, all we need to do is to choose which kind of command we want to execute, in this case Start WinAppDriver.
Now we're ready to start our tests, by adding a new task of type Visual Studio test:
The task will pick up the DLL we have produced in the build pipeline and stored in the tests folder. We just need to specify the search criteria in the Test files field. The easiest way is to use pattern matching. For example, in my sample project the test suite is called WpfUiTesting.Tests, so I've used the *test*.dll pattern to look for all the DLLs in the folder which contain the test keyword. This task, under the hood, will leverage a process called Vstest.console, which is a console-based test runner included in Visual Studio.
The last step is to stop WinAppDriver once we have completed the testing. We just need to again the WinAppDriver task but, this time, we need to choose Stop WinAppDriver as action.
That's it! Now you can test the release pipeline, either by manually clicking the Create release button or by committing new code to the repository which, due to the CI/CD configuration, will trigger a new build followed by a new release. During the execution of the release pipeline, the test will be executed and the same "voodoo magic" that we have seen on our PC will happen also on the hosted agent: the application will be launched and the interactions we have specified using the Appium SDK will be triggered. Of course, we won't actually see the test happening, since we don't have a visual connection to the hosted agent (like a remote session). However, we'll be able to follow the various tasks using the Azure DevOps log. At the end of the execution, the pipeline will succeed or fail based on the outcome of the test suite. Regardless, you will be able to see the details about the tests in the dashboard:
In the above screenshot, for example, you can see that the testing session has failed: 1 test didn't pass, which accounts for 50% of the whole test suite. If we click on it, we can see all the details we need to understand why the test has failed. As you can see in the screenshot below, these are the same information that Visual Studio was reporting us through Test Explorer on our local machine:
Now that we have our UI testing stage up & running, we can add a new stage to perform the actual deploy. Just go back to the edit mode of the release pipeline, choose Add → New stage and give it a meaningful name (like Deploy). Then click again on the label 1 job, 0 tasks to start adding the tasks that will need to be performed. The only fixed task to add here is the code signing one, which is the exact same one we have added in the UI testing stage. Every stage, in fact, starts from the artifacts published by the build pipeline, so the starting point is again an unsigned MSIX package. Before deploying it, we need to sign it. I won't repeat the steps, since we need to use the same exact configuration we have used in the UI testing stage.
The next step is to add a task which does the actual deployment. I won't provide step by step instructions here, because it all depends by which is your deployment scenario:
For example, in my scenario I'm deploying the WPF application to a blob storage on Azure, so this is how my deploy stage looks like:
At the end of the process, the whole release pipeline should look like this:
Since the Deploy stage is defined after the UI Testing one, the actual deploy will be performed only if all the tests are passing. If any of the test should fail, the whole UI testing stage will fail, meaning that the deploy one won't be executed. Smart, isn't it?
It's been a long journey! But we have achieved our goal: we have integrated UI testing for our WPF application inside a CI/CD pipeline on Azure DevOps. From now on, every time we're going to commit new code to the repository, the following actions will happen:
This workflow will ensure that the quality bar is always met and the whole test suite is executed every time we make any change to the source code, no matter how small or big.
As a reminder, you can find the full source code of the sample project, including the YAML file for the build pipeline, on GitHub.
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.