How to: Run Playwright .NET tests in Azure DevOps pipelines4 min read

Posted on by

TL/DR: Quick guide to running Playwright .NET tests in Azure DevOps CI/CD pipelines.

This week I decided to try out Microsoft’s new web testing framework, Playwright .NET. I haven’t done much web testing lately, but I do have a small end-to-end test project that currently uses Selenium WebDriver.

I planned to swap out a couple of tests so I could do a like-for-like comparison of the two tools.

Getting up and running was really straightforward. Within 30 minutes I’d installed the Playwright CLI, imported the package and written my first test. I’d also managed to run the test from Visual Studio against a test server and toggled between headed and headless modes.

Out-the-box the Playwright library has loads more in-built features than Selenium WebDriver. I also found the API to be really initiative. I only had to refer to the docs once to understand the element selectors, as they do differ slightly from Selenium.

With my first test complete, my next step was to integrate it with my Azure DevOps CI/CD pipelines.

Continuous Integration

Since I was working with an end-to-end test, I didn’t intend to run tests from the build pipeline. Instead I only wanted to build the project and package the binaries.

Playwright is published via NuGet, so all I needed to do was run dotnet restore, build and publish as usual.

 
 
jobs:
  - job: build
    steps:
    - checkout: self
      clean: true
    - task: UseDotNet@2
      displayName: Install .Net 5
      inputs:
        packageType: "sdk"
        version: "5.0.x"
    - task: DotNetCoreCLI@2
      displayName: Restore Packages
      inputs:
        command: 'restore'
        projects: 'tests/tests.csproj'
        feedsToUse: 'config'
        nugetConfigPath: 'nuget.config'
    - task: DotNetCoreCLI@2
      displayName: Build Projects
      inputs:
        command: 'build'
        projects: 'tests/tests.csproj'
        arguments: '--no-restore -c Release /p:Platform=x64 /p:Version=$(Build.BuildNumber)'
    - task: DotNetCoreCLI@2
      displayName: Publish Test Project
      condition: succeeded()
      inputs:
        command: 'publish'
        publishWebProjects: false
        projects: '$(testProject)'
        arguments: '-c Release -o $(Build.ArtifactStagingDirectory)'
        zipAfterPublish: false
        modifyOutputPath: false
        workingDirectory: '$(build.sourcesDirectory)'
    - publish: $(Build.ArtifactStagingDirectory)
      displayName: Publish Artifacts
      artifact: drop

After running the build I hit my first stumbling block. When building my test project locally, Playwright automatically copied all of its dependencies into a .playwright folder under the bin directory.

However, when using dotnet publish in my build pipeline the .playwright folder wasn’t copied across to the build aftifacts directory.

To get around this I added a few lines of MSBuild code to my project file.

 
 
<Target Name="PublishPlaywrightFiles" AfterTargets="Publish">
  <Message Text="Publishing Playwright files from $(OutDir).playwright\ to $(PublishDir).playwright\" />
  <ItemGroup>
    <PlaywrightFiles Include="$(OutDir).playwright\**\*" />
  </ItemGroup>
  <Copy SourceFiles="@(PlaywrightFiles)" DestinationFiles="@(PlaywrightFiles->'$(PublishDir).playwright\%(RecursiveDir)%(Filename)%(Extension)')" />
</Target>

I ran the CI build again and this time the Playwright files were successfully copied to the build artifacts during the dotnet publish step.

UPDATE: 19/02/2022 – With the latest version of the Playwright package (v1.19.0 at the time of writing), this step is no longer required, since the Playwright binaries are automatically copied to the bin directory on build.

Continuous Delivery

The next step was to hook the test project up to my deployment pipeline so the test would run after the app was deployed to the test server.

I started by importing the artifacts from the CI pipeline and running the test using dotnet test.

 
 
steps:
- task: VSTest@2
  displayName: 'End-to-end Tests'
  inputs:
    testAssemblyVer2: 'drop/tests/tests.dll'
    resultsFolder: '$(Agent.TempDirectory)/TestResults'
    uiTests: true
    vsTestVersion: toolsInstaller
    runSettingsFile: 'bin/ci.runsettings'
    runInParallel: true
    otherConsoleOptions: '/Platform:x64'
    testRunTitle: 'End-to-end Tests'
    failOnMinTestsNotRun: true
    diagnosticsEnabled: true

I ran the deployment and the test failed with: Microsoft.Playwright.PlaywrightException: Driver not found:

Playwright error

Despite including the Playwright dependencies in my build artifacts, the test was unable to run because I hadn’t yet installed the browsers or drivers onto the agent.

To fix this, I added an inline PowerShell step to install the Playwright CLI and then run the playwright install command to download the browsers and drivers:

 
 
steps:
- powershell: | 
   dotnet tool install --global Microsoft.Playwright.CLI
   playwright install
  displayName: 'Install Playwright'

This time my test ran and passed without error.

UPDATE: 19/02/2022 – Since writing the original post, I have discovered that there is no need to install the dotnet tool when running on the build agent. Instead the playwright drivers can be installed by running playwright install directly from the bin directory, as follows:

 
 
steps:
- pwsh: drop/tests/.playwright/node/win32_x64/playwright.cmd install
  displayName: 'Install Playwright'

Conclusion

On first impressions, Playwright appears to be a worthy opponent to the mighty Selenium WebDriver. It’s super quick to get started with and has a user-friendly API.

Unlike Selenium, there are no additional steps required to install browsers or compatible WebDriver versions. All of this is handled by the dotnet integrated CLI and the single command set up, making CI/CD integration a breeze.

Standing ovation for Playwright!

Share this post: