End-to-end testing

End-to-end testing is like checking your app from start to finish, just as a user would.

Imagine you’re using your Shiny app. You click buttons, enter data, and see results on a graph or a dashboard. End-to-end tests mimic these actions. Instead of manually clicking around, we write code to do this for us. The code interacts with your app like a user, checking if everything works as expected.

Benefits

  • End-to-end tests find issues early, like broken links or unexpected behavior.
  • As your app grows, it becomes harder to keep track of all parts. Tests help ensure nothing breaks.

Playwright

Playwright is an open-source library developed by Microsoft. It enables developers to automate browser interactions and perform end-to-end testing of web applications.

Benefits of using Playwright for Shiny App testing

  • End-to-End Testing: Playwright allows you to simulate real user interactions with your Shiny app, ensuring that the reactive components and user flows work as expected.
  • Cross-Browser Testing: Playwright supports multiple browsers like Chromium, Firefox, and Safari(Webkit), enabling you to test your Shiny app’s compatibility across different browser environments.
  • Dynamic wait times Playwright provides dynamic wait times, automatically waiting for elements to be ready before interacting with them, which eliminates the need for explicit waits and reduces flakiness caused by timing issues.

For detailed information and guidance, check out the official Playwright documentation.

How it works: a basic example

Consider the following app that simply displays a message with double the slider value:

app.py
from shiny import render, ui
from shiny.express import input

ui.panel_title("Hello Shiny!")
ui.input_slider("n", "N", 0, 100, 20)

@render.text
def txt():
    return f"n*2 is {input.n() * 2}"

If we want to test that the shiny app works for the following scenario:

  1. Wait for the Shiny app to finish loading
  2. Drag the slider to value as 55
  3. Verify the output text changes to reflect the value of n*2 is 110

The test code to test the shiny app to emulate the above scenario would be as following:

test_basic_app.py
from shiny.playwright import controller
from shiny.run import ShinyAppProc
from playwright.sync_api import Page
from shiny.pytest import create_app_fixture

app = create_app_fixture("../app.py")

def test_basic_app(page: Page, app: ShinyAppProc):
    page.goto(app.url)
    txt = controller.OutputText(page, "txt")
    slider = controller.InputSlider(page, "n")
    slider.set("55")
    txt.expect_value("n*2 is 110")

Here’s a breakdown of what’s happening in the test code:

  1. The code begins by importing the controller module. This module provides classes and methods for controlling Shiny components. With these classes, you can mimic user interactions (e.g., .set()) and verify state (e.g., .expect_value()). See here for more information on the available classes and methods.

  2. Defines test_basic_app function with page and app parameters. page is an instance of the Page class from the Playwright library, which represents a single tab in a browser, and app is an instance of the ShinyAppProc class, which represents the Shiny app being tested.

  3. Navigates to the app’s URL.

  4. Creates instances of OutputText and InputSlider for UI elements.

  5. Sets the slider value to 55.

  6. Checks if the output text displays n*2 is 110 as expected.

And visually, this is what happens when the test runs:

Run the test

To run end-to-end tests, you’ll first want to make sure pytest and the pytest-playwright plugin are installed:

pip install pytest pytest-playwright

Now, if you have the app.py and test_basic_app.py files in the same directory, from that same directory, run the pytest command:

pytest

======== test session starts ========
platform darwin -- Python 3.10.12, pytest-7.4.4, pluggy-1.4.0
configfile: pytest.ini
plugins: asyncio-0.21.0, timeout-2.1.0, Faker-20.1.0, cov-4.1.0, playwright-0.4.4, rerunfailures-11.1.2, xdist-3.3.1, base-url-2.1.0, hydra-core-1.3.2, anyio-3.7.0, syrupy-4.0.5, shiny-1.0.0
asyncio: mode=strict
12 workers [1 item]
.


======== 1 passed in 3.05s ========

Each test inside the file is shown by a single character in the output:

  • . for passing
  • F for failure.

For more information on different options for running tests (like running tests in headful mode or in different browsers), check out the Playwright documentation.

Add tests an existing app

If you already have a shiny app and want to add a test file to it, enter the following command in your terminal/console:

shiny add test

This command will ask you for a path to the app file as well as a name for the test file. Just make sure to follow pytest conventions for naming the test file (should start with test_). Note that the test file this command provides will need to be updated with the test logic you want to add.

Learn more

For more information about the testing methods available to the user, read the reference documentation about shiny testing API here.