Mutable objects

To write robust Shiny applications, it is important to understand mutability in Python objects. Simple objects like numbers, strings, bools, and even tuples are immutable, but most other objects in Python, like lists and dicts, are mutable. This means that they can be modified in place—modifying an object in one part of a program can cause it to be (unexpectedly) different in another part of the program. That makes mutable objects dangerous, and they are everywhere in Python.

In this article, you’ll learn exactly why mutable objects can cause problems for Shiny reactivity, and techniques for solving them.

The problem

Let’s first look at an example featuring (immutable) integer objects.

#| components: [editor, cell]

a = 1
b = a

a += 1
b

Initially, b gets its value from a. Then, the value of a changes. This doesn’t affect b, which retains its original value.

Now, what happens if a and b both point to the same (mutable) list object, and then we change that list in-place?

#| components: [editor, cell]

a = [1, 2]
b = a

a.append(3)
b

If our goal is to end up with a == [1, 2, 3] and b == [1, 2], then we’ve failed.

Mutability can cause unexpected behavior in any Python program, but especially so in reactive programming. For example, if you modify a mutable object stored in a reactive.value, or one returned from a reactive.calc, other consumers of those values will have their values changed. This can cause two different problems. First, the altered value will probably be unexpected. Second, even if the the change in value is expected and desired, it will not trigger downstream reactive objects to re-execute.

Solutions

There are a few ways to fix this problem and end up with the results we want (b == [1, 2]).

Copy on assignment

The first way is to avoid having two variables point to the same object in the first place, by copying the object every time you use it in a new context:

#| components: [editor, cell]

a = [1, 2]
b = a.copy()

a.append(3)
b

Copy on update

The second way is to be disciplined about never mutating the object in question, but using methods and operators that create a copy. For example, there are two ways to add an item to an existing list: x.append(value) which mutates the existing list, as we saw above; and x + [value], which leaves the original list x unchanged and creates a new list object that has the results we want.

#| components: [editor, cell]

a = [1, 2]
b = a

a = a + [3]
b

The advantage to this approach is not eagerly creating defensive copies all the time, as we must in the “copy on assignment” approach. However, if you are performing more updates than assignments, this approach actually makes more copies, plus it gives you more opportunities to slip up and forget not to mutate the object.

Python operations that create copies

We’ve seen that x + [value] creates a new list object and that x.copy() creates a new list object. There are some other common operations that create copies. You can use these patterns to avoid mutating reactive values in place.

  1. List comprehensions: [x for x in a] creates a new list with the same elements as a. This approach is particularly useful when you need to transform the elements of a list in some way, as in [x*2 for x in a].

  2. Slicing: a[:] creates a new list with the same elements as a. This is useful when you need to copy the entire list, or a subset of it.

  3. Star operator: [*a, value] creates a new list with the same elements as a, with the additional value appended after them. This is an easy way to add a single element to the end or start of a list ([value, *a]).

  4. Double star operator: {**a, key: value} creates a new dictionary with the same key-value pairs as a, with the additional key-value pair key: value added. This is an easy way to add a single key-value pair to a dictionary.

Use immutable objects

The third way is to use a different data structure entirely. Instead of list, we will use tuple, which is immutable. Immutable objects do not provide any way to change their values “in place”, even if we wanted to. Therefore, we can be confident that nothing we do to tuple variable a could ever affect tuple variable b.

#| components: [editor, cell]

a = (1, 2)
b = a

a = (*a, 3)  # alternatively, a = a + (3,)
b

For this simple example, a tuple was an adequate substitute for a list, but this won’t always be the case. The pyrsistent Python package provides immutable versions of several common data structures including list, dict, and set; using these objects in conjunction with reactive.value and reactive.calc is much safer than mutable versions.

Examples in Shiny

The rest of this article demonstrates these problems, and their solutions, in the context of a minimal Shiny app.

Example 1: Lack of reactive invalidation

This demo app demonstrates that when an object that is stored in a reactive.value is mutated, the change is not visible to the reactive.value and no reactive invalidation occurs. Below, the add_value_to_list effect retrieves the list stored in user_provided_values and appends an item to it.

Problem: No reactive update 
#| standalone: true
#| components: [editor, viewer]
#| layout: vertical
#| viewerHeight: 250

from shiny import reactive
from shiny.express import input, render, ui

ui.input_numeric("x", "Enter a value to add to the list:", 1)
ui.input_action_button("submit", "Add Value")

@render.code
def out():
    return f"Values: {user_provided_values()}"

# Stores all the values the user has submitted so far
user_provided_values =  reactive.value([])

@reactive.effect
@reactive.event(input.submit)
def add_value_to_list():
    # WATCHOUT! This doesn't work as expected!
    values = user_provided_values()
    values.append(input.x())

Each time the button is clicked, a new item is added to the list; but the reactive.value has no way to know anything has changed. (Surprisingly, even adding user_provided_values.set(values) to the end of add_value_to_list will not help; the reactive value will see that the identity of the new object is the same as its existing object, and ignore the change.)

Switching to the “copy on update” technique fixes the problem. The app below is identical to the one above, except for the body of add_value_to_list. Click on the button a few times–the results now appear correctly.

Solution: Copy on update 
#| standalone: true
#| components: [editor, viewer]
#| layout: vertical
#| viewerHeight: 250

from shiny import reactive
from shiny.express import input, render, ui

ui.input_numeric("x", "Enter a value to add to the list:", 1)
ui.input_action_button("submit", "Add Value")

@render.code
def out():
    return f"Values: {user_provided_values()}"

# Stores all the values the user has submitted so far
user_provided_values =  reactive.value([])

@reactive.effect
@reactive.event(input.submit)
def add_value_to_list():
    # This works by creating a new list object
    user_provided_values.set(user_provided_values() + [input.x()])

Example 2: Leaky changes

Let’s further modify our example; now, we will output not just the values entered by the user, but also a parallel list of those values after being doubled. This example is the same as the last one, with the addition of the @reactive.calc called doubled_values, which is then included in the text output.

In the example below, if you click the button three times, you’d expect the user values to be [1, 1, 1] and the doubled values to be [2, 2, 2]. Click the button below three times. What values do you actually get?

Problem: Mutating in place 
#| standalone: true
#| components: [editor, viewer]
#| layout: vertical
#| viewerHeight: 250

from shiny import reactive
from shiny.express import input, render, ui

ui.input_numeric("x", "Enter a value to add to the list:", 1)
ui.input_action_button("submit", "Add Value")

@render.code
def out():
    return f"User Values: {user_provided_values()}\n" + f"Doubled: {doubled_values()}"

# Stores all the values the user has submitted so far
user_provided_values =  reactive.value([])

@reactive.effect
@reactive.event(input.submit)
def add_value_to_list():
    user_provided_values.set(user_provided_values() + [input.x()])

@reactive.calc
def doubled_values():
    values = user_provided_values()
    for i in range(len(values)):
        values[i] *= 2
    return values

By the third click, the user input that should be [1, 1, 1] is instead [4, 2, 1]! This is because doubled_values does its doubling by modifying the values of the list in place, causing these changes to “leak” back into user_provided_values.

We could fix this by having doubled_values call user_provided_values().copy(). Or we can use a list comprehension, which creates a new list in the process. The second option is shown below.

Solution: Copy with list comprehension 
#| standalone: true
#| components: [editor, viewer]
#| layout: vertical
#| viewerHeight: 250

from shiny import reactive
from shiny.express import input, render, ui

ui.input_numeric("x", "Enter a value to add to the list:", 1)
ui.input_action_button("submit", "Add Value")

@render.code
def out():
    return f"User Values: {user_provided_values()}\n" + f"Doubled: {doubled_values()}"

# Stores all the values the user has submitted so far
user_provided_values =  reactive.value([])

@reactive.effect
@reactive.event(input.submit)
def add_value_to_list():
    user_provided_values.set(user_provided_values() + [input.x()])

@reactive.calc
def doubled_values():
    return [x*2 for x in user_provided_values()]