The Beginner’s Guide to Property-based Testing

Property-based testing is a type of software testing (usually unit-scoped) that allows developers to test software systems by defining properties (or invariants) that should hold true for a range of inputs. I imagine that, for a lot of readers, reading the word "invariant" makes the body shiver. Are we back in class? Will there be math in this article? Will there be a quiz at the end?!

I don’t blame anyone who has this knee-jerk reaction. The Wikipedia page for property-testing reads like an intro to a research paper and quickly jumps into Szemerédi’s regularity lemma. From what I can tell, the practice is mostly associated with Haskell enthusiasts and examples always use common CS problems, not "real world" examples.

And yet, when you bring up the idea of fuzzing (or randomizing) inputs, to create more thorough tests, folks are more open to the concept. And to some extent, that’s basically what this is all about. But, as you will soon see, there is a lot more structure to it and well-defined invariants can sometimes serve as a specification.

In this article, we’ll look at where property testing fits in your project, compare it to other types of tests, and discuss some common mistakes in the real world.

Property-based Vs Example-based Testing

When we think of our usual unit tests, we could classify them as example-based. We already compute the output in some way for the given input(s) and that forms our test’s main assertion.

But our example inputs are usually a very small subset of the entire set of possible inputs. This raises the question – are we really confident about our program? Have we really exhausted all possibilities and edge cases? Property-based testing gives us an approach to increase that confidence — we will define a sort of rule (an invariant or property) and generate a large number of valid inputs. Our test will then verify if our properties hold true for each input.

This brings out two powerful benefits:

• It simplifies searching for bugs and edge cases and offloads part of the process to our tool. (Yay!)
• Our property-based tests don’t have to make it into the code. We can set them up, run them, find out what we need, and then discard them or turn them into an example-based unit test that would bring more value.

But it also adds some costs:

• Writing a good property-based test takes a lot more time than writing unit tests. There’s a one-time cost to learn of course, but that is not the main issue: simply speaking, writing property-based tests can be tough and really time-consuming.
• These tests will run slower than your typical unit test. This increases wait time, and, to some extent, real dollar costs of your CI tool and dev time. See our cost of unit tests analysis example.

But, above all, the most frustrating and negative thing that can happen is for engineers to start using this for every single function they write. Because it is easy to come up with invariants for simple methods (and it feels powerful), developers feel productive writing these tests. But in reality, they will accomplish nothing.

An important part of property-based is knowing when it makes sense to do it.

Property-based Testing Example in TypeScript

Let’s have a look at the method we will be testing. It’s a simple calculateTotalPrice method that does some straightforward addition and multiplication to calculate the total price of a customer’s cart. There’s also a discount that could be applied to the whole order – either a flat write-off or a percentage.

Let’s give the code a read.

type Product = {
    name: string;
    price: number;
}

type Discount = {
    kind: 'flat' | 'percentage';
    amount: number; // if percentage, then a number between 0 and 1
}

type ShoppingCart = {
    items: {
        product: Product;
        quantity: number;
    }[];
    discount?: Discount;
}

const calculateTotalPrice = (cart: ShoppingCart): number => {
    const preDiscountPrice = cart.items.reduce((acc, item) => acc + item.product.price * item.quantity, 0);
    if (!cart.discount) { return preDiscountPrice; }

    if (cart.discount.kind === 'flat') {
        return preDiscountPrice - cart.discount.amount;
    }

    return preDiscountPrice * (1 - cart.discount.amount);
}

I believe engineers of all levels can spot all sorts of things wrong here:

1. Nothing stops the discount from being greater than the total cost.
2. Nothing verifies that quantity or price is greater than zero.
3. Nothing verifies that if type is percentage, then the amount is actually a number between 0 and 1.

Let’s argue that there is a pre-validation step that takes care of points 2 and 3. And let’s assume we missed the edge case with #1. Will property-based testing really help us spot the problem?

It sure will! Here’s an example using fast-check, a property-based testing library written in TypeScript.

First, we would define the schema of our test input as so:

const testInput = fc.record({
  items: fc.array(fc.record({
    product: fc.record({
      name: fc.string(),
      price: fc.float({ min: 1 }),
    }),
    quantity: fc.integer({ min: 1 }),
  }), { minLength: 1 }),
  discount: fc.record({
    kind: fc.constantFrom('flat', 'percentage'),
    amount: fc.float(),
  }),
}) as fc.Arbitrary<ShoppingCart>;

As you can see I’ve added that price and quantity should be at least 1 as we agreed that our pre-validation step should handle this.

So my invariant here is going to be quite simple – given at least one element in an array I expect the price to be greater than zero.

The code for this is quite simple with this library:

fc.assert(
  fc.property(testInput, (inputData: ShoppingCart) => {
    return calculateTotalPrice(inputData) > 0;
  })
);

And here is the sample output – after 15 tests the library was able to find a test case where my invariant failed. Have a look:

Error: Property failed after 15 tests
  { 
    seed: -1919472291, 
    path: "14:0:0:0:0:0:2:0:3:0:1:0:0:0:0:0:1:1:1:1:3:0:0:0", 
    endOnFailure: true 
  }
Counterexample: 
  [
    {
      "items": [
        {"product":{"name":"","price":1},"quantity":1}
      ],
      "discount": {"kind":"flat","amount":1}
    }
  ]
Got error: Property failed by returning false

After fixing the code and covering some of the edge cases the library was no longer able to find any inputs that fail to satisfy my property.

White-box vs Black-box Model

We could also make a test where we assert a property that specifies that when a discount is present, the returned value must be smaller than the pre-discount price. And we could also check if the pre-discount price actually matches the sum of all items.

This makes total sense with a simple example like this, but it starts to fall into white/open-box territory. From what I’ve seen this is quite common with property-based testing and examples on library websites often fall into that category.
In order to set up these two tests, we will have to either make the original code more modular (so we get the pre-discount price separate from the discount) or repeat the logic in our test (so that we can calculate it.)

The first option is not so bad! I like it when my tools help me notice a code smell or improvement idea. We saw an example of this with unit testing as well. The second option is something to avoid – and I see it all too often!
My general rule of thumb is to avoid writing white-box tests as much as possible. Sometimes this limits the level of detail you can pull off, but I find it to be a pragmatic principle that gives you the most bang for your buck. (And time is your buck in this case.)

Best Use-cases For Property-based Testing

Let’s go over some scenarios where property-based testing really shines:

1. New and complex data structures or algorithms. If you are creating your own data structures or your own algorithm then property-based testing will give you a lot of benefits. These "computer science-y" problems tend to have a lot of edge cases and, because it’s new ground, you don’t already know how to spot all of them. Anything to do with trees or graphs is usually a good candidate for property testing in my book.
2. Complex business logic (that usually involves a couple of tricky sub-algorithms) will rarely be fully covered by unit tests. Let’s say you are making your own scheduling heuristic to choose a time slot based on the calendars of many users. You will have a really hard time covering all edge cases through conventional example-based testing.
3. Cheap fuzz tests. Need to fuzz your APIs as part of an integration test suit? It can get quite expensive or hard to pull off because you will need to make thousands of API calls. If you expose your service code for unit tests with a property-based approach, you can run thousands of tests for a fraction of the time and money. They won’t give you the full benefit of a fuzz test since you will be mocking out external dependencies, but it can be a good start.
4. Maintaining a public library. If you own a library that is used heavily in the OSS world or internally within your company, then catching bugs before they are shipped is more important than with your usual software. When we change the code for our website we can roll it out gradually and quickly roll back in case of issues. But when we publish a new version of our library that people start using, we cannot quickly roll back other people’s code. Property-based testing gives you that extra confidence and is another valuable tool for a great library maintainer.

Mistakes to Avoid with Property-based Testing

Let’s finish off with some common mistakes that you should avoid:

1. Not using enough generated inputs. By default the library we used above generates and runs 100 tests for your assertion. In some cases, you might want to run more to exhaust all edge-case possibilities. Whatever, you do, don’t end up going in the other direction as it decreases the effectiveness of your tests. A common mistake occurs when a team has too many property-based test suits that take a long time to run. Decreasing the number of runs fixes the performance issue, but it defeats the whole purpose of the tests. Instead of decreasing, look to optimize your build pipeline by running only affected tests, doing it only in your first lower environment (i.e. beta or alpha), and removing tests that don’t bring any value.
2. Testing trivial things. With property-based testing, you can get a lot of cases with diminishing returns. A lot of the code we write is non-critical and easy to reason about. It can be covered sufficiently through unit tests. Adding property-based testing to confirm the obvious will just waste time and make your tests slower, leading to problem #1.
3. Forgetting that you still need to find edge cases. An incorrect assumption people sometimes make is that property-based testing somehow fully automates your search of edge cases and bugs. This is partially true, but throwing a bunch of different inputs at the problem is not enough. You will need to think if your properties are actually casting a net that is wide enough to catch all bugs.
4. Skipping other types of tests. Engineers can sometimes get too enthusiastic about a concept. This methodology is not a solution to all of your problems!