Properties for a budgeting exercise by Mark Seemann
Attempting to conjure good properties.
Ever since I learned about property-based testing, I've been fascinated by it. The notion of test data generators seem superior to Test Data Builders: Both require some degree of test-specific boilerplate code, but you get more bang for your bob with test data generators. Instead of hard-coding some (often arbitrary) constant value in a Test Data Builder, you define a range of values that, one hopes, belong to the same equivalence class.
Getting started with property-based testing isn't much more difficult than writing good unit tests with Test Data Builders. You can start with a simple unit test that exercises a single example, and fairly mechanically use test data generators to abstract away all the test code that doesn't really matter, but has to be there anyway. I've shown a comprehensive example in my article A restaurant example of refactoring from example-based to property-based testing.
This process, however, may ultimately produce what I've privately dubbed 'example-driven tests on steroids'. They can be useful enough, but don't seem to embody the notion of a property.
What's a property? #
First, it may be helpful to dispel any confusion. Some languages have properties: A special kind of class member that looks like a public class field, but provides a limited buffer zone between an object's internal data structures and the outside world. To be clear, I don't consider this kind of feature to be worth much in terms of encapsulation, but that's a different discussion.
Property-based testing is not about that kind of property.
Rather, the point of property-based testing is to test how an operation systematically behaves. The technique originates in functional programming as a means to test functions. If you think of functions as mathematical objects, a function may exhibit various properties, such as being monotonic, being differentiable, defining a relation, being associative, idempotent, etc.
When it comes to software design, the notion of a function having properties seems closely related to encapsulation, in the sense of design by contract. An invariant sounds to me like a synonym for property.
When working in the realm of mathematics, identifying properties may be fairly straightforward. On the other hand, when working with business rules, or some kind of domain with no direct line to mathematics, they can be harder to find. Particularly if you wish to avoid repeating implementation logic in the properties. For example, it took me years before I managed to identify good properties even for such a simple problem as FizzBuzz.
Even though Scott Wlashin, as ever, has a great list of suggestions, finding true properties seems like an elusive problem. Which often makes me grab onto it with even more resolve.
Hillel Wayne's budgeting kata #
When I ran into Hillel Wayne's budgeting problem, I knew I had to try my hand at it.
It takes less that five minutes to read the problem description, so there's no point in summarizing it here. A bit of context, on the other hand, may be in order, since the problem definition offers none. I found out about it via one of Brian Marick's blog posts, but that article doesn't give much context either. So I asked Brian Marick, who graciously supplied a little more information. Later, Hillel Wayne confirmed that it's a problem designed to compare property-based testing to example-based testing.
As I often do when I engage with katas, I imposed some constraints. I wanted to take on this problem using test-driven development (TDD) driven by true properties, rather than 'examples on steroids'.
This turned out to be surprisingly formidable.
The third time's a charm #
I don't know if my limited exposure to LLM-driven development had already dulled my skills, or if this is just a truly difficult problem, but I had two false starts before I managed to arrive at something that I consider adequate.
I've posted the relevant repository on GitHub. The first two attempts are visible in the first-attempt and second-attempt branches, but I'm going to offer no commentary. You may consult the commit messages if you are curious.
In the rest of this article series, I will cover my progress one property at a time.
- QuickCheck property: Uncategorised bill
- QuickCheck property: Forced category limit applies
- QuickCheck property: Monotonicity in items
- QuickCheck property: Monotonicity in categories
- QuickCheck property: Valid assignment
- QuickCheck property: Category zero-limits
- QuickCheck property: No category fit is unaffordable
- QuickCheck property: Adding unbudgeted category
- QuickCheck property: Distributivity
I chose QuickCheck for property-based testing (PBT) because it's so mature. It also has some drawbacks, particularly when it comes to shrinking. Thus, I may revisit this problem with another PBT framework, possibly in another programming language. That's the reason I've prefixed all article names with QuickCheck: It leaves room for more articles using another framework.
Conclusion #
Having also read Brian Marick's walkthrough of his example-driven attempt, I'd be inclined to accept defeat. It looks as though this problem would be easier to solve with 'old-fashioned' TDD.
On the other hand, I learned a thing or two along the way, so I still think it was a rewarding exercise, worth reporting on.
Next: QuickCheck property: Uncategorised bill.
