How to win at marbles
Out of the box, marbles gives you better failure messages, but it also gives you control over what information your failure messages contain. In this section, we’ll cover how to write your tests to get the most out of your failure messages.
Curating Locals
Local variables defined within the test are included in the “Locals” section of the failure message. This helps the test consumer to reconstruct the “state of the world” at the time the test failed. Marbles lets you control which locals will be included in this section.
Excluding Locals
Not all local variables will be relevant to the test consumer, and exposing too many locals could be as confusing as exposing too few. If you need to define variables in your test but don’t want them to show up in the output, you can exclude them from the “Locals” section by making them internal or name-mangled (prepending them with one or two underscores).
Note
The local variables self, msg, and note are automatically excluded from the “Locals” section.
import marbles.core
class IntermediateStateTestCase(marbles.core.TestCase):
def test_foo(self):
start_str = 'foo'
# Capitalize our string one character at a time
_intermediate_state_1 = start_str.replace('f', 'F')
__intermediate_state_2 = _intermediate_state_1.replace('o', 'O')
stop_str = __intermediate_state_2.lower()
self.assertNotEqual(start_str, stop_str)
if __name__ == '__main__':
marbles.core.main()
This will produce the following output. Notice that the variables _intermediate_state_1 and __intermediate_state_2 don’t appear in “Locals”.
$ python -m marbles docs/examples/exclude_locals.py
F
======================================================================
FAIL: test_string_case (docs.examples.exclude_locals.IntermediateStateTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 'foo' == 'foo'
Source (/path/to/docs/examples/exclude_locals.py):
14
> 15 self.assertNotEqual(start_str, stop_str)
16
Locals:
start_str=foo
stop_str=foo
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=1)
Locals-only Locals
Conversely, there may be some local state that you want to expose to the test consumer that your test doesn’t actually need to use. We recommend creating local variables for these anyway.
Note
Python linters like flake8 will complain about variables that are assigned but never used, but most linters provide ways of ignoring specific lines.
In the example below, even though we don’t need to define file_name, it’s useful for the test consumer to know what file has a size we don’t expect. We sidestep the flake8 warning with the comment # noqa: F841 (F841 is the code for “local variable is assigned but never used”)
import os
import marbles.core
class FileTestCase(marbles.core.TestCase):
def test_file_size(self):
file_name = __file__ # noqa: F841
self.assertEqual(os.path.getsize(__file__), 0)
if __name__ == '__main__':
marbles.core.main()
When we run this test, we’ll see file_name in locals
$ python -m marbles docs/examples/extra_locals.py
F
======================================================================
FAIL: test_file_size (docs.examples.extra_locals.FileTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 288 != 0
Source (/path/to/docs/examples/extra_locals.py):
9
> 10 self.assertEqual(os.path.getsize(__file__), 0)
11
Locals:
file_name=/path/to/docs/examples/extra_locals.py
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=1)
Using Notes
Note annotations are intended to help the test author communicate any context or background information they have about the test. For example, what’s the context of the edge case this particular test method is exercising? The note annotation is a good place to put information that doesn’t fit into the test method name or into the assertion statement.
Note annotations are accepted in addition to the msg argument accepted by all assertions. If specified, the msg is used as the error message on failure, otherwise it will be the standard message provided by the assertion.
The msg should be used to explain exactly what the assertion failure was, e.g., x was not greater than y, while the note can provide more information about why it’s important that x be greater than y, why we expect x to be greater than y, what needs to happen if x isn’t greater than y, etc. The note doesn’t (and in fact shouldn’t) explain what the assertion failure is because the msg already does that well.
For example, in the failure message below, the standard message (409 != 201) and the note annotation complement each other. The standard message states that the status code we got (409) doesn’t equal the status code we expected (201), while the note provides context about the status code 409.
$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
39 res = requests.put(endpoint, data=data)
> 40 self.assertEqual(
41 res.status_code,
42 201,
43 note=res.reason
44 )
Locals:
endpoint=http://example.com/api/v1/resource
data={'name': 'Little Bobby Tables', 'id': 1}
Note:
The request could not be completed due to a conflict with the current
state of the target resource.
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
Note
We recommend that you bind note annotations to a variable named note, or pass them to assertions directly, so that they’re not repeated in the “Locals” section. Otherwise, you’ll need to manually exclude them from the “Locals” section. See Excluding Locals for how to do this.
Dynamic Note
Note annotations can contain format string fields that will be expanded with local variables if/when the test fails. They’re similar to f-strings in that you don’t have to call str.format() yourself, but they differ in that they’re only expanded if and when your test fails.
Let’s add a format string field to our note annotation
--- /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py.annotated
+++ /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py.dynamic
@@ -36,11 +36,13 @@
endpoint = 'http://example.com/api/v1/resource'
data = {'id': 1, 'name': 'Little Bobby Tables'}
+ note = '{res.reason} at {endpoint}'
+
res = requests.put(endpoint, data=data)
self.assertEqual(
res.status_code,
201,
- note=res.reason
+ note=note
)
When this test fails, endpoint will be formatted into our note string
$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
42 res = requests.put(endpoint, data=data)
> 43 self.assertEqual(
44 res.status_code,
45 201,
46 note=note
47 )
Locals:
data={'name': 'Little Bobby Tables', 'id': 1}
endpoint=http://example.com/api/v1/resource
Note:
The request could not be completed due to a conflict with the current
state of the target resource at http://example.com/api/v1/resource.
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
Required Note
Depending on the complexity of what you’re testing, you may want to require that note be provided for all assertions. If you want to require notes, your test cases should inherit from marbles.core.AnnotatedTestCase instead of from marbles.core.TestCase. The only difference is that, while note is optional for assertions on TestCases, it’s required for all assertions on AnnotatedTestCases.
If you don’t provide notes to an assertion on an AnnotatedTestCase you’ll see an error
E
======================================================================
ERROR: test_for_edge_case (docs.examples.required_note.ComplexTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/docs/examples/required_note.py", line 7, in test_for_edge_case
self.assertTrue(False)
File "/path/to/marbles/core/marbles/core/marbles.py", line 535, in wrapper
list(args) + list(rem_args), kwargs) as annotation:
File "/path/to/marbles/core/marbles/core/marbles.py", line 407, in __enter__
self._validate_annotation(annotation)
File "/path/to/marbles/core/marbles/core/marbles.py", line 397, in _validate_annotation
raise AnnotationError(error)
marbles.core.marbles.AnnotationError: Annotation missing required fields: {'note'}
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (errors=1)
Custom assertions
unittest.TestCases expose several assert methods for use in unit tests. These assert methods range from very straightforward assertions like assertTrue() to the more detailed assertions like assertWarnsRegex(). These assertions allow the test author to clearly and concisely assert their expectations.
Mixins
The marbles.mixins package adds even more assertion methods that you can use, including assertions about betweenness, monotonicity, files, etc. For the most part, marbles.mixins assertions trivially wrap unittest assertions. The reason to use specific assertions is that the semantically-richer method names can give the test consumer valuable information about the predicate being tested, the types of the objects being tested, etc. For example, assertRegex() doesn’t tell you anything about the string being tested, assertFileNameRegex() immediately tells you that the string being tested is a file name.
For example, let’s say we’ve written a function that sorts a list of numbers (which we shouldn’t have done because sorted() is included in the standard library). We can write a concise unit test for this function using mixin assertions about monotonicity
import marbles.core
import marbles.mixins
def my_sort(i, reverse=False):
'''Sort the elements in ``i``.'''
# Purposefully sort in the wrong order so our unit test will fail
return sorted(i, reverse=~reverse)
class SortTestCase(marbles.core.TestCase, marbles.mixins.MonotonicMixins):
def test_sort(self):
i = [1, 3, 4, 5, 2, 0, 8]
self.assertMonotonicIncreasing(my_sort(i))
self.assertMonotonicDecreasing(my_sort(i, reverse=True))
if __name__ == '__main__':
marbles.core.main()
These custom assertions are provided via mixin classes so that they can use other assertions as building blocks. Using mixins, instead of straight inheritance, means that you can compose multiple mixins to create a test case with all the assertions that you need.
Warning
marbles.mixins can be mixed into a unittest.TestCase, a marbles.core.TestCase, a marbles.core.AnnotatedTestCase, or any other class that implements a unittest.TestCase interface. To enforce this, mixins define abstract methods. This means that, when mixing them into your test case, they must come after the class(es) that implement those methods instead of appearing first in the inheritance list like normal mixins.
Writing your own mixins
You can write your own assertions and mix them in to your test cases, too. We recommend reading the marbles.mixins source code to see how to do this. Here is the UniqueMixins source as an example:
class UniqueMixins(abc.ABC):
'''Built-in assertions about uniqueness.
These assertions can handle containers that contain unhashable
elements.
'''
@abc.abstractmethod
def fail(self, msg):
pass # pragma: no cover
@abc.abstractmethod
def assertIsInstance(self, obj, cls, msg):
pass # pragma: no cover
@abc.abstractmethod
def _formatMessage(self, msg, standardMsg):
pass # pragma: no cover
def assertUnique(self, container, msg=None):
'''Fail if elements in ``container`` are not unique.
Parameters
----------
container : iterable
msg : str
If not provided, the :mod:`marbles.mixins` or
:mod:`unittest` standard message will be used.
Raises
------
TypeError
If ``container`` is not iterable.
'''
if not isinstance(container, collections.abc.Iterable):
raise TypeError('First argument is not iterable')
standardMsg = 'Elements in %s are not unique' % (container,)
# We iterate over each element in the container instead of
# comparing len(container) == len(set(container)) to allow
# for containers that contain unhashable types
for idx, elem in enumerate(container):
# If elem appears at an earlier or later index position
# the elements are not unique
if elem in container[:idx] or elem in container[idx+1:]:
self.fail(self._formatMessage(msg, standardMsg))
def assertNotUnique(self, container, msg=None):
'''Fail if elements in ``container`` are unique.
Parameters
----------
container : iterable
msg : str
If not provided, the :mod:`marbles.mixins` or
:mod:`unittest` standard message will be used.
Raises
------
TypeError
If ``container`` is not iterable.
'''
if not isinstance(container, collections.abc.Iterable):
raise TypeError('First argument is not iterable')
standardMsg = 'Elements in %s are unique' % (container,)
# We iterate over each element in the container instead of
# comparing len(container) == len(set(container)) to allow
# for containers that contain unhashable types
for idx, elem in enumerate(container):
# If elem appears at an earlier or later index position
# the elements are not unique
if elem in container[:idx] or elem in container[idx+1:]:
return # succeed fast
self.fail(self._formatMessage(msg, standardMsg))
If you write assertions that you think would be useful for others, we would love to see a pull request from you!
Logging
You can configure marbles.core to log information about every assertion made during a test run as a JSON blob. This includes the test method name, the assertion, the result of the assertion, the arguments passed to the assertion, runtime variables, etc.
These logs can be transferred to another system for later analysis and reporting. For example, you could run logstash after a test run to upload your logs to Elasticsearch, and then use Kibana to analyze them, maybe creating dashboards that show how many assertion failures you get over time, grouped by whether or not assertions are annotated.
See marbles.core.log for information on configuring the logger.