Quickstart
Once you have marbles installed, you can run your existing unittest
tests with marbles and get marbles failure messages, without changing any code. This lets you switch between marbles and unittest failure messages. For instance, you might want to see marbles failure messages during interactive development and debugging, but see unittest failure messages in CI. To do this
$ python -m marbles docs/examples/getting_started.py # development
$ python -m unittest docs/examples/getting_started.py # CI
For example, let’s say we have the following unittest
test case
import requests
import unittest
class ResponseTestCase(unittest.TestCase):
def test_create_resource(self):
endpoint = 'http://example.com/api/v1/resource'
data = {'id': 1, 'name': 'Little Bobby Tables'}
res = requests.put(endpoint, data=data)
self.assertEqual(
res.status_code,
201
)
if __name__ == '__main__':
unittest.main()
If we run this test case with unittest, we’ll see a normal unittest failure message
$ python -m unittest docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/docs/examples/getting_started.py", line 42, in test_create_resource
201
AssertionError: 409 != 201
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
But, if we run this same test case with marbles instead, we get a marbles failure message, without changing any test code
$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
39 res = requests.put(endpoint, data=data)
> 40 self.assertEqual(
41 res.status_code,
42 201
43 )
Locals:
endpoint=http://example.com/api/v1/resource
data={'name': 'Little Bobby Tables', 'id': 1}
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
Reading marbles failure messages
In this section we’ll go over the different parts of a marbles failure message.
Source
Python tracebacks only show the last line of the statement that failed, which can be confusing if the statement that failed spans multiple lines. In a marbles failure message, the “Source” section contains the full assertion statement that failed
$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
39 res = requests.put(endpoint, data=data)
> 40 self.assertEqual(
41 res.status_code,
42 201
43 )
Locals:
endpoint=http://example.com/api/v1/resource
data={'name': 'Little Bobby Tables', 'id': 1}
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
This doesn’t look like a traceback, it looks like code, perhaps even code that I wrote. And, it’s a lot easier to recognize than when it’s inside a traceback
$ python -m unittest docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/docs/examples/getting_started.py", line 42, in test_create_resource
201
AssertionError: 409 != 201
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
Traceback
Speaking of the traceback, where is it? Marbles failure messages contain all of the information you would normally find in a traceback (and more), so we can hide the traceback to make failure messages easier to read without losing any information. If you still want to see the traceback, you can run your tests in verbose mode
$ python -m marbles docs/examples/getting_started.py --verbose
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/path/to/docs/examples/getting_started.py", line 42, in test_create_resource
201
File "/path/to/docs/examples/getting_started.py", line 520, in wrapper
return attr(*args, msg=annotation, **kwargs)
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
39 res = requests.put(endpoint, data=data)
> 40 self.assertEqual(
41 res.status_code,
42 201
43 )
Locals:
endpoint=http://example.com/api/v1/resource
data={'name': 'Little Bobby Tables', 'id': 1}
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
Locals
The “Locals” section of a marbles failure messages contains any variables that are in scope at the time the test failed
$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
39 res = requests.put(endpoint, data=data)
> 40 self.assertEqual(
41 res.status_code,
42 201
43 )
Locals:
endpoint=http://example.com/api/v1/resource
data={'name': 'Little Bobby Tables', 'id': 1}
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
This helps you recover the “state of the world” at the time the test failed and see what the actual and expected runtime values were, without having to put debugging statements in your test code (or even reading or changing your test code at all).
See Curating Locals to see how to control which local variables show up in this section.
Notes
Assertions on marbles.core.TestCase
s accept an optional annotation provided by the author. This annotation, if provided, will be included in the failure message.
Warning
You can provide annotations to assertions on vanilla unittest.TestCase
s only if you run them with marbles. If you try to run annotated unittest.TestCase
tests with unittest they will break.
Let’s add an annotation to our example and see what it looks like
--- /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py
+++ /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py.annotated
@@ -39,7 +39,8 @@
res = requests.put(endpoint, data=data)
self.assertEqual(
res.status_code,
- 201
+ 201,
+ note=res.reason
)
Now when we run our test, we see an additional section
$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
39 res = requests.put(endpoint, data=data)
> 40 self.assertEqual(
41 res.status_code,
42 201,
43 note=res.reason
44 )
Locals:
endpoint=http://example.com/api/v1/resource
data={'name': 'Little Bobby Tables', 'id': 1}
Note:
The request could not be completed due to a conflict with the current
state of the target resource.
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
We go into the Note annotation in more detail in How to win at marbles.
Writing marbles tests
This section will cover how to write marbles.core.TestCase
s and how to port unittest.TestCase
s to marbles.
To write marbles tests, all you need to do is inherit from marbles.core.TestCase
wherever you would normally inherit from unittest.TestCase
and write your test methods exactly as you would normally. Nothing else about your test cases or test methods needs to change (unless you want to add annotations).
For example, let’s take our example test case from earlier
import requests
import unittest
class ResponseTestCase(unittest.TestCase):
def test_create_resource(self):
endpoint = 'http://example.com/api/v1/resource'
data = {'id': 1, 'name': 'Little Bobby Tables'}
res = requests.put(endpoint, data=data)
self.assertEqual(
res.status_code,
201
)
if __name__ == '__main__':
unittest.main()
To turn this into a marbles test case
--- /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py.original
+++ /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py
@@ -1,5 +1,5 @@
import requests
-import unittest
+import marbles.core
# We stub out a put request for the purposes of creating an example
@@ -30,7 +30,7 @@
'state of the target resource.'))
-class ResponseTestCase(unittest.TestCase):
+class ResponseTestCase(marbles.core.TestCase):
def test_create_resource(self):
endpoint = 'http://example.com/api/v1/resource'
@@ -44,4 +44,4 @@
if __name__ == '__main__':
- unittest.main()
+ marbles.core.main()
Now, we get the following output, whether we run our tests with unittest or with marbles
$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201
Source (/path/to/docs/examples/getting_started.py):
39 res = requests.put(endpoint, data=data)
> 40 self.assertEqual(
41 res.status_code,
42 201
43 )
Locals:
endpoint=http://example.com/api/v1/resource
data={'name': 'Little Bobby Tables', 'id': 1}
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (failures=1)
Note
If you run your tests with -m unittest
, the failure message will always include the traceback, even if you don’t run your tests in verbose mode. To hide the traceback, you need to run your tests with -m marbles
.
Porting unittest tests
To replace all of your unittest test cases with marbles test cases
find /path/to/files -type f -exec sed -i 's/unittest/marbles.core/g' {} \;
Warning
This may not be safe. For example, it will replace unittest.mock
with marbles.core.mock
, which doesn’t exist. If you use this command, be sure to review the diff.
Don’t forget to declare marbles as a dependency.
Running tests
You can run marbles tests exactly like you run vanilla unit tests
python -m marbles /path/to/marbles_tests.py
# -or-
python -m unittest /path/to/marbles_tests.py
As we saw above, you can also run vanilla unit tests with marbles and get marbles failure messages, without changing the base class of you test cases
python -m marbles /path/to/unittest_tests.py
Marbles also creates a setuptools command so if you are used to running
python setup.py test
, you can now run:
python setup.py marbles
You can go one step further and alias the command test to run marbles
by adding the following to setup.cfg
:
[aliases]
test = marbles
Declaring marbles as a dependency
To ensure that marbles is available wherever you need to run your package’s unit tests you need to declare marbles.core
as a test dependency in your setup.py
script
setup(
...
tests_require=[
'marbles.core'
],
...
)