Quickstart

Once you have marbles installed, you can run your existing unittest tests with marbles and get marbles failure messages, without changing any code. This lets you switch between marbles and unittest failure messages. For instance, you might want to see marbles failure messages during interactive development and debugging, but see unittest failure messages in CI. To do this

$ python -m marbles docs/examples/getting_started.py  # development
$ python -m unittest docs/examples/getting_started.py  # CI

For example, let’s say we have the following unittest test case

import requests
import unittest


class ResponseTestCase(unittest.TestCase):

    def test_create_resource(self):
        endpoint = 'http://example.com/api/v1/resource'
        data = {'id': 1, 'name': 'Little Bobby Tables'}

        res = requests.put(endpoint, data=data)
        self.assertEqual(
            res.status_code,
            201
        )


if __name__ == '__main__':
    unittest.main()

If we run this test case with unittest, we’ll see a normal unittest failure message

$ python -m unittest docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/docs/examples/getting_started.py", line 42, in test_create_resource
    201
AssertionError: 409 != 201

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

But, if we run this same test case with marbles instead, we get a marbles failure message, without changing any test code

$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201

Source (/path/to/docs/examples/getting_started.py):
     39 res = requests.put(endpoint, data=data)
 >   40 self.assertEqual(
     41     res.status_code,
     42     201
     43 )
Locals:
        endpoint=http://example.com/api/v1/resource
        data={'name': 'Little Bobby Tables', 'id': 1}


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

Reading marbles failure messages

In this section we’ll go over the different parts of a marbles failure message.

Source

Python tracebacks only show the last line of the statement that failed, which can be confusing if the statement that failed spans multiple lines. In a marbles failure message, the “Source” section contains the full assertion statement that failed

$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201

Source (/path/to/docs/examples/getting_started.py):
     39 res = requests.put(endpoint, data=data)
 >   40 self.assertEqual(
     41     res.status_code,
     42     201
     43 )
Locals:
        endpoint=http://example.com/api/v1/resource
        data={'name': 'Little Bobby Tables', 'id': 1}


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

This doesn’t look like a traceback, it looks like code, perhaps even code that I wrote. And, it’s a lot easier to recognize than when it’s inside a traceback

$ python -m unittest docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/docs/examples/getting_started.py", line 42, in test_create_resource
    201
AssertionError: 409 != 201

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

Traceback

Speaking of the traceback, where is it? Marbles failure messages contain all of the information you would normally find in a traceback (and more), so we can hide the traceback to make failure messages easier to read without losing any information. If you still want to see the traceback, you can run your tests in verbose mode

$ python -m marbles docs/examples/getting_started.py --verbose
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/docs/examples/getting_started.py", line 42, in test_create_resource
    201
  File "/path/to/docs/examples/getting_started.py", line 520, in wrapper
    return attr(*args, msg=annotation, **kwargs)
marbles.core.marbles.ContextualAssertionError: 409 != 201

Source (/path/to/docs/examples/getting_started.py):
     39 res = requests.put(endpoint, data=data)
 >   40 self.assertEqual(
     41     res.status_code,
     42     201
     43 )
Locals:
        endpoint=http://example.com/api/v1/resource
        data={'name': 'Little Bobby Tables', 'id': 1}


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

Locals

The “Locals” section of a marbles failure messages contains any variables that are in scope at the time the test failed

$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201

Source (/path/to/docs/examples/getting_started.py):
     39 res = requests.put(endpoint, data=data)
 >   40 self.assertEqual(
     41     res.status_code,
     42     201
     43 )
Locals:
        endpoint=http://example.com/api/v1/resource
        data={'name': 'Little Bobby Tables', 'id': 1}


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

This helps you recover the “state of the world” at the time the test failed and see what the actual and expected runtime values were, without having to put debugging statements in your test code (or even reading or changing your test code at all).

See Curating Locals to see how to control which local variables show up in this section.

Notes

Assertions on marbles.core.TestCases accept an optional annotation provided by the author. This annotation, if provided, will be included in the failure message.

Warning

You can provide annotations to assertions on vanilla unittest.TestCases only if you run them with marbles. If you try to run annotated unittest.TestCase tests with unittest they will break.

Let’s add an annotation to our example and see what it looks like

--- /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py
+++ /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py.annotated
@@ -39,7 +39,8 @@
         res = requests.put(endpoint, data=data)
         self.assertEqual(
             res.status_code,
-            201
+            201,
+            note=res.reason
         )
 
 

Now when we run our test, we see an additional section

$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201

Source (/path/to/docs/examples/getting_started.py):
     39 res = requests.put(endpoint, data=data)
 >   40 self.assertEqual(
     41     res.status_code,
     42     201,
     43     note=res.reason
     44 )
Locals:
        endpoint=http://example.com/api/v1/resource
        data={'name': 'Little Bobby Tables', 'id': 1}
Note:
        The request could not be completed due to a conflict with the current
        state of the target resource.


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

We go into the Note annotation in more detail in How to win at marbles.

Writing marbles tests

This section will cover how to write marbles.core.TestCases and how to port unittest.TestCases to marbles.

To write marbles tests, all you need to do is inherit from marbles.core.TestCase wherever you would normally inherit from unittest.TestCase and write your test methods exactly as you would normally. Nothing else about your test cases or test methods needs to change (unless you want to add annotations).

For example, let’s take our example test case from earlier

import requests
import unittest


class ResponseTestCase(unittest.TestCase):

    def test_create_resource(self):
        endpoint = 'http://example.com/api/v1/resource'
        data = {'id': 1, 'name': 'Little Bobby Tables'}

        res = requests.put(endpoint, data=data)
        self.assertEqual(
            res.status_code,
            201
        )


if __name__ == '__main__':
    unittest.main()

To turn this into a marbles test case

--- /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py.original
+++ /home/docs/checkouts/readthedocs.org/user_builds/marbles/checkouts/latest/docs/examples/getting_started.py
@@ -1,5 +1,5 @@
 import requests
-import unittest
+import marbles.core
 
 
 # We stub out a put request for the purposes of creating an example
@@ -30,7 +30,7 @@
                                 'state of the target resource.'))
 
 
-class ResponseTestCase(unittest.TestCase):
+class ResponseTestCase(marbles.core.TestCase):
 
     def test_create_resource(self):
         endpoint = 'http://example.com/api/v1/resource'
@@ -44,4 +44,4 @@
 
 
 if __name__ == '__main__':
-    unittest.main()
+    marbles.core.main()

Now, we get the following output, whether we run our tests with unittest or with marbles

$ python -m marbles docs/examples/getting_started.py
F
======================================================================
FAIL: test_create_resource (docs.examples.getting_started.ResponseTestCase)
----------------------------------------------------------------------
marbles.core.marbles.ContextualAssertionError: 409 != 201

Source (/path/to/docs/examples/getting_started.py):
     39 res = requests.put(endpoint, data=data)
 >   40 self.assertEqual(
     41     res.status_code,
     42     201
     43 )
Locals:
        endpoint=http://example.com/api/v1/resource
        data={'name': 'Little Bobby Tables', 'id': 1}


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)

Note

If you run your tests with -m unittest, the failure message will always include the traceback, even if you don’t run your tests in verbose mode. To hide the traceback, you need to run your tests with -m marbles.

Porting unittest tests

To replace all of your unittest test cases with marbles test cases

find /path/to/files -type f -exec sed -i 's/unittest/marbles.core/g' {} \;

Warning

This may not be safe. For example, it will replace unittest.mock with marbles.core.mock, which doesn’t exist. If you use this command, be sure to review the diff.

Don’t forget to declare marbles as a dependency.

Running tests

You can run marbles tests exactly like you run vanilla unit tests

python -m marbles /path/to/marbles_tests.py
# -or-
python -m unittest /path/to/marbles_tests.py

As we saw above, you can also run vanilla unit tests with marbles and get marbles failure messages, without changing the base class of you test cases

python -m marbles /path/to/unittest_tests.py

Marbles also creates a setuptools command so if you are used to running python setup.py test, you can now run:

python setup.py marbles

You can go one step further and alias the command test to run marbles by adding the following to setup.cfg:

[aliases]
test = marbles

Declaring marbles as a dependency

To ensure that marbles is available wherever you need to run your package’s unit tests you need to declare marbles.core as a test dependency in your setup.py script

setup(
    ...
    tests_require=[
        'marbles.core'
    ],
    ...
)