Programming, Python, Uncategorized

Unit testing – Questioning your assertions

Testing is progressively under-emphasized for smaller and simpler programs. Of course. Who needs formal testing for a “Hello, world!” demonstration program?

For even moderately longer work, I’ve found unit testing can open shortcuts to solutions. When you write a test, you do several things.

  • You check for correct operation. Boring. You could do that by hand.
  • Functions are documented with working examples. Handy.
  • Return values are kept honest. Each time you run your test suite, you run tests going all the way back to the first one you wrote, catching unintended consequences from adapting incremental work.

That last point is really helpful. It’s natural to rethink how functionality should be factored. If you introduce a low-level bug, perhaps only manifesting in some obscure edge case, it’s easy to think everything is peachy at the higher levels.

Until down the road when things aren’t so peachy.

That’s why simultaneously developing an application and its testing is a pretty good idea. It keeps you on the rails for how things work. You’re encouraged to break a problem into manageable and mutually cooperative pieces. By the time your application is running, you’ve got assured functionality to build from in the future.

Best of all, Python provides tools that allow automated testing with near-zero effort. Seriously.

Every time you write a little throwaway code to exercise a function or class, you’re writing a unit test. That’s a good thing unless you throw your test away.

Pytest is a great way to preserve all those intermediate tests without cluttering up your application It takes near-zero extra effort, just the right foundation.

Coding for pytest

There are a couple of stylistic points to keep in mind to get the most out of pytest.

Write your code for import. For instance, given a Python script called hello.py, this statement shouldn’t actually execute anything:

import hello

Don’t panic if you want to run your code as a stand-alone program. You have at least two choices for that.

You can put all your working code in one importable file. In a second file, your “stand-alone” executable, import your module and run it.

Or, probably more what you’re looking for, write your operative code in an importable fashion and add this at the bottom:

if __name__ == '__main__':
        conquerWorld() # run entry points to your code here.

A more detailed example will follow. The special variable __name__ contains the name of the module currently running. The main script, the one you ran from the command line, is always called __main__.

That’s one tip for getting the most out of pytest. You can also make your code more testable if you separate initialization, modification, and computation in separate functions from output production. Consider never printing or writing anything to disk that isn’t a function’s return value. More in the following examples.

Hello, world, in full nerd mode

Yeah, I know. We agreed there is no need to formally test a “Hello, world” program. Let’s do so anyway.

Here’s an importable version of hello.py:

#!/usr/bin/python

def greetUser(s):
        s += '!'
        print(s)

if __name__ == '__main__':
        greetUser('Hello, world')

That fits the criteria of being a passive import. You could use greetUser from another file with:

#!/usr/bin/python

import hello

hello.greetUser('Hello, world')

Either way, hello.py will take a string, add a trailing exclamation point, and print it. We still have a testing limitation, though. Since greetUser doesn’t return a value, it’s a little inconvenient to test.

Better to separate output into separate functions. Let’s move the print statement into the “if” part of the script:

#!/usr/bin/python

def greetUser(s):
    s += '!'
    return(s)

if __name__ == '__main__':
    print(greetUser('Hello, world'))

Great. Now greetUser returns a value we can check.

Introducing pytest

Pytest is a well-designed, feature-rich, testing environment. It’s worth learning all its nuances. A few simple examples will cover a wide range of needs, so we’ll just take a sip of pytest in this essay – but it’s a high octane sip. These simple examples should offer good ideas for how to code more accurately.

First, let’s look at some simple mechanics, like what pytest actually, physically, runs.

If you run pytest without any command line arguments, it will look in the current directory and recursively search all subdirectories for files whose names match either “test_*py” or “*test.py”. For example:

test_hello.py
hellotest.py

In each file pytest runs, it runs functions whose names start with “test,” for example:

def test_greetUser():
        test code here...

Other functions can be defined for your internal use, and there are many possibilities for how to write tests with parameterization, decorators, and lots of potential to test your code however needed.

Tests use the assert keyword to flag incorrect answers. Once you get familiar with what’s covered here, read the pytest documentation. You can also run tests with expected failures. You might want to pass a string to a function expecting an integer to make sure you get a TypeError. That would be an example of an expected failure.

Pytest -h will give you an idea of how many options are available. Don’t worry if it seems like too much. Simple pytest usage is just that – simple.

A few handy command line options are:

pytest -x # stop after the first failed test
pytest --maxfail=2 # stop if you get 2 failures
pytest test_greetUser.py # only run tests in test_greetUser.py

Now let’s test our “hello” program.

Simple tests

A test file is like a module. You don’t have to specify an interpreter.

Create a file called test_greetUser.py:

import hello # read the python file we want to test

def test_greetUser():
        assert hello.greetUser('Hello, world') == 'Hello, world!'

That’s it. Run greetUser with the argument “Hello, world” and make sure it adds an exclamation point.

Here’s what that pytest run looked like:

$ pytest
==== test session starts ====
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/carl/pyjunk/testtxt
collected 1 item

test_greetUser.py .                                                      [100%]
==== 1 passed in 0.01s ====

Very un-scary, and actually useful. Let’s say you write another fifteen pages of code and decide it would be better if greetUser() returned a capitalized version of its argument:

def greetUser(s):
        s += '!'
        return(s.upper())

All cool, but you have altered the original intent of greetUser with that s.upper() in the return statement. The next time you run pytest you’ll get a reminder greetUser is no longer doing what it used to because the assert statement in the equality test will flag a failure.

Here’s what you would see in that case:

$ pytest
==== test session starts =====
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/carl/pyjunk/testtxt
collected 1 item

test_greetUser.py F                                                      [100%]
===== FAILURES =====
____ test_greetUser ____
    def test_greetUser():
>           assert hello.greetUser('Hello, world') == 'Hello, world!'
E           AssertionError: assert 'HELLO, WORLD!' == 'Hello, world!'
E             - Hello, world!
E             + HELLO, WORLD!

test_greetUser.py:4: AssertionError
===== short test summary info ====
FAILED test_greetUser.py::test_greetUser - AssertionError: assert 'HELLO, WOR...
===== 1 failed in 0.04s ====

The lines starting with “E” tell the tale. “Hello, world!” expected, “HELLO, WORLD!” returned.

When to use pytest

Unit testing isn’t needed for short, one-off scripts, but it won’t hurt them, either. If you find yourself writing a little snippet to exercise some portion of your code to review its operation, that’s a good reason to write a pytest file to run the snippet for you.

You could even plan your code by writing the tests first, before you write the program itself.

For instance, let’s say I want to write a class that will help me print a greeting as presented, in all upper case, or with a trailing exclamation point, or as a question. I could start with tests in a file called test_hello.py:

import hello # hello.py is what I'll write next

def test_raw():
    h = hello.Hello('sample initialization string')
    assert h.greetAsPresented() == 'sample initialization string'

def test_uppercase():
    h = hello.Hello('hello, world')
    assert h.greetUpper() == 'HELLO, WORLD'

def test_exclamation():
    h = hello.Hello('howdy')
    assert h.greetExclamation() == 'howdy!'

def test_question():
    h = hello.Hello('good day')
    assert h.greetQuestion() == 'good day?'

Ok, now we’ve got expectations. There will have to be a class called Hello with methods greetAsPresented, greetUpper, greetExclamation, and greetQuestion. To create the appropriate target for these tests we might write:

#!/usr/bin/python

class Hello:

        def __init__(self, greetStr):
                self.greetStr = greetStr

        def greetAsPresented(self):
                return self.greetStr

        def greetUpper(self):
                return self.greetStr.upper()

        def greetExclamation(self):
                return self.greetStr + '!'

        def greetQuestion(self):
                return self.greetStr + '?'

And a quick pytest:

$ pytest
==== test session starts =====
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/carl/pyjunk/testtxt
collected 4 items

test_hello.py ....                                                       [100%]

===== 4 passed in 0.01s =====

Bingo. All the tests passed. We have working code.

Installing pytest

First, check to see if pytest is already installed. “Which pytest” will work, or just type “pytest -h” on the command line and see if the pytest command can be found.

If not, the usual methods are the best.

Check to see if your yum/rpm/dnf/apt repos include pytest. If not, install with pip:

pip install -U pytest

If you don’t have pip, now’s a good time to bring that aboard, too. You can bootstrap pip and install pytest with:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install -U pytest

Mixing tests and production code

Not my favorite. In fact, don’t do this. It’s interesting to consider as a demonstration of how pytest finds tests. If you are dead set on one file including both your code and the tests that keep it safe, you can do that. You might want to roll everything into one file if there’s a chance your tests will be lost or discarded, but there are dangers.

Write your code with test functions mixed in. Your “real” function names must not start with “test,” but every test function will have to.

For instance, consider this version of hello.py:

#!/usr/bin/python

def greetUser(s):
        return s.capitalize() + '!'

def test_greetUser():
    assert greetUser('howdy') == 'Howdy!'

if __name__ == '__main__':
    print(greetUser('hello, world'))

If you run hello.py, it will print “Hello, world!” as expected. You can run your test functions with “pytest hello.py,” but there is something to be aware of.

If running your program creates a lot of output (or if your tests do anything destructive, like rewrite files), you won’t have a very nice test suite.

You can also trigger tests with a symlink:

ln -s hello.py test_hello.py

Both names, hello.py and test_hello.py, refer to the same file. Pytest, without adding any command line arguments, will find test_hello.py and run it like a test suite.

But please don’t mix tests with production code. Tests and functions can be mixed in a single file, but that’s an inadvisable idea. It’s instructive to know that it will work, but it’s a little ugly.

Building test data with JSON

If you have a function that returns complex data, it might be an error-prone and mind-numbing task to type a complex object by hand.

An alternative is to write expected answers to disk. Just don’t trust your code too much. Verify what you get.

Python’s pickle module is my go-to for saving data, but JSON is a better idea in this case.

JSON is human readable.

For instance, let’s write a function that returns a 26 member array of dictionaries, each containing values for one, two, three, and four character strings. This is an intentionally useless function. It’s an example, not a general theory of relativity. In the file genstuff.py:

#!/usr/bin/python

import json
from pprint import pprint

nameArray = ('One', 'Two', 'Three', 'Four')

def genStuff():
    retval = [] # return buffer
    for retMember in range(26):
        workingChar = chr(retMember + 0x61) # a-z
        retval.append({}) # add a dictionary to array
        for repCount in range(4): # make four strings
            strBuf = '' # initialize a buffer
            # count of characters to create
            for repeatChar in range(repCount + 1): 
                # add another copy of workingChar
                strBuf += workingChar 
            # set first character uppercase
            strBuf = strBuf.capitalize() 
            # add it to last dictionary in array
            retval[-1][nameArray[repCount]] = strBuf     return(retval)

That code outputs an array of 26 dictionaries. Each dictionary has four keys, One, Two, Three, and Four.

The values at those keys are one through four characters matching the dictionary’s position in the list:

[
    {
        "Four": "Aaaa",
        "One": "A",
        "Three": "Aaa",
        "Two": "Aa"
    },
    <snip>
    {
        "Four": "Zzzz",
        "One": "Z",
        "Three": "Zzz",
        "Two": "Zz"
    }
]

Quick show of hands – who wants to type 26 iterations of that by hand? Me either. That’s where the <snip> came from.

Let’s use this for a unit test called test_genstuff.py:

import genstuff # get the code to test
import json # for archived answers

def dump_genStuff():
    open('genstuff.dat', 'wt').write(json.dumps(genstuff.genStuff(), sort_keys = True, indent = 4))

def test_genStuff():
    dump_genStuff() ; assert True == False # keep commented out!
    genReturn = json.loads(open('genstuff.dat', 'rt').read())
    assert genstuff.genStuff() == genReturn

There’s a line that looks a little wonky.

dump_genStuff() ; assert True == False # keep commented out!

Winston Smith in Orwell’s 1984 might have thought True == False is business as usual, but there’s a less sinister motive here.

If that line isn’t commented out, it will create a file called genstuff.dat and immediately fail, and that’s what we want. We could use “raise SystemExit” just as well.

The function dump_genStuff writes a json-encoded version of genStuff’s output to a disk file. Let’s comment out that line asserting True equals False:

import genstuff # get the code to test
import json # for archived answers

def dump_genStuff():
    open('genstuff.dat', 'wt').write(json.dumps(genstuff.genStuff(), sort_keys = True, indent = 4))

def test_genStuff():
  # dump_genStuff() ; assert True == False # keep commented out!
  genReturn = json.loads(open('genstuff.dat', 'rt').read())
  assert genstuff.genStuff() == genReturn

Here’s the pytest run:

$ pytest
===== test session starts =====
platform linux -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /home/carl/pyjunk
collected 1 item

test_genstuff.py .                                                                             [100%]

====== 1 passed in 0.01s =====

Eureka! We ran pytest and everything passed!

But not so quick, sport. Before you can trust this, you have to look at genstuff.dat and make sure you’re not just comparing the last wrong answer it returned to the current wrong answer.

Here in the Southern tier of the US, we call that a “Bless your heart” moment.

Any time you need to recreate genstuff.dat, uncomment the line containing the call to dump_genStuff and True == False. Be sure to verify the new version of genstuff.dat is accurate, then re-comment that line.

Thanks for visiting, and don’t miss additional articles and Python examples while you’re here.

I hope my pytest musings awaken interest in routine unit testing. It’s as quick and easy as little snippets to test incremental progress, and it can save your bacon.

If your organization needs developers with strong foundations of verified application performance, drop me a line. I’ll have one of our placement representatives review our database of candidates and match you to…

Nah, just kidding. I could use a job, and I’d like the opportunity to contribute to your success, either through clear technical writing or software development.

I can be reached through the Get in Touch! link at the top of this page or by email to carl@carlhaddick.com.

Let me know how I can help!