Kivy recipe: selenium-like tests

I always wanted to have auto tests in kivy, and I made it work. I have 40 tests for kognitivo so far, and it looks pretty cool!

TL;DR: check out this repo, install dependencies and run with ./test.sh, enjoy.

Threading?

The first idea was to run kivy in one thread, and testing commands in another one, but it's not going to work, because:

  • I was too lazy to make exceptions from testing thread be propagated to stderr/stdout of the process, and they all got lost. But more important is
  • Kivy doesn't like time.sleep, and as much as we want to simulate user behaviour, it's not the way we can go. So what can we do?

Clock

The main reason of kivy's time.sleep allergy is that kivy has it's own scheduling mechanism. So wtf? Let's use it!

The problems start, when you try to make it usable. I have created bunch of decorators, but it's still not ideal.

First, we need @simulate, to wrap the test function and start it with some delay. Why? Because normally your app needs some time to initialize.

Second, we need some time gap between assertions and manipulations to let the application change its state (animations make the most butthurt), but we also can't make the thread just wait (see above). The only one solution is to call step after the previous one is complete.

Everything seems to be fine, but I didn't make it work. Honestly I can only remember some weird callback errors, so I realized it with an execution queue. Possibly it's not needed anymore.

App restart

Every single test case should be isolated from another ones. So it means, you need to restart the app for each test case. Yes, but:

  • If you try to say:
App.run()  
App.stop()  
App.run()  

you will get the errors, that video system is not initialized. I still don't know how and why works in my case :P

  • Kivy language builder won't flush the rules applied from the last start and you need to do it manually in tear down:
        from kivy.lang import Builder

        files = list(Builder.files)
        for filename in files:
            Builder.unload_file(filename)
        kivy_style_filename = os.path.join(kivy_data_dir, 'style.kv')
        if not kivy_style_filename in Builder.files:
            Builder.load_file(kivy_style_filename, rulesonly=True)

Py.test

I also wanted to use some cool pytest features like parametrization, but it's just workaround, what I did. You must decorate the test with empty decorator

@pytest.mark.parametrize("params", [{}])

and put parameters to your test, if you want to parametrize it like this

@pytest.mark.parametrize("params", [
    {"name": "LOGO", "family": None, "check_text": "ALL"},
    {"name": "ANALYTICS", "family": 'a'},
    {"name": "MEMORY", "family": 'm'},
    {"name": "ATTENTION", "family": 't'},
    {"name": "REACTION", "family": 'r'},
])
@simulate
def test_press_button(simulator, name, family, check_text=None, ):  
...

Also pytest-xdist plugin works like a charm, you can just specify number of processes and look at the magic.

Assertions and manipulations

I like xpath and there is no alternatives to make elements selection as flexible as it does. I have decided to build the copy of widgets tree using lxml and then select the elements for assertions and manipulations. It's also very useful to take some properties in the xml tree, so we can select elements more precisely. The problem is that properties could have all possible values, so I decided to get booleans and strings only.

It's awesome how much you can make with two things only: press simulation and checking if selector exists in the tree. I also realised tap (press and release) and wait. trigger_event method allows you to simulate all the kivy event types.assert_attr is also a quite useful assertion for checking if widget has attribute with particular value (and it's implication assert_text).

Example

Here is complex example of how I use it in kognitivo:

@pytest.mark.parametrize("params", [{}])
@simulate
@override_settings(ACTIVE_TASK_CLASSES=['ColorTable'],
                   DEFAULT_TASKS_COUNT=1,
                   INTRO_HINT_LENGTH=10)
def test_color_table(simulator):  
    to_task(simulator)

    simulator.assert_invisible("IntroHintColorTable", manager_selector="//TasksScreen/ScreenManager")
    simulator.assert_attr("//ColorTable//ColorTableDescriptionWidget", "hidden", False)

    from widgets.tasks.table_element.table_element import TableElement

    simulator.assert_count("//ColorTable//ColorTableTile", TableElement.SIZE ** 2)
    simulator.assert_text("//ColorTable//ColorTableTile[9]", "#000000ff")

    simulator.wait(TableElement.SHOW_TIME)

    simulator.assert_attr("//ColorTable//ColorTableDescriptionWidget", "hidden", True)
    simulator.assert_text("//ColorTable//ColorTableTile[9]/Label", "?")

    simulator.assert_visible("ColorTable//ColorTableAnswerWidget",
                             manager_selector="//TasksScreen/ScreenManager")
    simulator.assert_count("//ColorTable//ColorTableAnswerWidget//ColorAnswerButton", TableElement.SIZE)

    simulator.tap('//ColorTable//ColorTableAnswerWidget//ColorAnswerButton[1]')
    simulator.assert_attr('//ColorTable', 'successful', False)
    simulator.assert_attr('//ColorTable', 'mistakes_count', 1)

    simulator.tap('//ColorTable//ColorTableAnswerWidget//ColorAnswerButton[%s]' % TableElement.SIZE)
    simulator.assert_attr('//ColorTable', 'successful', True)

What's next?

I think there is no reason not to try to start this test on mobile devices: you can start py.test cases from the code. Imagine: you build your app with some special flag, connect all point devices to your laptop and deploy. The app starts on all the devices and auto tests itself. I like when it makes it "on its own".

I wouldn't mind to bring it into kivy itself. I'm not quite sure if py.test would fit existing testing subsystem in kivy, and when I clone kivy repo I open it, scare and close it. I have just too small head for that :P


comments powered by Disqus