💺 Blog

I’m now living the high life over at https://kde.haraldsitter.eu/

The other week the question came up how one can debug an application crash when the Windows Store crash tracking system is unable to produce a usable stack trace. Seemed a good enough opportunity to share some wisdom

Generally speaking in order to get a stack trace you first need a minidump. minidumps are kind of like core dumps on POSIX systems, well, except, mini. Acquiring that is should be your first goal.

There are a million ways to get a dump, I’ll highlight two of the easiest that I know of.

Ideally the Microsoft Partner Center will have a dump for you. You can find it usually where the stack trace is as well. To get access to KDE’s applications you need to be a KDE developer and file a sysadmin request. Once you have access you have to head from the Dashboard to Insights then navigate in the left hand pane to Health there use the drop-down to select the application you want to look at. This should give you every bit of information about the application health your heart could desire. You’ll probably also want to switch from viewing the last 72 hours to the last month, unless the application is particularly faulty of course.

Now all you need to do is click on the crash you want to look at, and not get too annoyed over the unknown crashes you can’t do anything about .

At this point you should be able to find a stack trace link and an additional download link. Sometimes the download link is not there, I have no idea why but I’m sure it’s documented somewhere. The download link is what we are after, it contains the minidump along with some other metadata.

Now, sometimes the partner center is not able to help us for whatever reason. Maybe the download link is missing, maybe it just doesn’t show the crash we are after, maybe the dump on the partner center is useless. Who knows. In that case we need some help from the user. Thankfully it’s not too painful. They need to enable collection of user-mode dumps by creating the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps, which then causes the Windows Error Reporting to throw a minidump into %LOCALAPPDATA%\CrashDumps. The user then needs to reproduce the crash and obtain the dmp file from the aforementioned location.

Once you have obtained a minidump it’s time to find us some debug symbols. The sad truth here is that I can’t really help with that. Depending on how your application was built you’ll be able to get PDBs somehow hopefully. They will either float around as PDBs somewhere or at the very least will be available inside the .appxupload or .appxsym zip files. As a general best practice for KDE software I would advise that when you do a binary release to the Windows Store you also release the x86_64-dbg.7z file to download.kde.org so we can get the relevant PDBs when needed.

Alright, I hope you had luck with finding your debug symbols, because now it’s time to do some tracing! Whee. You’ll need Microsoft Visual Studio. Any edition will do. File->Open->File... the minidump and you should be greeted by a nice overview of metadata about the crash.

First we’ll want to setup our debug symbols. For that you first want to place your PDBs somewhere in convenient in your file system. I’m lazy and usually just slap them on the Desktop. In Visual Studio you should find the option Set symbol paths in the right hand list of actions. The option opens the settings window on the correct page. Simply hit the and type out the path where you extracted the PDBs.

Once the symbol paths are set up you can hit Debug with Mixed and off the tracer goes. Slowly, because it needs to download a million symbols. But eventually you’ll arrive at your stack trace.

(nevermind my crazy setup, I was doing some wonky multi threaded debugging earlier and don’t know how to restore the UI )

Hope this helps some!

…very hard.

KDE relies heavily on web services, and many of them need to be kept responsive even under strenuous load. I’ve recently had the opportunity to spend some time on load testing one of our websites and would like to share how that worked out.

To properly test things I wanted to have multiple computers make concurrent requests to the service and ensure that the service still performs within acceptable limits. To that end I needed a bunch of servers, software that can pressure the web service, and software that can make sure the service works.

The server bit is the easiest task there… any cloud provider will do.

The software also seemed easy. After very quick research Locust seemed as good a choice as any to poke at the service and make sure it responds. Except, after some pondering I came to realize that this is actually not so. You see, Locust does HTTP performance testing. That is: it makes HTTP requests and tracks their response time / error rate. That is amazing for testing an API service, but when dealing with a website we also care about the javascripty bits on top being responsive. Clearly a two-prong approach was necessary here. On the one hand Locust can put pressure on the backend and then something else can poke the frontend with a stick to see if it is dead. Enter our old friend: Selenium. An obvious choice given my recent work on a Selenium-based application testing framework. The advantage here is that Selenium can more or less accurately simulate a user using the website giving us a fairly good idea about perceived performance being up to spec. Better yet, both Locust and Selenium have master/client architectures whereby we can utilize the cloud to do the work while a master machine just sits there orchestrating the show.

The three building blocks I’ve arrived at are:

• A cloud to scale in
• Locust for performance testing (that the thing stays responsive)
• Selenium for interaction testing (that the thing actually “works”)

I actually thought about showing you some code here, but it’s exceptionally boring. You can go look at it at https://invent.kde.org/sitter/load.

At first I needed to write some simple tests for Locust and Selenium. They were fairly straight forward, a bit of login, a bit of logout. Just to start putting pressure on the server-under-test.

With simple tests out of the way it was time to glue everything together. For this I needed a couple more puzzle pieces. I’ve mentioned that both Locust and Selenium have “server” components that can manage a number of clients. For Locust that is distributed load generation, and for Selenium it’s called a Grid. For convenience I’ve opted to manage them using docker-compose.

The last piece of the puzzle was some provisioning logic for the cloud server to install and start Selenium as well as Locust Workers.

When all the pieces were in place amazing things started happening!

On my local machine I had a Selenium Grid and a Locust master running. Magically, cloud servers started connecting to them as workers and after a while I didn’t have the Selenium and Locust power of one machine, no, UNLIMITED POWER! (cheeky Star Wars reference).

By starting a load test in Locust it was distributed across all available nodes, simulating more concurrent access than one machine would or could ordinarily do.

A simple loop also starts a number of concurrent Selenium tests that get distributed across the available grid nodes.

for i in {1..5}; python3 test.py& done

The end result is a Locust making hundreds of requests per second while Selenium is trying to use the UI. Well, for a while anyway… I naturally wanted to know the limits so I kept bumping the request numbers. At first all was cool.

Response times in the sub 100ms at 300 users is pretty good I think. CPU usage was also at a comfortable level.

So I increased it to 600 users, which was still OKish. But when I started going towards 1200 users the problems started to appear.

In the bottom graph you can see the two bumps from 300 to 600 and then to 1200. What you can see super clearly is how the response time keeps getting poorer and poorer the difference is so enormous that you almost can’t make out the response time changes from 300 to 600 anymore. Eventually the service started having intermittent interruptions when the Selenium tests were also trying to get their work done. Yet CPU and memory were not fully at capacity – In particular the intermittent failure hike is very suspicious. A look at the failures gave the hint: it was running into software limits. I bumped up the limits because the hardware still had leeway, and presto: no more failure spikes even when at 2048 users. Hooray! Response time does suffer though, so in the end there would need to be more reshuffling if that was an expected user count.

Knowing the limits of our services is very useful for a number of reasons, ranging from knowing how many users we can support, to how oversized our servers are for the task they are performing, to whether our service is susceptible to malicious use. Knowing the capabilities and weaknesses of our systems helps us ensure high availability.

To discuss this blog post check out KDE Discuss.

People keep lamenting how lackluster Qt’s SVG renderer is. It leads to poorly rendered icons and wallpapers and it mostly only implements the SVG Tiny specification. As a weekend project I put together a resvg based image handler replacement. It was super easy because resvg is amazing!

An application that uses QSvgRenderer actually can, for the most part, just use resvg’s Qt header and call it a day. It has a very convenient interface that lets you replace most uses of QSvgRenderer with resvg’s. That’s all lovely, except who wants to port every application to a new library.

Luckily, most applications don’t actually use QSvgRenderer directly, they use it through 2 plugins. The iconengine plugin, providing a QIconEngine. And the imageformats plugin, providing a QImageIOHandler. Understanding when which is used is the first step.

Duh! The icon engine is used for icons! – Well, yes, usually…

The SVG QIconEngine confusingly isn’t actually used on KDE Plasma, instead we route all icon lookups through KIconEngine which does some magic. Ah! But where does it then get the icon from? If we follow the code paths a bit we end up here and after some reading on QImageReader we learn that on KDE Plasma icon lookup is actually not running through the SVG iconengine plugin but the imageformats plugin. Slightly confusing but actually beneficial because it means that icon rendering is ultimately running through the same code paths as regular SVG image reading when e.g. loading an Image in QML.

By creating an imageformats plugin we can replace most uses of QSvgRenderer with resvg without having to touch every application. Hooray!

And the good news keeps on coming. A basic QImageIOHandler is super easy to implement.

    bool canRead() const override;
    bool read(QImage *image) override;

All we need to do is implement reading

    *image = ResvgRenderer(device()->readAll(), ResvgOptions()).renderToImage();

That gets us a working plugin. There are some extra features one can and should implement but for the most part that is all that’s needed. You can check out the complete source to see where things are at.

Now that we have a plugin we just need to make Qt actually use ours instead of its own. Unfortunately this is where our luck runs out. Qt appears to have no facilities for manipulating which imageformats plugin is used when there are multiple candidates for the same format. In our case we have qsvg and qresvg both supporting SVG and it appears undefined behavior which of the two gets used. So, for the time being we’ll have to overwrite qsvg to get our plugin to reign supreme. Somewhat unfortunate.

Plugins are cool. Resvg is also cool. Making a resvg plugin is double cool.

You can give it a try at https://invent.kde.org/sitter/qresvgimageiohandler

Here’s what it looks like. On the left hand side you can see the image from a bug report where the image viewer icon is misrendered, on the right you can see the same icon rendered correctly using resvg.

To discuss this blog post head on over to KDE Discuss.

At KDE we make software for many different platforms. One of them is Microsoft Windows. But what if an application crashes on Windows? New tech enables us to track crashes right in Sentry! Time to learn about it.

When an application crashes on Windows the user can submit crash data to Microsoft. Later KDE, as publisher of the app, can retrieve the crashes from there. This is the standard crash handling for the platform and it works incredibly well. What’s more, it means we don’t need to engineer our own custom solution for the entire process. So, that is all lovely.

Alas, since we are rolling out a KDE-wide crash tracking system called Sentry it would be even nicer if we had Windows crashes in there rather than third party service. That is just what I’ve built recently.

Crashes for our Windows applications now get imported into Sentry!

There are two pieces to this puzzle. One is a symstore and one is the actual importer.

In the Linux world we not so long ago grew debuginfod, it’s basically a tiny service that lays out debug symbols in a standardized file path so it may be consumed by any number of tools by making HTTP GET requests to well-known URIs.

Windows has something like this called symbol storage. It’s incredibly simple and merely lays out Windows debug information in a well defined path setup for consumption in tools.

To maintain a symstore of our own we use the excellent python module symstore along with some custom rigging to actually retrieve our debug information files.

The second piece is our importer. It’s a trivial program that uses Microsoft’s APIs to retrieve crash data along with the minidump (the dump format used by Windows crashes), converts it into a Sentry payload, and sends it off to our Sentry instance. Sentry then consumes the crash, does symbolication using the debug symbols from our symstore, and generates a crash event.

If you are shipping your application on the Windows Store you should consider publishing your dbg.7z files on download.kde.org (for example here is Okular), simply file a sysadmin ticket same as with any other upload to get this published. Once that is done you can submit a change to our symstore to publish the debug symbols, and our importer to import data into Sentry. If you need help just give me a poke.

If you are publishing builds for Windows but are not yet on the Window Store, you definitely should change that. It increases your application’s reach considerably. Get in touch with sysadmins to get yourself set up.

Importing crashes into Sentry adds another piece towards higher quality, more actionable crash reports. It’s going to be amazing.

Discuss this blog post on KDE Discuss.

Every once in a while I hear developers having trouble staying up to date with KDE Discuss(ions). No great surprise when I then find out they aren’t tracking things (in other words: not subscribed to categories)!

Discourse, the software that powers KDE Discuss, has a number of handy notifications settings that we can use to have the software bring posts to us instead of having to go to the website every once in a while. We can use these options to tailor a good experience for ourselves.

Three setting types are relevant for the notification experience:

Tracking preferences control which discussions to track. Settings here range from when automatic tracking happens – such as when you view a topic for a couple minutes it automatically becomes tracked – to ignoring things. You can track categories as well as tags. Say you only care about kdenlive you can simply track the kdenlive tag and you are good to go.

Emails contain the second most relevant pool of preferences. There are two principal modes of operation:

• Forum mode
• Mailing list mode

The names should be obvious enough, the functionality probably less so.
In forum mode tracking preferences apply and you get emails for posts or threads as configured.
In mailing list mode only negative tracking preferences apply! You get an email for every post on the entire site unless you’ve opted to mute something.
In both modes you can simply reply by email to post a reply.

These preferences have to do with live notifications i.e. if you have a browser open it will throw up a system notification when new posts happen. The settings here aren’t terribly interesting unless you always have a browser tab pointing to KDE Discuss. If you do, you can set a time frame for notifications such that you don’t get notifications when you are at work-work for example.

So, what do I recommend you do?

First off, if you want to keep things low-volume make sure you have the email preference Activity Summary enabled. It sends a summary email once a day with tracked topics. You can disable most everything else. You definitely want to set up tracking for what you want to receive in your summary (unless of course you want to see all threads). e.g. if you only care about dev topics then set the Development category to tracking.

One step up from the summary is actually the default mode of KDE Discuss. By default if you simply configure tracking as desired you’ll get a reasonable mix of notifications and a summary if you haven’t been around in a while.

The most intense setup is to activate mailing list mode. This operates like a mailing list and causes the highest volume of emails. You’ll want to set up email filtering to go along with this. I would definitely not recommend this unless you really care about all discussions on the entire site.

You should also note that you can reply by mail to threads, regardless of whether mailing list mode is enabled. It’s amazing!

And lastly: you can configure most anything, so if you find that you are getting too many emails or too few just head to the preferences and play with some of the options. I’m confident everyone is able to find a good mixture that makes the most out of KDE Discuss

Discuss this blog post on KDE Discuss.

Akademy, KDE’s annual conference, recently took place in Thessaloniki, Greece. Lots of people were super excited about the prospect of getting GUI Testing off the ground based on the Selenium tech I built last year. Since KDE produces cross-platform applications an obvious question arose though…

What about Windows?

It’s surprisingly easy! Indeed the most time consuming part is probably getting your hands on a Windows Development Virtual Machine. Once you have a Windows installation we need to only spin up our toolchain and off we go. Here’s a handy command list:

# Download and install WinAppDriver: https://github.com/microsoft/WinAppDriver/releases

winget install openjs.nodejs
winget install python.python.3.11

npm install --location=global appium
# restart terminal to apply PATH change
set-executionpolicy -scope currentuser remotesigned # allow script execution
appium driver install --source=npm appium-windows-driver 

pip install appium-python-client
appium # start server, needs firewall exception on first start

Before we go further into the nitty gritty of testing on Windows I suggest you read the earlier blog post Selenium + AT-SPI = GUI Testing, since a lot of the concepts are the same regardless of platform.

First let us get our ducks in a row.

What Accerciser is to Linux is inspect.exe to Windows, namely an inspector tool for applications. You can find it in your Windows SDK folder %ProgramFiles(x86)%\Windows Kits\10\bin\10.0.22000.0\x64\inspect.exe or there abouts. Opening it greets you with this beauty:

Ignoring the verbosity for a moment we’ll note that it contains similar information to Accerciser on Linux, albeit in a more flat overview. Most importantly what is called the AutomationId is constructed from QObject objectNames, similar to the Accessible IDs on Linux. This is insofar interesting as it means we have a couple of avenues for cross-platform element locating – specifically we could match elements by their name (e.g. the text of a Label or Button), or more uniquely by their objectName-based ID (applicable to all QObjects).

For the purposes of this post we’ll do some trivial testing on Filelight and try to make it work for both Linux and Windows by using the element names. Relying on objectNames is more reliable but unfortunately requires some retrofitting in the source code. To avoid having to build Filelight on Windows we’ll work with what we have got: names. Let’s write our test. Don’t forget to install Filelight first

First thing, as always, is our setup boilerplate

#!/usr/bin/env python3

# SPDX-License-Identifier: MIT
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>

import unittest
import sys
from appium import webdriver
from appium.options.windows import WindowsOptions
from appium.options.common import AppiumOptions
from appium.webdriver.common.appiumby import AppiumBy
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

class SimpleFilelightTests(unittest.TestCase):
    @classmethod
    def setUpClass(self):
        options = WindowsOptions()
        options.app('KDEe.V.Filelight_7vt06qxq7ptv8!KDEe.V.Filelight')
        self.driver = webdriver.Remote(
            command_executor='http://127.0.0.1:4723',
            options=options)

    @classmethod
    def tearDownClass(self):
        self.driver.quit()

if __name__ == '__main__':
    unittest.main()

The only really interesting bit here is how we specify the application on Windows. Since Filelight is a store application we can start it by the Application User Model ID (AUMID) instead of a path; this is pretty much the same as starting by-desktop-file-id on Linux.

Now then. With the boilerplate out of the way we can write an incredibly simple test that simply switches pages a bit: If we click on ‘Scan Home Folder’ it should take us to the scan page and clicking there on ‘Go to Overview’ should take us back to the overview page.

    def test_scan(self):
        self.driver.find_element(by=AppiumBy.NAME, value="Scan Home Folder").click()
        overview = WebDriverWait(self.driver, 120).until(
            EC.element_to_be_clickable((AppiumBy.NAME, "Go to Overview"))
        )
        overview.click()
        WebDriverWait(self.driver, 4).until(
            EC.element_to_be_clickable((AppiumBy.NAME, "Scan Home Folder"))
        )

Cool. We now can test Filelight on Windows. Next we should try to make this test also work for Linux. Thankfully we only need to switch out our app name for a desktop file id.

    def setUpClass(self):
        if sys.platform == 'nt':
            options = WindowsOptions()
            options.app = 'KDEe.V.Filelight_7vt06qxq7ptv8!KDEe.V.Filelight'
        else:
            options = AppiumOptions()
            options.set_capability('app', 'org.kde.filelight.desktop')

Putting it all together we get our final test which runs on both Linux and Windows.

# Windows
# start appium in a terminal
python .\test.py

# Linux
selenium-webdriver-at-spi-run ./test.py

The complete test code:

#!/usr/bin/env python3

# SPDX-License-Identifier: MIT
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>

import unittest
import sys
from appium import webdriver
from appium.options.windows import WindowsOptions
from appium.options.common import AppiumOptions
from appium.webdriver.common.appiumby import AppiumBy
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

class SimpleCalculatorTests(unittest.TestCase):

    @classmethod
    def setUpClass(self):
        if sys.platform == 'nt':
            options = WindowsOptions()
            options.app = 'KDEe.V.Filelight_7vt06qxq7ptv8!KDEe.V.Filelight'
        else:
            options = AppiumOptions()
            options.set_capability('app', 'org.kde.filelight.desktop')

        self.driver = webdriver.Remote(
            command_executor='http://127.0.0.1:4723',
            options=options)

    @classmethod
    def tearDownClass(self):
        self.driver.quit()

    def test_scan(self):
        self.driver.find_element(by=AppiumBy.NAME, value="Scan Home Folder").click()
        overview = WebDriverWait(self.driver, 120).until(
            EC.element_to_be_clickable((AppiumBy.NAME, "Go to Overview"))
        )
        overview.click()
        WebDriverWait(self.driver, 4).until(
            EC.element_to_be_clickable((AppiumBy.NAME, "Scan Home Folder"))
        )

if __name__ == '__main__':
    unittest.main()

Discuss this blog post on KDE Discuss.

Flatpaks are amazing and all that. But application sandboxing, so an application cannot do anything it wants, is a challenge – even more so when you have two applications that need to talk to each other. Perhaps it shouldn’t come as a surprise that native-messaging sandboxing support for Flatpak has been in development for over a year. To celebrate its anniversary I thought I’d write down how to drill a native-messaging sized hole into the sandbox. This enables the use of native messaging even without portal integration, albeit also without sane degrees of sandboxing.

First off, please understand that this undermines the sandbox on a fairly fundamental level. So, don’t do this if you don’t keep your Firefox updated or visit particularly dodgy websites.

For the purposes of this post I’m assuming Firefox and KeePassXC are installed as Flatpaks in user scope.

First order of business is setting up KeePassXC so it writes its definition file in a place where Firefox can read it. Fortunately it has a setting for this:

~/.var/app/org.mozilla.firefox/.mozilla/native-messaging-hosts/ is the path inside Firefox’ home where the defintion file will be written. Naturally we’ll also need to adjust the Flatpak permissions so KeePassXC can write to this path.

flatpak override --user --filesystem=~/.var/app/org.mozilla.firefox/.mozilla/native-messaging-hosts org.keepassxc.KeePassXC

At this point Firefox knows about the native messaging host but it won’t be able to run it. Alas. We need some rigging here. The problem is that Firefox can’t simply flatpak run the native messaging host, it needs to spawn a host process (i.e. a process outside its sandbox) to then run the KeePassXC Flatpak and that then runs the NMH.

Fortunately the NMH definition files are fairly straight forward:

{"allowed_extensions":["keepassxc-browser@keepassxc.org"],
"description":"KeePassXC integration with native messaging support",
"name":"org.keepassxc.keepassxc_browser",
"path":"/home/me/.local/share/flatpak/exports/bin/org.keepassxc.KeePassXC",
"type":"stdio"}

The problem of course is that we cannot directly use that Flatpak bin but need the extra spawn step in between. What we need is a way to manipulate the definition file such that we can switch in a different path. systemd to the rescue!

systemctl edit --user --full --force keepassxc-native-messaging-mangler.path

# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>

[Path]
PathChanged=/home/me/.var/app/org.mozilla.firefox/.mozilla/native-messaging-hosts/org.keepassxc.keepassxc_browser.json

[Install]
WantedBy=default.target

and the associated service file…

systemctl edit --user --full --force keepassxc-native-messaging-mangler.service

# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>

[Unit]
Description=keepassxc mangler

[Service]
ExecStart=/home/me/keepassxc-native-messaging-mangler

lastly, enable the path unit.

systemctl --user enable --now keepassxc-native-messaging-mangler.path

Alright, there’s some stuff to unpack here. KeePassXC on startup writes the aforementioned definition file into Firefox’ NMH path. What we do with the help of systemd is monitor the file for changes and whenever it changes we’ll trigger our service, the service runs a mangler to modify the file so we can run another command instead. It’s basically an inotify watch.

Here’s the mangler (~/keepassxc-native-messaging-mangler):

#!/usr/bin/env ruby
# frozen_string_literal: true

# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>

require 'json'

file = "#{Dir.home}/.var/app/org.mozilla.firefox/.mozilla/native-messaging-hosts/org.keepassxc.keepassxc_browser.json"
blob = JSON.parse(File.read(file))
blob['path'] = "#{Dir.home}/Downloads/keepassxc"
File.write(file, JSON.generate(blob))

It simply replaces the path of the executable with a wrapper script. Here’s the wrapper script (~/Downloads/keepassxc):

#!/bin/sh

# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>

exec /usr/bin/flatpak-spawn --host --watch-bus "$HOME/.local/share/flatpak/exports/bin/org.keepassxc.KeePassXC" "$@"

flatpak-spawn is a special command that allows us to spawn processes outside the sandbox. To gain access we’ll have to allow Firefox to talk with the org.freedesktop.Flatpak DBus session service.

flatpak override --user --talk-name=org.freedesktop.Flatpak org.mozilla.firefox

And that’s it!

KeePassXC writes its NMH definition to Flatpak specific path
systemd acts on changes and starts mangler
mangler changes the path inside the definition to our wrapper
Firefox reads the definition and calls our wrapper
wrapper flatpak-spawns KeePassXC flatpak
Firefox (flatpak) talks to KeePassXC (flatpak)

At KDE we have multiple levels of quality assurance ranging from various degrees of a humans testing features to fully automated testing. Indeed automated testing is incredibly important for the continued quality of our software. A big corner stone of our testing strategy are so called unit tests, they test a specific piece of our software for its behavior in isolation. But for many aspects of our software we need a much higher level view, testing pieces of Plasma’s application launcher in isolation is all good and well but that won’t tell us if the entire UI can be easily navigated using the keyboard. For this type of test we require a different testing approach altogether. A couple months ago I’ve set set out to create a testing framework for this use case and I’m glad to say that it has matured enough to be used for writing tests. I’d like to walk you through the technical building blocks and a simple example.

Let us start of by looking at the architecture at large. So… there’s Selenium which is an incredibly popular, albeit web-oriented, testing framework. Its main advantages for us are its popularity and that it sports a server-client split. This means we can leverage the existing client tooling available for Selenium without having to write anything ourselves, we only need to grow a server. The server component, called a WebDriver, implements the actual interaction with UI elements and is generic enough to also apply to desktop applications. Indeed so thought others as well: there already exists Appium – it extends Selenium with more app-specific features and behaviors. Something for us to build upon. The clients meanwhile are completely separate and talk to the WebDriver over a well defined JSON REST protocol, meaning we can reuse the existing clients without having to write anything ourselves. They are available in a multitude of programming languages, and who knows maybe we’ll eventually get one for writing Selenium tests in QML

That of course doesn’t explain how GUI testing can work with this on Linux. Enter: AT-SPI. AT-SPI is an accessibility API and pretty much the standard accessibility system for use on Linux. Obviously its primary use is assistive technologies, like the screen reader Orca, but to do its job it essentially offers a toolkit-independent way of introspecting and interacting with GUI applications. This then gives us a way to implement a WebDriver without caring about the toolkit or app specifics. As long as the app supports AT-SPI, which all Qt apps do implicitly, we can test it.

Since all the client tooling is independent of the server all we needed to get GUI testing going was a WebDriver that talks to AT-SPI.

That is what I set out to write and I’m happy to announce that we now have an AT-SPI based WebDriver, and the first tests are popping into existence already. There is also lovely documentation to hold onto.

So, without further ado. Let us write a simple test. Since the documentation already writes one in Python I’ll use Ruby this time around so we have some examples of different languages. A simple candidate is KInfoCenter. We can test its search functionality with a couple of lines of code.

First we need to install selenium-webdriver-at-spi, clone it, cmake build it, and cmake install it. You’ll also need to install the relevant client libraries. For ruby that’s simply running gem install appium_lib.

Then we can start with writing our test. We will need some boilerplate setup logic. This is more or less the same for every test. For more details on the driver setup you may also check the wiki page.

  def setup
    @appium_driver = Appium::Driver.new(
      {
        'caps' => { app: 'org.kde.kinfocenter.desktop' },
        'appium_lib' => {
          server_url: 'http://127.0.0.1:4723',
          wait_timeout: 10,
          wait_interval: 0.5
        }
      }, true
    )
    @driver = @appium_driver.start_driver
  end

The driver will take care of starting the correct application and make sure that it is actually running correctly. Next we’ll write the actual test. Let’s test the search. The first order of business is using a tool called Accerciser to inspect the AT-SPI presentation of the application. For more information on how to use this tool please refer to the wiki. Using Accerciser I’ve located the search field and learned that it is called ‘Search’. So, let’s locate it and activate it, search for the CPU module:

  def test_search
    search = driver.find_element(:name, 'Search')
    search.click
    search.send_keys('cpu')

Next let us find the CPU list item and activate it:

    cpu = driver.find_element(:class_name, '[list item | CPU]')
    assert(cpu.displayed?)
    cpu.click

And finally let’s assert that the page was actually activated:

    cpu_tab = driver.find_element(:class_name, '[page tab | CPU]')
    assert(cpu_tab.displayed?)

To run the complete test we can use the run wrapper: selenium-webdriver-at-spi-run ./kinfocentertest.rb (mind that it needs to be +x). If all has gone well we should get a successful test.

Finished in 1.345276s, 0.7433 runs/s, 1.4867 assertions/s.

1 runs, 2 assertions, 0 failures, 0 errors, 0 skips
I, [2022-12-14T13:13:53.508516 #154338]  INFO -- : tests done
I, [2022-12-14T13:13:53.508583 #154338]  INFO -- : run.rb exiting true

This should get you started with writing a test for your application! I’ll gladly help and review your forthcoming tests.
For more detailed documentation check out the writing-tests wiki page as well as the appium command reference.

Of course the work is not done. selenium-webdriver-at-spi is very much still a work in progress and I’d be glad for others to help add features as they become needed. The gitlab project is the place for that.

The complete code of the example above:

#!/usr/bin/env ruby
# frozen_string_literal: true

# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
# SPDX-FileCopyrightText: 2022 Harald Sitter <sitter@kde.org>

require 'appium_lib'
require 'minitest/autorun'

class TestKInfoCenter < Minitest::Test
  attr_reader :driver

  def setup
    @appium_driver = Appium::Driver.new(
      {
        'caps' => { app: 'org.kde.kinfocenter.desktop' },
        'appium_lib' => {
          server_url: 'http://127.0.0.1:4723',
          wait_timeout: 10,
          wait_interval: 0.5
        }
      }, true
    )
    @driver = @appium_driver.start_driver
  end

  def teardown
    driver.quit
  end

  def test_search
    search = driver.find_element(:name, 'Search')
    search.click
    search.send_keys('cpu')

    cpu = driver.find_element(:class_name, '[list item | CPU]')
    assert(cpu.displayed?)
    cpu.click

    cpu_tab = driver.find_element(:class_name, '[page tab | CPU]')
    assert(cpu_tab.displayed?)
  end
end

It’s a Plasma widget that visualizes what’s going on on your system, music-wise that is. I’ve started this project years ago but only recently found the motivation to get it to a somewhat acceptable state. It’s pretty amazing to have bars flying across the screen to Daft Punk’s `Touch`.

https://store.kde.org/p/1953779