Blog

iOS UI Testing and Why It Does Not Always Work: a.k.a. Pushing The Limits of the XCTest Framework

14 Feb, 2017
Xebia Background Header Wave

XCTest UI Testing

When Apple introduced the new UI automation framework in 2015 we became excited. We could finally test the UI and the behaviour of our applications using Swift (or Objective-C) and the native library.

Previously, we had to write scripts in JavaScript and execute them using Instruments. This was not the easiest way to write, maintain, and debug tests. Another option was to use the KIF framework, which allows us to write integration tests and to test the UI of the application. One major downside of this is the usage of the private API which could change every iOS / Xcode version. Thankfully, since Apple introduced the new framework things began to change…but did they really?

When you start playing with it for a longer period of time you will notice that the behaviour of some methods is different depending on the iOS and/or Xcode version. But this is something Apple has been used to for some time now. Although, the real fancy things are just starting to appear. When you finally manage to write working UI tests, and you reach the magical number of 128 test cases you will find the first Easter egg.

Pseudo Terminal Limit

At the beginning, you won’t notice anything. Run Xcode, open the project, execute tests, and everything will work as expected. But when you configure Fastlane, Travis CI, Jenkins, the Xcode Server, or any other Continuous Integration system to build your project you will notice that after around test 128 all of the tests begin to fail and the same error occurs

Error Domain=IDEPseudoTerminalDomain Code=1 "(null)"

This issue is very interesting because it does not occur on Xcode. It only occurs if you are executing xcodebuild directly (or by using wrappers like gym), so debugging the issue is not straightforward.

I searched for a solution on Google, but unfortunately I did not manage to find one. I only came upon information regarding some issues reported in 2014 without any results. Resetting the iOS Simulator did not help either, so I came up with the idea to conduct some research, more specifically, to dive into Xcode and try to find my own solution.

At the beginning I had no idea how to debug this issue. Everything was working on Xcode so I was unable to use any breakpoints or a debugger. My first approach was to execute only the failing tests by using an -only-testing flag. This did not help. I managed to execute the tests without any problems. My next thought was that the execution order can have something to do with it. Disabling tests from the scheme did not help either. After reaching 128 tests, they started to fail. After a few tries, I grew tired and felt that I have had enough. We had written almost 150 tests already. Executing the whole test suite took almost an hour. Checking a few cases took me the whole day.

In the next step, I wanted to speed up the test execution. I hoped that this issue would not be not related to my project, so I created a small test project. This simple application has 200 UI tests which do not do anything:

func testExample000(){}
func testExample001(){}
func testExample002(){}
// ...
func testExample199(){}

I executed the tests and "thankfully" they failed and the same error message appeared. This was proof that this issue was not related to my project, but rather to the xcodebuild tool.

After that, I started looking for a more general solution. If the error is related to some resource, here to the pseudo terminals, maybe I reached some kind of limit? After conducting a short investigation on Google, I found that macOS has a default limit for ptys set to 127.

$ sysctl kern.tty.ptmx_max
kern.tty.ptmx_max: 127

So it seems we have found the cause! But what is the source? Which process creates so many pseudo terminals?

When we execute a test from Xcode we can use lsof to see how many terminals are already allocated.

$ lsof /dev/ptmx

COMMAND   PID   USER   FD   TYPE DEVICE   SIZE/OFF NODE NAME
iTerm2   1688   user    0u   CHR   15,0    0t13596  572 /dev/ptmx
iTerm2   2164   user    0u   CHR   15,1 0t71369123  572 /dev/ptmx
Xcode    2173   user   31u   CHR   15,6     0t7887  572 /dev/ptmx
Xcode    2173   user   49u   CHR   15,5        0t0  572 /dev/ptmx
Xcode    2173   user   54u   CHR   15,8        0t0  572 /dev/ptmx
Xcode    2173   user   56u   CHR   15,7        0t7  572 /dev/ptmx

Xcode creates about 5 pseudo terminals and keeps this number constant. When we execute a test from xcodebuild we can notice a more interesting output:

COMMAND     PID   USER   FD   TYPE DEVICE   SIZE/OFF NODE NAME
iTerm2     1688   user    0u   CHR   15,0    0t13596  572 /dev/ptmx
iTerm2     2164   user    0u   CHR   15,1 0t71369123  572 /dev/ptmx
xcodebuil  8567   user   21u   CHR   15,6    0t22726  572 /dev/ptmx
xcodebuil  8567   user   23u   CHR   15,6    0t22726  572 /dev/ptmx
xcodebuil  8567   user   29u   CHR   15,5        0t0  572 /dev/ptmx
xcodebuil  8567   user   31u   CHR   15,5        0t0  572 /dev/ptmx
xcodebuil  8567   user   32u   CHR   15,7        0t0  572 /dev/ptmx
xcodebuil  8567   user   34u   CHR   15,7        0t0  572 /dev/ptmx

...

xcodebuil  8567   user  143u   CHR  15,45        0t0  572 /dev/ptmx
xcodebuil  8567   user  145u   CHR  15,45        0t0  572 /dev/ptmx
xcodebuil  8567   user  146u   CHR  15,46        0t0  572 /dev/ptmx
xcodebuil  8567   user  148u   CHR  15,46        0t0  572 /dev/ptmx

Each test creates a new terminal which is not deallocated. Around test 128, when the last allowed pty is spawn, Xcode is not able to start a new process on the Simulator and the rest of the tests fail.

This issue takes place only on the iOS Simulator. iOS Devices are not affected by this problem.

The First Workaround

Obviously the Apple developer forgot to release something after every test and we are unable to fix this issue without their help. So the only possibility is some kind of workaround. Thankfully, we can increase the maximum number of pseudo terminals to 999.

sudo sysctl -w kern.tty.ptmx_max=999

When using sysctl the change is only temporary (until the system is restarted). To permanently save the value, create or open the file /etc/sysctl.conf by using:

sudo touch /etc/sysctl.conf
sudo chown root:wheel /etc/sysctl.conf
sudo chmod 644 /etc/sysctl.conf

echo "kern.tty.ptmx_max=999" | sudo tee -a /etc/sysctl.conf

By changing ptmx_max we can now test up to 999 test cases in a single scheme, but only in theory…

App Accessibility

When we start testing once again we will find another interesting issue. Around test 170 Xcode or xcodebuild welcome us with the following error:

Waiting for accessibility to load
Wait for app to idle
    App event loop idle notification not received, will attempt to continue.
    App animations complete notification not received, will attempt to continue.

After this error , all of the next tests fail with another message:

Waiting for accessibility to load
    Assertion Failure: testUITests.swift:21: UI Testing Failure - App accessibility isn't loaded

This bug is more interesting than the previous one. It does not only happen for xcodebuild, but also for Xcode. What is even worse, is that it happens on both the iOS Simulator and the iOS device. Unfortunately Google does not help. There are topics on the Apple Developer Forum, but no solutions.

To investigate this issue, I set up the "Test Failure" and "Exception" breakpoints to catch the problem as soon as possible. The retrieved call stack does not contain anything useful. I also used fs_usage to track the created process and lsof to check how many resources were used by the Simulator and the Xcode tools:

sudo fs_usage -f exec

sudo lsof -p 

Unfortunately, without any results. "App accessibility" is not a separate process, so it is not a problem connected with launching an app. lsof has shown about 700 open files, while macOS has a limit of 10240 open files per process (sysctlkern.maxfilesperproc). Bash sets this limit to 4864 open files (ulimit -S -n). Both values are not even close to what Xcode allocates.

I was not able to find a cause or a fix.

The Second Workaround

Without a good fix the only possibility is to create a workaround. The only one we came up with at PGS Software would be to create more shared schemes in the project. I would suggest to add no more than 160 test cases into one scheme.

Update

On Xcode 8.3 beta the situation looks a little bit better. The tests fail after 206 test cases. Three more years and maybe we will be able to run 300 tests.

Update 14.06.2017

On Xcode 9.0 beta it’s even better. The tests fail after 256 test cases. Apple is on the right track to reach 300 soon.

Summary

Until Apple finally fixes those, and probably more issues, writing and executing UI tests will be problematic for all iOS developers. For now, we can only look for workarounds and keep our fingers crossed that Apple will finally create a working and stable UI automation tool.

Links

Questions?

Get in touch with us to learn more about the subject and related solutions

Explore related posts