I recently finished a project to automate the deployment of my iOS hobby app Weather Hunt to the App Store with a CI pipeline. In the end it looks like this:

  1. Check in the code.
  2. Click ‘deploy’ in the CI pipeline after all the tests pass
  3. Review and Submit the fully prepared update via App Store Connect.

Behind the scenes this is all possible through:

  1. XCUIApplication based test automation
  2. Fastlane and its rich ecosystem of plugins
  3. Gitlab CI (see my post on The Best CI).

For this post let’s dive into XCUIApplication based test automation.

Background: UI Testing in XCode

UI Tests in general are notoriously brittle. UI Tests for iOS are no exception. Actually, be prepared for your UI tests to be quite a bit more quirky and inconsistent than any other UI testing frameworks you have used in the past.

Why bother with UI testing in iOS with its lackluster reputation? Well, if you get it right:

  1. It can be used to test your app across all supported devices and resolutions saving significant amounts of manual testing time.
  2. It can be used to auto-capture refreshed screenshots for display in the App Store as part of a CD pipeline.
  3. It can catch inconsistencies that your unit tests won’t be able to catch.

Be careful though, because if you get this step wrong:

  1. It will significantly slow down your feedback cycle by adding painfully slow tests that take a long time to execute across all supported device simulators.
  2. The screenshots that are captured will be inconsistent across devices causing your App Store branding to look unprofessional.
  3. You will be plagued with seemingly random build failures that will cause you and your team to lose trust in the CI pipeline all together.

I suffered through all of the above in my first attempt of comprehensive UI testing. The following is a list of 7 lessons that I learned the hard way in pursuit of creating a set of solid UI tests that are a part of a CI/CD pipeline.

Lesson 1: Don’t trust the Record UI Test button.

I heard about this magic button that will write your UI tests for you. It is built right into XCode in the Debug panel when a UI Test file is open. It looks like this.

Record button

The theory is that it will launch the iOS simulator and then as you click and swipe around your app it will auto-capture all the events in a programatic way.

This is my experience:

Timeout error

This error usually happens after a couple of clicks. From here your only way to resume recording is to restart the record session. Oops.

If you do manage to get lucky and have the recorder run error-free, here is an example recording output:

let scrollViewsQuery = app.scrollViews
scrollViewsQuery.children(matching: .other).element(boundBy: 2).children(matching: .other).element.tap()

Yikes. Good luck trying to decipher what is being tapped in the above code. While it may be worth experimenting with this magic button, I suggest you are better off writing the code yourself.

Lesson 2: Add accessibility identifiers to explicitly target specific elements

To reliably target taps on UI elements in a readable manner you should take advantage of accessibility identifiers. This can be done directly in the Identity Inspector on a storyboard:

By adding accessibility identifiers, it allows simple targeting of UI elements like this:

let app = XCUIApplication()
app.buttons["switchAnnotation"].tap()

This is much easier to understand than the rather verbose and indirect output of the Record UI Test button.

PROTIP

To even further simplify your tests add an app specific extension to XCUIApplication like so:

extension XCUIApplication {
    func switchAnnotation() {
        self.buttons["switchAnnotation"].tap()
    }
}

Now the same action in the test is as simple as:

    app.switchAnnotation()

Thanks to @johnsundel for this one

Lesson 3: Take advantage of breakpoints and LLDB

One of the ongoing challenges of writing UI tests via XCUIApplication is coming up with the correct element queries to execute user actions programatically. At first I would litter the test cases with print statements of the app.debugDescription() output. This is a super helpful method that shows the complete tree of available elements currently viewable in the UI.

The problem though with using print statements in the tests is that it requires you to have the upfront insight to place the debug statements in all the possible places you may want to inspect the view hierarchy. If you forget one place then you need to stop the simulator, add the extra print, rebuild and retest. This is a rather slow feedback loop and you end up with tests that look like this:

let search = app.searchFields.element
print(app.debugDescription())
search.tap()
print(app.debugDescription())
search.typeText(query)
print(app.debugDescription())
app.cells.element(boundBy: 0).tap()
print(app.debugDescription())

Enter the LLDB debugger! Swift has a powerful debugger built-in to XCode that allows you to dynamically inspect the view hierarchy at anytime. Simply set a breakpoint and then use the LLDB po (print object) command in the debug window to see what you want on the fly.

lldb po command

PROTIP

Add the Test Failure Breakpoint in the Breakpoints Navigator to enter in the debugger on any assertion failure.

XCode Test Failure Breakpoint

Lesson 3: Create separate scheme to only run UI Tests

I found a lot of time spent creating UI tests is just waiting. Waiting for a rebuild. Waiting for the simulator. Waiting for the tests to run. If there are ways to reduce this waiting it not only directly improves build times but also psychologically reduces the likelihood for developer context switching to another task out of boredom.

One quick improvement is to create a dedicated scheme that only runs UI tests. By default, when clicking “Product->Test” in XCode it will build and run both the Unit Test suite and the UI Test suite.

To create a custom scheme go to Product -> Manage Schemes and duplicate the default Scheme. Open the duplicated scheme. Under the Test panel you’ll see the multiple test targets for your unit tests and UI tests. Remove the Unit Tests and name the scheme something like ‘UI Tests’.

Manage Schemes

Now when you want to run only your UI Tests just select the newly created scheme and hit play.

Select Unit Test Scheme

Lesson 4: Explicitly wait for everything

After performing an action in a UI Test, XCUIApplication will automatically wait for the app to ‘idle’ before continuing with the next action. This is to allow time for the animations and new views to properly load before starting the next action. The problem is that this wait time is all magic and can lead to some periodic build failures if it doesn’t get it right.

Explicitly waiting for each new view to become visible can avoid these errors. This can be done with the waitForExistance method on an XCUIElement. Here is an example:

let search = app.searchFields.element
search.tap()
XCTAssert(app.keyboards.firstMatch.waitForExistence(timeout: 10))
search.typeText("my search")

In this example after tapping search, we explicitly wait for the keyboard to show up before typing.

Lesson 6: Use the macOS Accessibility Inspector

The Accessibility Inspector is a built-in tool on macOS that allows viewing all the accessibility properties and actions for any running application. Since the XCUIApplication depends on programmatically navigating the application with many of the accessibility features this can be a really useful debugging tool.

This proved particularly useful when I was debugging a UI test attempting to create multiple annotations on a map. This test case would take the center coordinate of an existing annotation, add an offset to the coordinate, and then use that new coordinate to create a second annotation. The problem was the second annotation wasn’t showing up in the expected location. By using the Accessibility Inspector I was able to quickly see that the bounding box for the Annotation View was inherently wrong.

Accessibility Inspector

Lesson 7: Don’t expect swipes to be consistent

Unfortunately, I don’t have a great tip here. There is no way I’ve found to make the velocity of the swipe gestures (e.g. swipeLeft()) deterministic in the simulator. Most of the time this is probably fine, but in an app where the velocity of a swipe matters (e.g. in panning a map to a new location) the best bet is to avoid this action in the tests if possible.

Wrapping it up

Writing UI Tests for iOS has plenty of quirks. By taking these lessons into consideration it is possible to create UI tests that run efficiently, are easy to debug, and produce consistent results. Once the UI tests are in place it can provide a fully automated continuous deployment pipeline by automatically extracting screenshots to upload to the App Store. More on that in a future blog post.

If you’ve written UI Tests for iOS in the past, what pain points have you encountered? Do you have any other takeaways you wish you knew about before you started?