Introducing E2E Tests to React Native Core

React Native is released by a rotating team of four developers from partner companies and volunteers from the community. Together they form a Release Crew. As a part of release chores, the Crew needs to test each and every version before they release it to npm. 

Every commit is automatically unit-tested (separately for JS, Android, iOS layers). On top of that, it passes through a variety of internal tests of various fidelity at Meta.

Until very recently, this testing included a primarily manual process of building, opening, and clicking through the RNTester app.

The need for end-to-end automation

To some extent, manual testing has served its purpose for years, relying on human capacity and limited automation at the unit level. That’s not the best position for the React Native framework to be in, though, especially from an open-source perspective. Manual testing depends on human capacity and error; automation is only at the unit level. When internal Meta tests fail, the failure information is scarce, and the feedback loop gets longer as Meta engineers are necessary to resolve it.

As the repository started to move to a proper monorepo setup and larger independence from Meta’s internal infrastructure, it became apparent that it needed a good suite of automated end-to-end tests.

The React Native Core team meets a group of their partners every month. As Callstack is one of the partner companies, we attend those meetings to discuss current issues we deal with. In one of such meetings the core team, together with Lorenzo Sciandra (Microsoft), voiced a concern that to improve the confidence and reliability of the framework, we need a suite of reliable automated end-to-end tests. And it’s not that there were none. 

React Native tried Detox E2E tests in the past with various successes, but due to issues over the years, they were turned off. As we internally grew our QA engineering team at the time, Michał Pierzchała, Callstack’s representative at those meetings, offered a helping hand in researching the best solution using the experience of engineers specialized in this field. Detox engineers from Wix offered help as well. All this gave birth to a dedicated E2E working group whose aim was to analyze and find the right solution for React Native Core.

Creating the Request For Comments (RFC)

Lorenzo Sciandra, Senior Software Engineer at Microsoft and member of the Release Crew, led the creation of a document summarizing the E2E working group's investigation. This document was shared in the discussion and proposal repository as a Request For Comments (RFC). An RFC is a kind of proposal that presents new initiatives in the React Native Community. All proposals are in their own repository: Discussions and Proposals.

A good quality RFC needs to check a few boxes: motivation, detailed design, alternatives, drawbacks, and so on. All should be documented in the template - here’s an RFC related to introducing E2E.

When it comes to adding significant changes to React Native (or React too), RFC is the right tool to kick off the conversation and set some basic shared concepts. It serves as a discussion forum and initial technical documentation for anyone who wants to learn more about the topic or how it evolved through comments from interested parties.

Cooperation between Microsoft, Meta, and Callstack

We began working on implementation in November 2022. Back then, we started with a comparison between tools for writing E2E tests. In a smaller E2E working group, we shortlisted Detox and Appium as technologies we want to evaluate. The idea was to create an E2E test for the RNTester. 

If you’re not familiar with the RNTester as such: it’s an internal showcase app inside the react-native repository that presents features and components that come with the package. You can play with scroll views, lists, and all sorts of available components and edge-cases around them. We highly recommend running it locally and giving it a go if you like to experiment with React Native.

The requirements were very strict because of the RNTester custom setup, Meta’s engineers used to a certain quality of writing tests, and our future plans. We wanted to have a tool that:

• didn’t have to touch the RNTester app, so it’ll be pure black box testing,
• had to be run and orchestrated by Jest to very closely match the internal Meta monorepo tooling used by the engineers,
• is easy to extend for out-of-tree platforms (e.g. MacOS, Windows),
• supports running tests on a device farm.

Unfortunately, we couldn't choose tools like Maestro or Detox, despite the eagerness of their teams to assist us up to this point. But it doesn’t mean that these are not the right tools for your project; simply for React Native’s own very particular use case they weren’t sufficient.

After thorough analysis, prior-art experiments by Lorenzo, and talking with tools maintainers, we chose Appium + WebDriverIO + JS as a tech stack. At that time, it was (and still is!) the best option.

On the way to introducing E2E tests in React Native Core

After choosing the tech stack we started working on—soon to be—a gigantic PR introducing the setup for iOS and Android platforms. It took us a few months and hundreds of interactions to get the PR merged. Callstack’s QA engineers weren’t involved in this effort full time.

We’ve worked through our R&D program for a few hours every week. But that turned out not to be the main thing slowing us down, as everyone involved in the process of adding and reviewing this setup had other work on their plate too. In other words, assigning a full-time engineer wouldn’t speed this effort up proportionally. Many stakeholders were included in the review process both on the code and functional side. And we all accepted that trade-off.

We created a separate ^{rn-tester-e2e} package in React Native’s monorepo where config files and tests live. Nearly a hundred of comments, 50+ trials of setting up CI. We handled the integration with CircleCI, and it proved to be one of our most challenging tasks in a while. At one juncture, we needed to reach out to a CircleCI employee for assistance, and fortunately, they were very supportive and helped us resolve the issue. 

Finally, we’ve got an operating setup that runs tests on every commit and every pull request. At the time of publication of this article, it’s only a few weeks after E2E tests landed, and there were already a few situations where these tests found regressions or crashes—and potentially saved the core team from shipping them. It’s really great to see this so early on.

E2E tests - next steps

As of the time of writing this post (Sep ‘23) we have a working setup integrated with CircleCI. Our QAs have dedicated time to working on writing actual tests in RNTester, we’re also planning to create an umbrella issue that will contain components that are not tested yet, anyone will be able to grab a component to test and contribute to React Native.

Together with the React Native Core Team, we’re thinking of introducing a device farm so that E2E tests will be run on physical devices.

We’re also doing research related to screenshot testing as another test technique that we can use in React Native core to ensure that the features that we’re shipping are working correctly. And we’re in a tight feedback loop with Meta engineers who report to us some missing quality-of-life functionality that they have internally and could potentially be reimplemented in the open-source space.

Conclusion

At first sight, we thought that this would require less time and work to get the setup working, but after a few months of interrupted work, we’re happy with the result that we have. In a collaborative effort, we’ve laid a solid foundation for the stability of the framework, and the Release Crew will hopefully be more confident when it comes to releases. This collaboration between Callstack, Microsoft, and Meta engineers certainly reaffirmed our confidence in the positive force of partnerships that are formed around the React Native project. Async communication and cooperation between partners is important to introduce meaningful changes to the core. Huge thanks to all who contributed to this effort:

• Lorenzo Sciandra
• Nicola Corti
• Riccardo Cipolleschi
• Mateusz Ulańczyk
• Adrian Orszulik
• Mateusz Michalec
• Elżbieta Berger
• Michał Pierzchała
• Szymon Rybczak