- I had a great experience during Google Summer of Code 2020.
- Puppeteer with Jest is, at least based on stats, a better framework than WebdriverIO.
- We decided to stick to WebdriverIO based on prescience.
To view all projects undertaken by Wikimedia Foundation in GSoC 2020, click here
Wikimedia Foundation is a massive organization which manages a plethora of repositories and extensions. In an environment where code complexity is ubiquitous, an extensive amount of good code testing practices are substantive.
While a considerable amount of time was spent researching WebdriverIO before selecting it as the goto framework, the ever-evolving world of software development made Željko think re-evaluation and that’s how we land in GSoC 2020.
Wikimedia uses Gerrit, a free, web-based team code collaboration tool, as their primary version control. You can find all my patches on Wikimedia Gerrit at https://gerrit.wikimedia.org/r/q/owner:+Soham
*Note: All data below is relevant as of August 2020*
We did some preliminary research to make a list of frameworks that we should consider. This shortlisting was done keeping two factors in mind:
- Popularity: as indicated by the number of monthly downloads over a period of 6 months. Clearly Cypress, Puppeteer and WebdriverIO stood out the most.
- Community: What’s a better indicator of this than the Github repository’s activity itself!
It’s as they say, “the numbers never lie”. After some initial research and a trivial process of elimination, we decided to go with Cypress and Puppeteer as the potential contenders.
*Note: We also gave Microsoft Playwright a shot but owing to some issues integrating it with the CI, we decided to do away with it, a decision I wish I had somehow not made!*
Pick a fairly large repository. In order to make this process a breeze, we computed the Source Lines of Code (SLoC) and Number of Tests for each repository
Re-implement the existing tests using Puppeteer and Cypress. This phase was broken down into several sub-stages, each having their own separate patches:
- Install dependencies
- Setup test environment and eslint
- Create a sample test
- Re-write individual tests with the new framework
*Note: We had a separate Setup video recording and screenshots for Puppeteer but more on that later.*
New tests are bound to crash, be it locally or on the CI. While referencing CI logs was a huge help during the debugging process, we soon started to feel the need for screenshots and videos of test runs. While Cypress comes with a video recorder out of the box, with Puppeteer, we were required to be a bit innovative.
While it is true that we were evaluating replacements for WebdriverIO, the probability of us not choosing to migrate to a new framework were equally high. This made us come to the conclusion that we should probably introduce video recording for existing tests as well.
*Note: Introducing video recording for WebdriverIO had a dependency on an external package. I had to spend a considerable amount of time trying to create patches upstream to upgrade the video recorder to v6 and fix some issues in order to make this work. Not to mention Johan was nice enough to review the patches.*
During the start of the coding period, we had really high hopes with one framework in particular and that being Cypress. It came out of the box with nearly everything you’d need from a browser automation framework. I highly recommend you read my post on Cypress if you haven’t already so that we have some context.
However there were a number of challenges that we faced with Cypress. We found the tests running locally to fail on the CI, primarily because of certain security flags. We suspect the errors to trace back to the internal protocols (probably dug a little too deep here 😵) that Cypress uses to communicate between the Node environment and the user agent. After spending a considerable amount of time brainstorming over the possible causes for the errors, we finally decided to pause evaluating Cypress (primarily because we had a deadline to respect).
WebdriverIO is a next-gen browser and mobile test-automation framework for Nodejs.
WebdriverIO boldly claims itself as being:
- Extensible: Adding helper functions, or more complicated sets and combinations of existing commands is simple and really useful
- Compatible: WebdriverIO can run on WebDriver Protocol for true cross browser testing as well as Chrome DevTools Protocol for Chromium based automation using Puppeteer.
- Feature Rich: The huge of variety of community plugins allows you to easily integrate and extend your setup to fulfill your requirements.
- Supports all modern frameworks: WebdriverIO allows you to automate any application written with modern web frameworks like React, Angular, Polymer or Vue.js as well as native mobile applications for Android and iOS.
- Applitools Support: WebdriverIO comes with integrated support for Applitools Eyes allowing you to write seamless visual regression tests.
Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol.
Together, Jest and Puppeteer bring a lot to the party:
- Selenium free: Puppeteer does not use Selenium.
- E2E Testing: Spinning up a custom end-to-end testing environment with Jest & Puppeteer is a breeze.
- Relevant: Puppeteer can essentially work on any front-end framework or website.
- Promise based
- Parallelism: Jest and Puppeteer have an excellent support for running parrallel tests.
- Device Emulation: Puppeteer exposes an API with excellent device-emulation features.
Following data collection process was followed:
- The data above was collected by running Puppeteer and WebdriverIO tests for mediawiki/core and extensions/EntitySchema
- We ran a total of 30 test runs per test suite spanning over 2 separate machines. Graph shows data for 15 such tests.
- For measuring reliability, % tests failures for each run were computed. Interestingly, no such failures were encountered.
- Additionally, a short PoC to test the behaviour of the frameworks when running tests in parallel was also evaluated. The data (not shown) was collected for an additional of 10 test runs with 2 workers.
*Note: You can click on individual rows to expand*
A comparison without evaluating the various capabilities of a framework would not only be incomplete but also far from being accurate. WebdriverIO surprisingly has a provision of supporting both sync and async modes. Inspired by the clean
async-await syntax that Puppeteer follows, we decided to conduct a short Proof of Concept using WebdriverIO Async and I must say, I was somewhat impressed on the myriad of options that WebdriverIO exposes.
Curious more? You can find all patches related to this here.
Should we migrate and if so with what effort?
Both Puppeteer and WebdriverIO are excellent browser automation frameworks but based on the above results, clearly Puppeteer makes an excellent choice over WebdriverIO. However before we make any precipitous decision, let us see what the future has in store for WebdriverIO.
While Željko was away busy scoring GoPros on his vacation, I got some time to play around with WebdriverIO. I mean Cypress wouldn’t work and I was pretty much done with Puppeteer so I had plenty of time to kill. Since WebdriverIO makes this bold claim of being able to automate any modern web framework, I decided to put this to test with none other than this very website (which has been growing increasing complex as time progresses).
v6.3, several new network primitives were introduced. Particularly, the ability to easily mock web resources in tests and define custom responses seem really promising. In fact, this update makes WebdriverIO at par with Puppeteer as well as Cypress.
Stubbing a RESTful API response has now become as simple as:
I highly recommend skimming through the Mocks and Spies section of the docs.
This is where things get intense!
Throttling refers to the modification of the network throughput of the browser. This allows for really complex test scenarios such as those under different network conditions like 3G, 4G or even in an offline mode.
A little known fact about WebdriverIO (which is why it lost points for documentation) is that it ships Puppeteer as a second automation driver. This basically means a tonne of extra features including the Chrome DevTools. While this is still in beta, WebdriverIO is headed towards a glittering future.
In conclusion, while the stats show Puppeteer as a better framework, WebdriverIO is evolving at an unprecedented rate.
It is important to remember that Puppeteer was not meant to be an automated testing framework nor was Jest intended to transform Puppeteer into one. WebdriverIO has and shall continue to remain an automated testing framework and conclusively constantly keep improving itself to keep up with other modern tools.
As far as Wikimedia is concerned, WebdriverIO ain’t going anywhere.
Man what can I say! I have been working closely with the QTE team (or the
Cute team as Željko jokingly likes to call it) for the past 5 months and it has been an excellent experience. While I have come a long way as far as general coding goes, my most cherished learnings have been philosophical than practical.
Words cannot describe how greatful I am to my mentors:
- Željko Filipin, for teaching me about the importance of good code review and coding practices.
- Gabriel Pita, for keeping the atmosphere convivial with his jocular disposition and insane sense of humor.
- John Bolorinos, for constantly livening me up and sharing his experience as a software developer at Apple, Google and Wikimedia, which is something I would never forget.
- I plan to introduce video recording into tests across all the repositories in Wikimedia and do this as soon as I am done with GSoC.
- A plan to contribute to Cypress upstream seems inevitable. While diving deep into Cypress’s source code, I think I might have found a few gray areas which if refactored might just resolve our problems.
- Željko, based on his recent experience with vacation, recommended that I take a two week hiatus from any form of coding and I’d be doing just that…hopefully soon.