rans>Skip to main content

Soham Parekh

Software developer. Currently a core maintainer and co-author @Devstation. Likes coding, tech, reading/writing, founding things.

7 min read · August 24th 2020

Wikimedia Foundation - Google Summer of Code 2020

TL;DR

  • I had a great experience during Google Summer of Code 2020.
  • Puppeteer with Jest is, at least based on stats, a better framework than WebdriverIO.
  • We decided to stick to WebdriverIO based on prescience.

To view all projects undertaken by Wikimedia Foundation in GSoC 2020, click here

Project Background

Wikimedia Foundation is a massive organization which manages a plethora of repositories and extensions. In an environment where code complexity is ubiquitous, an extensive amount of good code testing practices are substantive.

Wikimedia has had a remarkable history of migrations when it comes to their browser automation frameworks (and rightfully so…you wouldn’t and shouldn’t imagine the day when Wikipedia goes down). After playing around with Selenium based frameworks in PHP and Ruby, they finally decided to stick to Javascript with the good ol’ WebdriverIO.

While a considerable amount of time was spent researching WebdriverIO before selecting it as the goto framework, the ever-evolving world of software development made Željko think re-evaluation and that’s how we land in GSoC 2020.

Repository

Wikimedia uses Gerrit, a free, web-based team code collaboration tool, as their primary version control. You can find all my patches on Wikimedia Gerrit at https://gerrit.wikimedia.org/r/q/owner:+Soham

Preliminary Research

*Note: All data below is relevant as of August 2020*

We did some preliminary research to make a list of frameworks that we should consider. This shortlisting was done keeping two factors in mind:

Comparison

  1. Popularity: as indicated by the number of monthly downloads over a period of 6 months. Clearly Cypress, Puppeteer and WebdriverIO stood out the most.

Stats

  1. Community: What’s a better indicator of this than the Github repository’s activity itself!

It’s as they say, “the numbers never lie”. After some initial research and a trivial process of elimination, we decided to go with Cypress and Puppeteer as the potential contenders.

*Note: We also gave Microsoft Playwright a shot but owing to some issues integrating it with the CI, we decided to do away with it, a decision I wish I had somehow not made!*

Approach

StartRepoTestCrashEnd

Repo

Pick a fairly large repository. In order to make this process a breeze, we computed the Source Lines of Code (SLoC) and Number of Tests for each repository

Test

Re-implement the existing tests using Puppeteer and Cypress. This phase was broken down into several sub-stages, each having their own separate patches:

  1. Install dependencies
  2. Setup test environment and eslint
  3. Create a sample test
  4. Re-write individual tests with the new framework

*Note: We had a separate Setup video recording and screenshots for Puppeteer but more on that later.*

Crash

New tests are bound to crash, be it locally or on the CI. While referencing CI logs was a huge help during the debugging process, we soon started to feel the need for screenshots and videos of test runs. While Cypress comes with a video recorder out of the box, with Puppeteer, we were required to be a bit innovative.

While it is true that we were evaluating replacements for WebdriverIO, the probability of us not choosing to migrate to a new framework were equally high. This made us come to the conclusion that we should probably introduce video recording for existing tests as well.

*Note: Introducing video recording for WebdriverIO had a dependency on an external package. I had to spend a considerable amount of time trying to create patches upstream to upgrade the video recorder to v6 and fix some issues in order to make this work. Not to mention Johan was nice enough to review the patches.*

Problems with Cypress

During the start of the coding period, we had really high hopes with one framework in particular and that being Cypress. It came out of the box with nearly everything you’d need from a browser automation framework. I highly recommend you read my post on Cypress if you haven’t already so that we have some context.

However there were a number of challenges that we faced with Cypress. We found the tests running locally to fail on the CI, primarily because of certain security flags. We suspect the errors to trace back to the internal protocols (probably dug a little too deep here 😵) that Cypress uses to communicate between the Node environment and the user agent. After spending a considerable amount of time brainstorming over the possible causes for the errors, we finally decided to pause evaluating Cypress (primarily because we had a deadline to respect).

Some Facts About WebdriverIO

WebdriverIO is a next-gen browser and mobile test-automation framework for Nodejs.

WebdriverIO boldly claims itself as being:

  1. Extensible: Adding helper functions, or more complicated sets and combinations of existing commands is simple and really useful
  2. Compatible: WebdriverIO can run on WebDriver Protocol for true cross browser testing as well as Chrome DevTools Protocol for Chromium based automation using Puppeteer.
  3. Feature Rich: The huge of variety of community plugins allows you to easily integrate and extend your setup to fulfill your requirements.
  4. Supports all modern frameworks: WebdriverIO allows you to automate any application written with modern web frameworks like React, Angular, Polymer or Vue.js as well as native mobile applications for Android and iOS.
  5. Applitools Support: WebdriverIO comes with integrated support for Applitools Eyes allowing you to write seamless visual regression tests.

Some Facts About Jest + Puppeteer

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol.

Most things that you can do manually in the browser can be done using Puppeteer! That includes creating an up-to-date automation testing environment. This is where Jest, a Javascript Testing Framework written by Facebook, comes into picture.

Together, Jest and Puppeteer bring a lot to the party:

  1. Selenium free: Puppeteer does not use Selenium.
  2. E2E Testing: Spinning up a custom end-to-end testing environment with Jest & Puppeteer is a breeze.
  3. Relevant: Puppeteer can essentially work on any front-end framework or website.
  4. Promise based
  5. Fast
  6. Parallelism: Jest and Puppeteer have an excellent support for running parrallel tests.
  7. Device Emulation: Puppeteer exposes an API with excellent device-emulation features.

Puppeteer vs WebdriverIO

Following data collection process was followed:

  • The data above was collected by running Puppeteer and WebdriverIO tests for mediawiki/core and extensions/EntitySchema
  • We ran a total of 30 test runs per test suite spanning over 2 separate machines. Graph shows data for 15 such tests.
  • For measuring reliability, % tests failures for each run were computed. Interestingly, no such failures were encountered.
  • Additionally, a short PoC to test the behaviour of the frameworks when running tests in parallel was also evaluated. The data (not shown) was collected for an additional of 10 test runs with 2 workers.

*Note: You can click on individual rows to expand*

Legend

Icon

Feature Availability

Excellent (fully available)
Good (partially available, e.g. plugins)
Fair (needs customization or limited)
Poor (not possible)

Icon

Feature Availability

Excellent (fully available)
Good (partially available, e.g. plugins)
Fair (needs customization or limited)
Poor (not possible)

Performance

CategoryPuppeteerWebdriverIOGeneral
Time and effort required for bootstrapping a automation test environment
Cross Browser Support
Device emulation capabilities exposed by the frameworks
Ability to run tests that incorporate tab switching and iframes
Dispatching events by the user agent which allows for user agent behaviors like hoversTest Optimization
Support for Page Object Model syntax
Average time per test for a fairly large test suite
How often tests fail after being authored, other than when detecting a real application bug
Support for running test suites in parallel

Developer Experience

CategoryPuppeteerWebdriverIOMaintainability and debuggability
Cleanliness and ease with which the syntax can be grasped.
Support for adding custom commands and methods over existing API
Use environment variables for easy environment configurations

Documentation

CategoryPuppeteerWebdriverIOTutorials and guides
Read an introductory tutorial with a step-by-step process to help new users get started quickly
Read a guide with a step-by-step process to help you stub data from a mock data sourceDevelopment guides
Read a guide with a set of instructions to perform testing and quality assurance such as linting within a development workflow
Read a guide with a set of instructions to perform debugging and troubleshooting
Read a guide with a set of instructions to improve performance benchmarks within the framework

Ecosystem

CategoryPuppeteerWebdriverIOIntegrations
Leverage a framework that has integrations off the shelf with other systems

WebdriverIO Async

A comparison without evaluating the various capabilities of a framework would not only be incomplete but also far from being accurate. WebdriverIO surprisingly has a provision of supporting both sync and async modes. Inspired by the clean async-await syntax that Puppeteer follows, we decided to conduct a short Proof of Concept using WebdriverIO Async and I must say, I was somewhat impressed on the myriad of options that WebdriverIO exposes.

Curious more? You can find all patches related to this here.

The Big Question

Should we migrate and if so with what effort?

Both Puppeteer and WebdriverIO are excellent browser automation frameworks but based on the above results, clearly Puppeteer makes an excellent choice over WebdriverIO. However before we make any precipitous decision, let us see what the future has in store for WebdriverIO.

While Željko was away busy scoring GoPros on his vacation, I got some time to play around with WebdriverIO. I mean Cypress wouldn’t work and I was pretty much done with Puppeteer so I had plenty of time to kill. Since WebdriverIO makes this bold claim of being able to automate any modern web framework, I decided to put this to test with none other than this very website (which has been growing increasing complex as time progresses).

With WebdriverIO v6.3, several new network primitives were introduced. Particularly, the ability to easily mock web resources in tests and define custom responses seem really promising. In fact, this update makes WebdriverIO at par with Puppeteer as well as Cypress.

Network stubbing

Stubbing a RESTful API response has now become as simple as:

I highly recommend skimming through the Mocks and Spies section of the docs.

Throttling

This is where things get intense!

Throttling refers to the modification of the network throughput of the browser. This allows for really complex test scenarios such as those under different network conditions like 3G, 4G or even in an offline mode.

Are We Switching to Puppeteer?

A little known fact about WebdriverIO (which is why it lost points for documentation) is that it ships Puppeteer as a second automation driver. This basically means a tonne of extra features including the Chrome DevTools. While this is still in beta, WebdriverIO is headed towards a glittering future.

In conclusion, while the stats show Puppeteer as a better framework, WebdriverIO is evolving at an unprecedented rate.

It is important to remember that Puppeteer was not meant to be an automated testing framework nor was Jest intended to transform Puppeteer into one. WebdriverIO has and shall continue to remain an automated testing framework and conclusively constantly keep improving itself to keep up with other modern tools.

As far as Wikimedia is concerned, WebdriverIO ain’t going anywhere.

Acknowledgements

Man what can I say! I have been working closely with the QTE team (or the Cute team as Željko jokingly likes to call it) for the past 5 months and it has been an excellent experience. While I have come a long way as far as general coding goes, my most cherished learnings have been philosophical than practical.

Words cannot describe how greatful I am to my mentors:

  • Željko Filipin, for teaching me about the importance of good code review and coding practices.
  • Gabriel Pita, for keeping the atmosphere convivial with his jocular disposition and insane sense of humor.
  • John Bolorinos, for constantly livening me up and sharing his experience as a software developer at Apple, Google and Wikimedia, which is something I would never forget.

What’s next for me?

  1. I plan to introduce video recording into tests across all the repositories in Wikimedia and do this as soon as I am done with GSoC.
  2. While we did evaluate WebdriverIO async, we are yet to deliberate over it and hopefully a meet with Timo would soon be in place to conclude this PoC. I remember Željko telling me about Timo being one of the best Javascript developers he’s ever known which personally has me really excited.
  3. A plan to contribute to Cypress upstream seems inevitable. While diving deep into Cypress’s source code, I think I might have found a few gray areas which if refactored might just resolve our problems.
  4. Željko, based on his recent experience with vacation, recommended that I take a two week hiatus from any form of coding and I’d be doing just that…hopefully soon.

Comments

Tagged with gsoc, testingView all Tags

Enjoyed this post? Receive the next one in your inbox!

Hi
Blog
Tutorials
Notes
About