Flakiness.io
This service provides test analytics for GitHub and GitLab, connecting test results and artifacts with commit history and testing environment so test reports can show whether a pull request failure is a real regression or a known flake from the target branch. Its core is a specialized analytics engine for storing and querying large amounts of test results, with 2.5 billion test results processed to date. It offers a 1GB free plan (roughly 10M+ test results), with limits increasable to 100GB for large open source projects.
What it addresses
It is hard to tell whether a pull request test failure is a real regression or a known flake from the target branch.
Shared over time
Each entry reflects what was shared on that date. Details may have changed since then.
Flakiness.io
This service provides test analytics for GitHub and GitLab, connecting test results and artifacts with commit history and testing environment so test reports can show whether a pull request failure is a real regression or a known flake from the target branch. Its core is a specialized analytics engine for storing and querying large amounts of test results, with 2.5 billion test results processed to date. It offers a 1GB free plan (roughly 10M+ test results), with limits increasable to 100GB for large open source projects.
Problem: It is hard to tell whether a pull request test failure is a real regression or a known flake from the target branch.
Audience: Developers and QA engineers using GitHub or GitLab CI, including large open source projects (e.g., nuxt/nuxt, wordpress/gutenberg).
Posted by @aslushnikov · Original comment ↗
“https://flakiness.io: test analytics for GitHub and GitLab. The idea is to connect test results and artifacts with commit history and testing environment. This way test reports can show whether a failure in a pull request is a real regression or a known flake from a target branch. The core of the service is a specialized analytics engine that makes it possible to store and query large amounts of test results [1]. Thanks to this engine, we've processed 2.5 billion test results to date.”