Hey Meshery Contributors 👋

As we continue evolving Meshery’s end-to-end (E2E) UI test suite, we wanted to proactively communicate some updates and best practices around these tests.

Our ongoing efforts are focused on improving visibility, performance, and coverage, while also reducing noise from flaky or unstable test cases. This work is essential to ensure our test suite continues to provide confidence without becoming a blocker to fast iteration.

Why You Might See Some Test Failures

As we refine and improve the test suite, you may start noticing failures from time to time. Don’t panic—this might actually mean the test is working as intended! 🎉 Here’s what we suggest when this happens:

✅ Step-by-Step: What to Do if a Test Fails

1. Verify Your Changes

First, double-check that your code changes didn’t break something. If a test is consistently failing after your update, it might be due to a legitimate bug. 🐛

2. Retry the Test Suite

If the failure seems random or inconsistent, go ahead and retry the test. Sometimes flaky tests just need another shot.

3. Still Failing? Let’s Triage Based on Test Type

  • Newly Added Tests
    If the test was just added and it’s failing consistently, we can mark it as unstable to prevent it from blocking merges. Please call this out in the Slack channel or during a community meeting, and we’ll open an issue to track and stabilize it.

  • Existing/Older Tests
    If a legacy test fails, but the feature work is critical, we can cautiously proceed with the merge after creating an issue to investigate the test.
    In the meantime, we can temporarily mark the test as unstable to reduce noise, but please remember to flag it in Slack or in a meeting so we stay on top of it.

🧰 Resources for Investigating Failures

Here are some handy tools and references to help you diagnose and debug test failures:

👏 Thanks for Your Patience and Help

This is a community effort, and we appreciate your support as we make our UI test infrastructure more robust and developer-friendly. If you have feedback or ideas to improve the process, drop a note in the #meshery-ci channel or bring it up during our community calls!

Together, we’re building better tooling—for everyone.

🧪💚
The Meshery Maintainers