Nordic Testing Days 2026

In this talk, I show how we run a lot of full end-to-end Cypress web application tests every day. In addition to running the full data set, we do separate feature test runs based on test tags. We also allow everyone from all teams to trigger the tests right from GitHub Actions UI. This lets every group quickly test their feature before merging into the main branch.

For pull requests, we employ source code analysis based on data test IDs to run the affected tests first for quicker feedback. The software automation team uses the flake test information to chase the sources of the underlying errors to minimize noise and make every passing test run give us confidence in the released code, and every failing test run useful to quickly diagnose the real underlying issue.

The presentation covers test writing, test organization, selecting tests to run based on the source code changes, running tests in different resolutions. I also look into making the tests faster by employing data creation and caching, as well as using API calls to bypass the user interface in some places. Finally, making the tests robust and flake-free and triaging the failed runs is an ongoing activity for the automation team.

Key takeaways:

• How to run 1000 of end-to-end tests quickly
• Which tests to run on a pull request
• How AI is helping us pick tests to run

Problem Context

• Brittle selectors: We spend hours fixing fragile XPaths and CSS selectors just to verify if a button is visible or a chart is correct
• Release changes: Automation code that is stable "now" suddenly becomes flaky after the release. The reason is the ever-changing platform. And automation is not always able to cope with it
• Code volume: Code analysis in our company showed us that assertion logic is typically five times (5x) larger in code size than action logic, consuming up to three months of dedicated maintenance effort every year

Solution
In this session, I introduce ViTO (Visual Testing Oracle), a production-deployed framework that leverages multimodal Generative AI (GenAI). ViTO "sees" the application exactly like a human does. The best part is that, in the end, it's just another block of code that can be embedded inside any framework.

Summary of what's in the talk
I will share:

1. The logic and algorithm of how we used GenAI to decouple verification from the underlying code resulted in a 50% reduction in our assertion codebase.
2. How we replaced thousands of lines of brittle verification logic with resilient, prompt-driven visual oracles that can handle complex data visualisations and unseen UI faults with zero extra effort. If you are tired of your tests breaking because a div changed, it's time to shift from structural selectors to a visual AI oracle.
3. The lessons learned from our initiative, and above all, where NOT to use GenAI
4. Access to the boilerplate code that you can implement within your repo
5. If time permits, a demo of the framework in action. If short on time, the link to the boilerplate is provided in the slides :)

Who is this for?

• QA Architects, Senior SDETs, Automation Engineers, Manual testers looking to transition to GenAI-based testing;
• Managers/architects looking for a language-agnostic framework to build GenAI-based assertions
• Anyone who wishes to know where to and where NOT to use GenAI in testing
• QA professionals looking for a starting point (boilerplate) code to embed GenAI in their automation

Key takeaways:

• In-code GenAI: How to implement GenAI directly "in-code" using any programming language
• Prompt Engineering for Testers: How to write resilient "Assertion Prompts" that replace complex conditional code and handle visual regression automatically.
• Real-World ROI: Evidence-based results from a production environment, showing a 50% reduction in code maintenance and expanded coverage for rich UI components.
• Deterministic AI: Practical strategies to control GenAI hallucinations using "concentrated screenshots"
• A sneak peek into what's coming in the future in GenAI for test automation

Teams often rely on test cases executed, bugs reported, and pass rates to measure success. These numbers might look impressive, but do they truly reflect software quality? Vanity metrics can mislead teams, encourage the wrong behaviours, and create a false sense of progress.

This talk introduces a 7-step framework to move beyond superficial KPIs and focus on metrics that drive real value. Inspired by analytical approaches in competitive sports, this model helps teams make better decisions, align testing efforts with business goals, and ensure that data supports meaningful improvements.

Key takeaways:

• The risks of vanity metrics and how they can mislead decision-making.
• How to design KPIs that focus on value, not just activity.
• A practical framework to ensure testing metrics drive meaningful change
  

The security impact of missing input validation is usually underestimated.

The presentation explains and gives examples of how missing logical limits may lead to denial of service attacks on the application that seems quite secure - no injection or execution vulnerabilities needed.

As the presenter is a co-lead of the OWASP ASVS project, related security requirements are also pointed out.

Not a single word about AI.

Visual testing is supposed to protect QA teams from the familiar “it looks wrong” bug, yet traditional pixel-diff approaches only show that something changed, not whether that change actually matters. As modern interfaces grow more dynamic and design systems become more complex, teams need smarter ways to detect meaningful visual regressions.

This talk presents a practical approach to automated visual bug detection using multimodal LLMs. Drawing on a real-world implementation, it shows how AI models from providers such as OpenAI, Anthropic, and Google can be orchestrated to analyze screenshots and identify issues that pixel-based tools often cannot interpret on their own. These include layout breaks, missing elements, accessibility concerns, color contrast problems, and platform-specific guideline violations.

The session explores how AI-driven visual analysis can move beyond pixel-perfect comparison toward semantic understanding, helping teams distinguish intentional UI changes from genuine defects. It also addresses one of the biggest challenges in visual testing at scale: false positives, demonstrating how agent-based review systems can reduce noise while still surfacing critical issues.

Attendees will leave with practical ideas for using multimodal AI to strengthen visual testing workflows and make automated UI validation more accurate, scalable, and useful.

Key Takeaways:

• How to evolve from “pixel diffs” to impact-based automated visual feedback
• Patterns that turn image feedback into structured results (what changed, where, severity, why it matters)
• Tips for integrating automated LLM-powered visual feedback into existing automated UI test frameworks