Problem Context- Brittle selectors: We spend hours fixing fragile XPaths and CSS selectors just to verify if a button is visible or a chart is correct
- Release changes: Automation code that is stable "now" suddenly becomes flaky after the release. The reason is the ever-changing platform. And automation is not always able to cope with it
- Code volume: Code analysis in our company showed us that assertion logic is typically five times (5x) larger in code size than action logic, consuming up to three months of dedicated maintenance effort every year
SolutionIn this session, I introduce ViTO (Visual Testing Oracle), a production-deployed framework that leverages multimodal Generative AI (GenAI). ViTO "sees" the application exactly like a human does. The best part is that, in the end, it's just another block of code that can be embedded inside any framework.
Summary of what's in the talkI will share:
- The logic and algorithm of how we used GenAI to decouple verification from the underlying code resulted in a 50% reduction in our assertion codebase.
- How we replaced thousands of lines of brittle verification logic with resilient, prompt-driven visual oracles that can handle complex data visualisations and unseen UI faults with zero extra effort. If you are tired of your tests breaking because a div changed, it's time to shift from structural selectors to a visual AI oracle.
- The lessons learned from our initiative, and above all, where NOT to use GenAI
- Access to the boilerplate code that you can implement within your repo
- If time permits, a demo of the framework in action. If short on time, the link to the boilerplate is provided in the slides :)
Who is this for?- QA Architects, Senior SDETs, Automation Engineers, Manual testers looking to transition to GenAI-based testing;
- Managers/architects looking for a language-agnostic framework to build GenAI-based assertions
- Anyone who wishes to know where to and where NOT to use GenAI in testing
- QA professionals looking for a starting point (boilerplate) code to embed GenAI in their automation
Key takeaways:- In-code GenAI: How to implement GenAI directly "in-code" using any programming language
- Prompt Engineering for Testers: How to write resilient "Assertion Prompts" that replace complex conditional code and handle visual regression automatically.
- Real-World ROI: Evidence-based results from a production environment, showing a 50% reduction in code maintenance and expanded coverage for rich UI components.
- Deterministic AI: Practical strategies to control GenAI hallucinations using "concentrated screenshots"
- A sneak peek into what's coming in the future in GenAI for test automation