Nordic Testing Days 2026

As organizations rush to adopt Large Language Models (LLMs), many discover that building reliable, trustworthy applications is far from straightforward. Unlike traditional software, LLM outputs are non-deterministic, context-dependent, and vulnerable to issues like bias, hallucinations, and prompt injection. Ensuring quality requires more than testing—it demands a holistic approach that blends architecture, safety, observability, and continuous feedback. This talk explores practical strategies for embedding quality into LLM-powered systems from the ground up. We’ll cover methods for prompt design, evaluation frameworks, guardrails, and hybrid architectures that improve accuracy and safety. Attendees will leave with a clearer understanding of how to balance innovation with reliability and how to design AI applications that are not only powerful but also consistent, secure, and user-focused.

Key takeaways:

1. Testing LLMs requires new methods, not just old QA practices.Combine automation + human oversight for best results.
2. Build feedback and safety into the system from the start.
3. Quality is a continuous journey, not a release milestone.

The general QA engineer role is on the rise as software projects become larger and more complex. General QA engineers have a broad set of skills to test complex systems end-to-end, going beyond the traditional QA role focused on component testing. They bring new perspectives by not being tied to one specific domain. This helps them test software more holistically across traditional boundaries.

Key Takeaways:

•  Adapt. Modern QA engineers must continuously expand their skills beyond traditional manual testing to stay relevant in a rapidly evolving industry.
• Collaborate. Quality is a shared responsibility — successful QA professionals work closely with developers, product teams, and business stakeholders beyond classic QA boundaries.
• Lead. QA engineers should drive quality culture, influence change, and become strategic contributors within their organizations, not just testers
  
  

Pull requests, aka merge requests, are a goldmine of information that many testers are missing out on. Although pull requests are often seen as a tool mainly for developers, they offer testers an equal opportunity to provide feedback. When joining the review process testers can contribute to the software quality before changes are added to the codebase. They can use their unique perspective to uncover insights that go beyond the code. They will learn more about the bigger picture of the application. And ultimately, testers can apply these observations to their testing and help the team prevent bugs.

In this talk, Andrea will introduce how testers can get involved in the pull request review process. She will demonstrate how testers can shift left and proactively engage with the source code changes. The step-by-step approach covers everything from checking policies and standards. It also includes looking at build pipelines, code changes, and unit tests. Starting with the basics, the session gives clear examples at every stage, making it easy to follow along.

Andrea will highlight practical examples and insights drawn from real-life experiences, providing actionable tips on how to get the most out of pull request reviews. The session illustrates how testers can analyze pull requests for quality, learn more about the source code, and make a real impact through their feedback.

At the end of the session, attendees will have a hands-on guide to reviewing pull requests. They will be ready to take a closer look at their team’s work and discover new insights, just as curious koalas explore branches to find tasty leaves of eucalyptus. It’s time for testers and their teams to realise the untapped potential of making pull request reviews a shared responsibility.

Key takeaways:

• Understand the elements of a pull request and know about the basic principles of the Git workflow
• Learn how to shift-left and contribute to the pull request review process
• By the end of the session, participants will be able to start reviewing pull requests

Explore how critical thinking, curiosity, empathy, and human judgment are becoming even more important as AI continues to reshape software testing, education, and the workplace. Through engaging real-world examples and reflections on history, ethics, and technology, the session highlights why humans remain essential in guiding, questioning, and improving AI-driven systems. Attendees will gain practical insights into human-in-the-loop collaboration, ethical AI usage, and the future skills needed to thrive in an increasingly AI-powered world.

Key Takeawyas

• The presentation argues that as AI and LLMs become better at automation, the real differentiator becomes human judgment, curiosity, empathy, and critical thinking. The future is not “human vs AI,” but humans guiding AI responsibly.
• Critical thinking is becoming a survival skill in the AI era 
  A major theme is that people must learn to question information, recognize bias, and resist blindly trusting systems, algorithms, or AI-generated outputs. Especially in testing and quality assurance, asking the right questions matters more than ever.The best results come from “Human + AI” collaboration
• Rather than fearing AI, the presentation encourages using it as an amplifier for creativity, quality, and productivity, while keeping humans accountable for ethics, context, trust, and meaning. It positions testers and quality professionals as essential “humans in the loop.”

Intro:Artificial Intelligence challenges almost every assumption the testing discipline is built on. Traditional testing depends on fixed inputs and predictable logic, but AI systems are adaptive, probabilistic, and context-dependent. That means our classical test cases are no longer stable reference points.

In this 20-minute talk, Nicole van Gijn explores what testing looks like when your system learns, reasons, and occasionally hallucinates. She introduces the AI Quality Grid, a structured framework co-developed with John Kronenberg, that helps define quality attributes, risks, and validation strategies for AI applications. The session bridges theory and practice through concrete examples from a real AI test project, showing how LLM-evals and risk-based thinking can be combined to test prompt robustness, output consistency, and bias control within modern CI/CD pipelines.

Attendees will walk away with a lightweight but actionable structure for AI quality assessment and a new mindset: understanding quality not as a checklist, but as an intelligent, adaptive discipline.

Key Takeaways:

• AI systems are rapidly entering production pipelines, yet testing methods lag behind.
• Testers and QA leads urgently need practical models to evaluate non-deterministic outputs.
• The AI Quality Grid offers a bridge between AI model evaluation (LLM-evals) and classical test strategy, providing testers with new tools and thinking patterns to stay relevant in the AI era.

Deploying to production shouldn't require a meeting, three approvals, and a prayer. Yet most teams treat every release like launching a rocket - mission control on standby, everyone watching the countdown, but no one wanting to press the red button.

At Sokos Hotels, our web booking system handles thousands of reservations daily, processing millions monthly across 50 hotels all over Finland and Estonia. One critical bug means lost revenue; one outage means thousands of unhappy guests. We were trapped in weekly releases, manual verification, and the kind of Thursday tension that put everyone in the mission control room. We asked ourselves if Continuous Deployment is just a myth.

In less than a year, we broke the cycle. We went from weekly manual releases to deploying seven times per day with higher confidence than ever. The results? 4.5/5.0 customer effort score, 40% higher conversion rates and 4.3+/5.0 overall team happiness. The secret? A testing strategy that made deployment boring for the last 2 years.

Join my talk to find out how we did it!

Your takeaway: A practical, battle-tested roadmap for testing-enabled Continuous Deployment. You'll leave knowing which tests to automate first, how to build confidence without sacrificing speed, and how to prove to skeptics that this isn't just another risky experiment. Real patterns, real failures, real results - ready to implement Monday morning.

Who should attend: QA engineers, test automation engineers, developers, engineering managers, and DevOps practitioners who believe testing should accelerate delivery, not slow it down.

Key Takeaways:

• The three signs your team is not ready for Continuous Deployment, and the technical enabler that breaks the testing tension. 
• How we shifted from “QA signs off on releases” to "Quality is built-in", and why the cultural change was harder than the technical one. 
• The valuable failure lessons we learned along the way and what it taught us about green pipeline.

The problem: it is too hard to understand and improve GenAI quality, and yet organizations are moving ahead regardless. For AI engineers it’s hard to:

• Increase accuracy due to lack of repeatable & representative testing
• Understand reliability: know how, why, or when an agent will fail.

This leads to poor reliability and accuracy, which:

• Increases operational costs and can increase reputational damage
• Erodes user trust, reduces customer engagement, and increases churn
• Reduces business confidence, slowing down AI adoption

In this talk I will discuss the limitations of how we are current testing AI agents, and why this means we are not adequately ensuring the safety of agentic AI systems. With non-deterministic systems like Generative/Agentic AI, we need to simulate a large number of inputs (millions) and measure the outputs using judge agents to find the statistical success rate. This a process that is more similar to how we traditionally do load testing rather than the simple functional testing we’re using with AI right now.

I will explain how you can instead use tools like AgentCore to create orchestration agents that build other types of agent to make this new type of non-deterministic testing possible. This approach will be for GenAI what traditional automated tests are for deterministic code:

• Auto generate representative testing material
• Orchestrate tests against real AI endpoints
• Judge outputs (minimum standards, accuracy quantification)
• Improve accuracy and reliability

Key takeaways:

• Current functional testing techniques are inadequate for testing agentic/generative AI systems
• What does it mean to use LLM as Judge agents? What are input agents?
• How can you create an AI testing orchestration pipeline for testing AI agents

Most of us unknowingly carry limiting beliefs about our abilities, often in the very areas where we have the most room to grow! A talented developer won't apply for a senior role because they think “I'm just not a natural leader”, never considering that leadership can be learned. Or a designer brushes off feedback believing, "Either you have an eye for design or you don't."

Sound familiar?These are not signs of missing innate talent, but rather the result of a fixed mindset, a belief that your innate talent is set in stone. The good news is that this belief can be changed by adopting a growth mindset, a belief that you are capable of growing and improving. Drawing on insights from Dr. Carol S. Dweck’s pioneering research, this talk explores how a fixed mindset makes us avoid challenges and crumble at criticism, while a growth mindset helps us see both as fuel for improvement.

I'll break down the real difference between fixed and growth mindset and we'll do an interactive exercise where you'll experience the shift yourself.The best part? Your potential isn't fixed. You will walk away with a practical strategy to stop protecting your ego and start building the skills necessary to get that senior role and tackling the challenges you've been avoiding.

Key takeaways:
Participants will leave this session with:

• An understanding of how we overestimate our knowledge
• A clear distinction between fixed mindset and growth mindset
• Practical strategies for adopting a growth mindset

Back in 2016 at trivago, we were building a new Selenium-based test framework with Cucumber, but the standard reporting tool wasn't quite fitting our needs. It showed lots of information, but finding the key details about which scenarios failed and why meant digging through charts and stats that weren't really helpful for our workflow. During a company hackathon, I decided to build something more focused on what we actually needed to see.

I used Cucumber's JSON output and some templating to create Cluecumber—a cleaner way to view test results that puts the important stuff up front. It worked well enough that we open sourced it with company backing, and eight years later it's had about 90 releases and is being used by testing teams around the world. It's been rewarding to see something that started as a weekend project actually help other people solve similar problems.

This talk covers the technical choices behind Cluecumber, but focuses more on what I learned from maintaining an open source project. From handling feature requests and common questions to keeping code clean while adding new functionality, plus the benefits of company-backed open source for everyone involved. I'll share why sometimes building your own solution makes sense, what works well for creating tools people want to use, and some insights from eight years of project maintenance.

Key takeaways:

• Understand why clear and concise reporting of test results is beneficial for all parties of the software development lifecycle
• Learn about when and where our test reports help in further exploratory testing and bug tracking
• See why it can be better to reinvent the wheel instead of going with using existing ones

Traditional software testing assumes deterministic behaviour: predictable inputs produce expected outputs. Agentic AI systems shatter this assumption. These autonomous agents make independent decisions, learn from interactions, and exhibit emergent behaviours that render traditional unit and integration testing insufficient.

This talk examines critical testing challenges through three real-world case studies:

• Voice AI Agent: Deployed across 20+ corporate environments, this system processes natural speech, maintains conversational context, and autonomously decides what additional information to provide. Traditional testing covered individual components but missed integration issues where the agent would correctly understand "Q3 sales figures" but autonomously add irrelevant market trend analysis.
• Phone Caller Agent: Handling 5,000+ patient interactions for healthcare appointment scheduling and reminders. Standard integration tests passed, but the agent failed in production when encountering background noise, elderly patients requiring slower conversations, or unexpected human responses that weren't in test scenarios.
• Chat Agent: Processing 100+ daily customer service conversations with multi-session context retention. While individual NLP components performed well, the integrated agent exhibited unexpected behaviours during complex, multi-issue conversations that spanned several sessions.

These case studies reveal five critical testing gaps:

• Non-deterministic behavior validation - the same inputs can produce different valid outputs
• Contextual decision testing - validating autonomous choices about escalation, information depth, and communication style
• Multi-modal integration complexity - components work individually but fail in integrated agent workflows
• Continuous learning validation - ensuring agent improvements don't introduce biases or degrade existing capabilities
• Real-world variability simulation - testing across acoustic environments, human communication patterns, and infrastructure variations

The presentation introduces a practical testing framework specifically designed for agentic systems: Behavioural Goal Testing (testing achievement rather than outputs), Probabilistic Validation (acceptable outcome ranges vs. exact matches), Adversarial Scenario Generation (systematic edge case creation), and Contextual Journey Simulation (multi-session user interactions).

Key takeaways:

• How to test non-deterministic AI systems with confidenceParticipants will learn how to move beyond exact assertions and design test oracles based on intent, semantics, and properties, enabling reliable validation of probabilistic LLM and agent outputs.Practical frameworks for validating LLMs and multi-agent architectures
• Attendees will gain hands-on experience testing AI systems across layers, including orchestration, inference, and inter-agent communication, using structured frameworks and real-world scenarios.
• Actionable tools to operationalize AI quality in productionThe workshop equips participants with Python-based evaluators, red teaming techniques, and automated quality metrics that can be integrated into CI/CD pipelines and governance strategies immediately.

In this talk, I show how we run a lot of full end-to-end Cypress web application tests every day. In addition to running the full data set, we do separate feature test runs based on test tags. We also allow everyone from all teams to trigger the tests right from GitHub Actions UI. This lets every group quickly test their feature before merging into the main branch.

For pull requests, we employ source code analysis based on data test IDs to run the affected tests first for quicker feedback. The software automation team uses the flake test information to chase the sources of the underlying errors to minimize noise and make every passing test run give us confidence in the released code, and every failing test run useful to quickly diagnose the real underlying issue.

The presentation covers test writing, test organization, selecting tests to run based on the source code changes, running tests in different resolutions. I also look into making the tests faster by employing data creation and caching, as well as using API calls to bypass the user interface in some places. Finally, making the tests robust and flake-free and triaging the failed runs is an ongoing activity for the automation team.

Key takeaways:

• How to run 1000 of end-to-end tests quickly
• Which tests to run on a pull request
• How AI is helping us pick tests to run

Problem Context

• Brittle selectors: We spend hours fixing fragile XPaths and CSS selectors just to verify if a button is visible or a chart is correct
• Release changes: Automation code that is stable "now" suddenly becomes flaky after the release. The reason is the ever-changing platform. And automation is not always able to cope with it
• Code volume: Code analysis in our company showed us that assertion logic is typically five times (5x) larger in code size than action logic, consuming up to three months of dedicated maintenance effort every year

Solution
In this session, I introduce ViTO (Visual Testing Oracle), a production-deployed framework that leverages multimodal Generative AI (GenAI). ViTO "sees" the application exactly like a human does. The best part is that, in the end, it's just another block of code that can be embedded inside any framework.

Summary of what's in the talk
I will share:

1. The logic and algorithm of how we used GenAI to decouple verification from the underlying code resulted in a 50% reduction in our assertion codebase.
2. How we replaced thousands of lines of brittle verification logic with resilient, prompt-driven visual oracles that can handle complex data visualisations and unseen UI faults with zero extra effort. If you are tired of your tests breaking because a div changed, it's time to shift from structural selectors to a visual AI oracle.
3. The lessons learned from our initiative, and above all, where NOT to use GenAI
4. Access to the boilerplate code that you can implement within your repo
5. If time permits, a demo of the framework in action. If short on time, the link to the boilerplate is provided in the slides :)

Who is this for?

• QA Architects, Senior SDETs, Automation Engineers, Manual testers looking to transition to GenAI-based testing;
• Managers/architects looking for a language-agnostic framework to build GenAI-based assertions
• Anyone who wishes to know where to and where NOT to use GenAI in testing
• QA professionals looking for a starting point (boilerplate) code to embed GenAI in their automation

Key takeaways:

• In-code GenAI: How to implement GenAI directly "in-code" using any programming language
• Prompt Engineering for Testers: How to write resilient "Assertion Prompts" that replace complex conditional code and handle visual regression automatically.
• Real-World ROI: Evidence-based results from a production environment, showing a 50% reduction in code maintenance and expanded coverage for rich UI components.
• Deterministic AI: Practical strategies to control GenAI hallucinations using "concentrated screenshots"
• A sneak peek into what's coming in the future in GenAI for test automation

Teams often rely on test cases executed, bugs reported, and pass rates to measure success. These numbers might look impressive, but do they truly reflect software quality? Vanity metrics can mislead teams, encourage the wrong behaviours, and create a false sense of progress.

This talk introduces a 7-step framework to move beyond superficial KPIs and focus on metrics that drive real value. Inspired by analytical approaches in competitive sports, this model helps teams make better decisions, align testing efforts with business goals, and ensure that data supports meaningful improvements.

Key takeaways:

• The risks of vanity metrics and how they can mislead decision-making.
• How to design KPIs that focus on value, not just activity.
• A practical framework to ensure testing metrics drive meaningful change
  

The security impact of missing input validation is usually underestimated.

The presentation explains and gives examples of how missing logical limits may lead to denial of service attacks on the application that seems quite secure - no injection or execution vulnerabilities needed.

As the presenter is a co-lead of the OWASP ASVS project, related security requirements are also pointed out.

Not a single word about AI.

Visual testing is supposed to protect QA teams from the familiar “it looks wrong” bug, yet traditional pixel-diff approaches only show that something changed, not whether that change actually matters. As modern interfaces grow more dynamic and design systems become more complex, teams need smarter ways to detect meaningful visual regressions.

This talk presents a practical approach to automated visual bug detection using multimodal LLMs. Drawing on a real-world implementation, it shows how AI models from providers such as OpenAI, Anthropic, and Google can be orchestrated to analyze screenshots and identify issues that pixel-based tools often cannot interpret on their own. These include layout breaks, missing elements, accessibility concerns, color contrast problems, and platform-specific guideline violations.

The session explores how AI-driven visual analysis can move beyond pixel-perfect comparison toward semantic understanding, helping teams distinguish intentional UI changes from genuine defects. It also addresses one of the biggest challenges in visual testing at scale: false positives, demonstrating how agent-based review systems can reduce noise while still surfacing critical issues.

Attendees will leave with practical ideas for using multimodal AI to strengthen visual testing workflows and make automated UI validation more accurate, scalable, and useful.

Key Takeaways:

• How to evolve from “pixel diffs” to impact-based automated visual feedback
• Patterns that turn image feedback into structured results (what changed, where, severity, why it matters)
• Tips for integrating automated LLM-powered visual feedback into existing automated UI test frameworks

99.59%.That’s not uptime, not code coverage - it’s our yearly stability rate for mobile end-to-end test runs. It sounds almost impossible, especially if you’ve ever managed a growing Slack thread titled #iHateMobile.

For three years, we fought the usual suspects of mobile automation: Appium timeouts, vanishing selectors, and flaky infrastructure. This talk condenses that journey into a survival guide for anyone who has ever wanted to throw their test phone across the room.In this fast-paced session, we will bypass the basics and dive straight into the specific architecture decisions that turned chaos into trust. We will look at how we moved beyond standard WebdriverIO implementations to build a system that is fast, predictable, and relied upon by the entire engineering organization.

We will cover the "Big Three" that solved our flakiness:

• The Framework: How small, low-level fixes in element interaction and strict state management snowballed into massive stability gains.
• The Shortcuts: Why we killed UI-based setup in favor of API data seeding and custom app states to drastically reduce execution time.
• The Orchestration: Introducing our homemade "device-thread balancer" and CI triggers that made testing "one-click" easy.

Finally, we’ll touch on the human element: how stable builds transformed our culture, turning skeptics into believers and making "just run the tests" the team's favorite phrase.

Key takeaways:

• Root Cause Analysis: Techniques for diagnosing the real source of mobile flakiness (it's not always the device).
• Speed vs. Stability: How to use API seeding and backend shortcuts to stabilize frontend tests.
• DevOps Integration: Blueprints for a "device-thread balancer" that optimizes cost and speed.
• Culture: How to build trust with developers so they treat E2E tests as an asset, not a blocker.

How do you test an application that relies heavily on cloud services? Do you have a specific strategy for testing it, or do you simply run your tests regardless of the infrastructure costs?

Nowadays, many applications rely on different cloud services, such as databases, message queues, and file storage offered by cloud providers. Those cloud services bring considerable infrastructure costs and complexity in terms of testing cloud applications. The challenges include teams relying on mocks to test the application locally and in CI/CD, as well as extra costs to create test environments that use real services. However, a good alternative to deal with that is to use emulation tools to simulate those cloud services, providing good confidence and saving a lot of costs.

In this talk, we’ll explore how Testcontainers and LocalStack offer an affordable and scalable solution to cloud application testing without compromising on quality. This session will demonstrate (including a live showcase) how to use Testcontainers in combination with LocalStack to spin up containerized services to emulate all the AWS cloud services that your application depends on, enabling you to have your own cloud running locally on your machine or CI/CD. Together, those two tools can provide an efficient, cost-saving alternative to traditional cloud testing strategies.


Key takeaways:

• Learn how to create and run tests for cloud applications using free and open-source tools while reducing the costs of infrastructure and cloud services.
• Discover some common challenges when testing cloud applications and how to deal with them.
• Understand how to use real containerized cloud services instead of mocks to make your tests more closely mimic the production environment setup.