Nordic Testing Days 2026

Traditional software is deterministic: the same input yields the same output. Large Language Models (LLMs) and AI Agents have shattered this rule, introducing an inherently probabilistic paradigm. How do we ensure quality when the ground truth is shifting? This tutorial bridges the gap between traditional QA and AI evaluation. We will move beyond simple prompt testing into validating complex multi-agent systems. Participants will learn to build test oracles that evaluate intent and semantics rather than exact matches, evolving the QA role from a code verifier to an evaluation framework architect.

Learning Objectives
By the end of this workshop, participants will be able to:

• Deconstruct AI Architectures: Identify specific testable layers such as Shell (API/UI), Orchestration (Context/Tools), and Inference Core (Probabilistic).
• Build Modern Test Oracles: Implement aggregated and property-based oracles using Python to handle non-deterministic outputs.
• Validate Multi-Agent Systems: Apply a four-level framework to test communication, delegation, and error propagation between AI agents.
• Execute AI Red Teaming: Identify vulnerabilities such as prompt injection, hallucinations, and safety bypasses.
• Automate Quality Metrics: Integrate BERTScore and RAG-specific metrics such as Faithfulness and Relevance into CI/CD pipelines.

Target Audience & Prerequisites

• Target Audience: QA Engineers, SDETs, and Developers working with or transitioning to Generative AI systems.
• Prerequisites: Basic knowledge of Python and API fundamentals. Participants must bring a laptop with VS Code and Python installed.

Workshop Outline

1. The Paradigm Shift
   • Theory: Deterministic vs. probabilistic testing. Agent taxonomy.
   • Practice: Environment setup and executing your first fuzzy test.
2. Oracles & Orchestration
   • Theory: Atomic vs. aggregated oracles. Testing the orchestration layer.
   • Practice: Writing scripts to validate JSON schemas and output consistency.
3. Semantic Evaluation
   • Theory: RAG metrics such as Faithfulness and Relevance. Introduction to BERTScore.
   • Practice: Building an LLM-as-a-Judge evaluator to grade complex answers.
4. Multi-Agent Testing
   • Theory: Inter-agent communication and task delegation loops.
   • Practice: Debugging a workflow where a Travel Agent delegates to a Finance Agent.
5. Red Teaming & Security
   • Theory: Prompt injection, mutation testing, and metamorphic testing.
   • Practice: Simulated attack scenarios, bypassing safety filters, and implementing guardrail fixes.
6. QA Strategy & Governance
   • Theory: Human-in-the-loop workflows and production monitoring.
   • Practice: Designing a full-scale QA strategy for a real-world GenAI product.