Agentic AI Technical Test Lead Poland, GDANSK POZNAN WARSAW WROCLAW

• Strong background in Quality Engineering, test automation, AI/ML testing, software testing or technical test leadership
• Practical understanding of generative AI, LLM-based applications, RAG and Agentic AI solutions
• Ability to define test strategies, quality gates and evaluation approaches for AI solutions
• Experience with AI quality dimensions such as accuracy, pecision / recall, relevance, groundedness, hallucination risk, robustness, consistency, safety and compliance
• Ability to design evaluations for prompts, AI-generated responses, agent workflows, tool-calling, multi-step reasoning, RAG pipelines and human-in-the-loop review
• Good technical understanding of APIs, integrations, cloud platforms, CI/CD and solution architecture
• Hands-on experience with Python or similar scripting/programming language
• Excellent English, both written and spoken
• Strong presentation and stakeholder management skills
• Consulting mindset, ownership attitude and ability to support sales / pre-sales activities
• Passion for AI and willingness to actively share knowledge across teams
• Residence in the European Union

🎯Preferred experience

Hands-on experience testing LLM, RAG or Agentic AI solutions
Familiarity with modern AI evaluation and testing tools such as DeepEval, RAGAS, promptfoo, OpenAI Evals, Braintrust, LangSmith, TruLens, Giskard, MLflow Evaluation or Arize Phoenix
Experience with AI observability and monitoring platforms such as LangSmith, Langfuse, Arize Phoenix, MLflow, Weights & Biases Weave, Comet Opik, AgentOps, Datadog LLM Observability or Helicone
Familiarity with AI agent / orchestration frameworks such as LangGraph, LangChain, Microsoft Semantic Kernel, AutoGen, CrewAI or similar
Experience with RAG components, including vector databases, embeddings, document ingestion and retrieval evaluation
Understanding of Responsible AI, AI governance, privacy, security and regulatory expectations
Experience with DevOps, MLOps or LLMOps practices
Previous consulting, client-facing delivery or business development experience

As an Agentic AI Technical Test Lead, you will be responsible for defining and leading the quality assurance approach for AI agent solutions delivered to enterprise clients.

You will create testing standards, prepare technical solutions, support proposals and presentations, and lead execution across AI testing workstreams.

We are looking for an experienced Agentic AI Technical Test Lead to define, build and execute testing approaches for AI agent solutions in a consulting environment.
The ideal candidate combines strong Quality Engineering expertise with hands-on understanding of LLMs, RAG, AI agents, evaluation frameworks and enterprise delivery. This person should be proactive, client-facing, passionate about AI, and able to set standards while also supporting execution, presentations and business development.

What we offer:

• Opportunity to shape testing standards for Agentic AI solutions


• Work on innovative AI engagements for enterprise clients


• Collaboration with AI, data, cloud, cybersecurity and consulting experts


• Career path toward AI Quality Engineering Lead, AI Assurance Architect or AI Testing Offering Lead


• Space to build thought leadership, accelerators and reusable AI testing assets


• A collaborative environment focused on innovation, ownership and knowledge sharing

,[Define reusable testing standards for Agentic AI solutions, Create evaluation frameworks for prompts, RAG, agent workflows and tool-calling, Define quality gates, acceptance criteria, test metrics and reporting templates, Establish approaches for accuracy, precision, hallucination, robustness and safety evaluation, Build reusable assets, accelerators and best practices for AI testing, Lead test design and execution for AI agents and LLM-based applications, Create golden datasets, evaluation scenarios and adversarial test cases, Validate agent reasoning, tool usage, API interactions and workflow orchestration, Support automated and human-in-the-loop evaluation processes, Analyze results and provide clear recommendations to technical and business stakeholders, Design technical testing solutions for client engagements, Embed evaluation into CI/CD, DevOps, MLOps or LLMOps pipelines, Support observability, monitoring, feedback loops and production quality controls, Work with architects, AI engineers, data engineers and business teams, Identify quality risks and propose practical mitigation actions, Support pre-sales, proposals, estimations and client presentations, Prepare technical materials explaining AI testing approaches, Facilitate workshops and discovery sessions, Act as a trusted advisor on AI quality, reliability and risk, Promote AI testing knowledge internally and externally] Requirements: AI, Testing, Cloud platform, CD, Python, Stakeholder management, MLflow, Datadog, Security, DevOps, MLOps