Free · takes a minute · your resume fills your profile
CoreHRx never asks candidates for money. If anyone does, .
Role at a glance
You will
Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on…
Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving…
Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry…
Experience
4+ years
Must-have skills
GalileoOpenTelemetryPythonRoot Cause AnalysisInstrumentation
How well do your skills match this job?
Sign in to see your fit: the skills, experience and location this job asks for, checked against your profile.
Job description
Location: India
Job Description
Test Analyst - AI Assurance Role Overview Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on experience in implementing Python-based AI evaluation frameworks such as Ragas, DeepEval, and similar tools. Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving Responsible AI (RAI) validation and reporting key RAI metrics. Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry (OTEL) instrumentation to enable effective monitoring, diagnostics, and continuous quality improvement of AI systems. Key Responsibilities
Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
Experience in Golden dataset curation & Synthetic dataset generation
Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
Experience in Integrating AI evaluation into CI/CD pipeline
Experience in Responsible AI validation
Required Skills and Capabilities
Strong understanding of GenAI System Architecture of RAG & Agents
Proficiency in Python & AI evaluation SDKs
AI Evaluation Frameworks (Ragas, DeepEval)
Model Evaluation
Agent Evaluation
Responsible AI (RAI) Validation
RAI Metrics Reporting
Golden Dataset Curation
Synthetic Dataset Generation
AI Observability
AI Tracing
OpenTelemetry (OTEL)
LLM Evaluation
AI Quality Engineering
AI Governance
AI Monitoring
AI Testing & Assurance
Root Cause Analysis (AI Systems)
Knowledge of regulatory considerations for AI systems
Role-Specific Requirements
Proficiency in Python & AI evaluation SDKs
Strong understanding of RAG & agentic system architecture
Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
Experience in Golden dataset curation & Synthetic dataset generation
Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
Experience in Integrating AI evaluation into CI/CD pipeline
Experience in Responsible AI validation
Knowledge of regulatory considerations for AI systems
Experience Requirements More than 4 years of experience in AI Assurance Testing
Job Info Job Identification 689789 Posting Date 11/11/2026 Apply Before 11/30/2026 Locations India Lane of Recruitment Lane 3
This page helps Indian job seekers prepare for Python interviews for backend, data engineering and data analyst roles, whether you are a fresher facing your first technical round or an experienced developer aiming for a senior position. It has 60 questions in three levels, easy, medium and hard, each with a concise sample answer and, where it helps, runnable Python 3.12 code. Topics include data structures, OOP, generators, decorators, the GIL and concurrency, type hints, testing and performance. Answer each question aloud in your own words first, then compare with the sample answer and run the code to check your understanding.
Questions you may be asked
What is the difference between mutable and immutable types in Python?
What is the difference between a list and a tuple?