CoreHRx Jobs

Apply in one click, save jobs and take free mock tests.

Register free Log in
Hiring? Post jobs with CoreHRx
My account
user

View & update profile

Test Analyst - AI Assurance

Hexaware

Full time
Experience
4+ years
Location
India
Work mode
Not stated
Posted 8 days ago

Free · takes a minute · your resume fills your profile

CoreHRx never asks candidates for money. If anyone does, .

Role at a glance

You will
  • Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on…
  • Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving…
  • Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry…
Experience
4+ years
Must-have skills
GalileoOpenTelemetryPythonRoot Cause AnalysisInstrumentation

How well do your skills match this job?

Sign in to see your fit: the skills, experience and location this job asks for, checked against your profile.

Job description

Location: India

Job Description

Test Analyst - AI Assurance
Role Overview
Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on experience in implementing Python-based AI evaluation frameworks such as Ragas, DeepEval, and similar tools. Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving Responsible AI (RAI) validation and reporting key RAI metrics. Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry (OTEL) instrumentation to enable effective monitoring, diagnostics, and continuous quality improvement of AI systems.
Key Responsibilities

  • Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
  • Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
  • Experience in Golden dataset curation & Synthetic dataset generation
  • Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
  • Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
  • Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
  • Experience in Integrating AI evaluation into CI/CD pipeline
  • Experience in Responsible AI validation

Required Skills and Capabilities

  • Strong understanding of GenAI System Architecture of RAG & Agents
  • Proficiency in Python & AI evaluation SDKs
  • AI Evaluation Frameworks (Ragas, DeepEval)
  • Model Evaluation
  • Agent Evaluation
  • Responsible AI (RAI) Validation
  • RAI Metrics Reporting
  • Golden Dataset Curation
  • Synthetic Dataset Generation
  • AI Observability
  • AI Tracing
  • OpenTelemetry (OTEL)
  • LLM Evaluation
  • AI Quality Engineering
  • AI Governance
  • AI Monitoring
  • AI Testing & Assurance
  • Root Cause Analysis (AI Systems)
  • Knowledge of regulatory considerations for AI systems

Role-Specific Requirements

  • Proficiency in Python & AI evaluation SDKs
  • Strong understanding of RAG & agentic system architecture
  • Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
  • Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
  • Experience in Golden dataset curation & Synthetic dataset generation
  • Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
  • Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
  • Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
  • Experience in Integrating AI evaluation into CI/CD pipeline
  • Experience in Responsible AI validation
  • Knowledge of regulatory considerations for AI systems

Experience Requirements
More than 4 years of experience in AI Assurance Testing

Job Info
Job Identification
689789
Posting Date
11/11/2026
Apply Before
11/30/2026
Locations
India
Lane of Recruitment
Lane 3

Job data: Headcount Labs (headcountlabs.com), ODbL 1.0. The job description belongs to the employer. Location data © OpenStreetMap contributors and Overture Maps.

Key skills

Required

GalileoOpenTelemetryPythonRoot Cause AnalysisInstrumentation

Tags

Job market insights

10active Galileo jobs in India
182active OpenTelemetry jobs in India
5,900active Python jobs in India
30 more open jobs at Hexaware

Jobs asking for Galileo in India also ask for

  • Stakeholder Management40%
  • Amadeus30%
  • Python30%
  • CSS20%
  • Design Systems20%

From live job posts on CoreHRx, updated 19 Nov 2026. Skill figures count posts that list their skills; salary figures only posts that show a salary.

Questions about this job

How much experience does this job need?

The post asks for 4 Yrs experience.

What kind of job is this?

It is a full time job.

This page helps Indian job seekers prepare for Python interviews for backend, data engineering and data analyst roles, whether you are a fresher facing your first technical round or an experienced developer aiming for a senior position. It has 60 questions in three levels, easy, medium and hard, each with a concise sample answer and, where it helps, runnable Python 3.12 code. Topics include data structures, OOP, generators, decorators, the GIL and concurrency, type hints, testing and performance. Answer each question aloud in your own words first, then compare with the sample answer and run the code to check your understanding.

Questions you may be asked

  1. What is the difference between mutable and immutable types in Python?
  2. What is the difference between a list and a tuple?
  3. What is the difference between is and ==?
  4. What are *args and **kwargs?

Get IT jobs by email

Create a free profile and we email you new IT jobs, at most once a week.

Similar jobs

Free · takes a minute · your resume fills your profile