Consultant - GenAI Risks & Governance

Chryselys is a Great Place to Work Certified Pharma Analytics & Business consulting company that delivers data-driven insights leveraging AI-powered, cloud-native platforms to achieve high-impact transformations. We specialize in digital technologies and advanced data science techniques that provide strategic and operational insights.

Practice: Generative AI, AI Governance & Engineering
Team: GATE (Generative AI Technology, Architecture & Evaluation)
Experience: 4–8 years

Chryselys is seeking a hands-on Generative AI Consultant to design, develop, evaluate, and improve reliable, safe, and scalable AI solutions. The role focuses on agentic workflows, LLM evaluation, AI safety, and governance for enterprise applications, particularly in healthcare and pharmaceutical domains.

The candidate will work with AI engineers, data scientists, domain experts, product teams, and client stakeholders to develop production-oriented AI prototypes, establish evaluation frameworks, implement guardrails, and ensure AI systems meet defined standards for quality, reliability, privacy, and responsible use.

1. Generative AI & Agentic Systems

• Develop and refine LLM-powered applications for interactive question answering, intelligent search, knowledge discovery, and workflow automation.
• Implement agentic workflows involving tool calling, multi-step execution, retrieval, structured outputs, and error handling.
• Build and optimize RAG pipelines, including document processing, chunking, metadata extraction, embeddings, hybrid search, and reranking.
• Integrate foundation models and AI services using Python, APIs, and frameworks such as LangChain, LangGraph, or equivalent technologies.

2. LLM Evaluation & Reliability Engineering

• Design scalable evaluation frameworks to benchmark LLM outputs, retrieval quality, agent reliability, tool-calling accuracy, and task completion.
• Develop evaluation datasets, test cases, ground-truth references, edge cases, and regression suites.
• Define and implement metrics for correctness, groundedness, hallucination, citation accuracy, retrieval precision/recall, latency, and cost.
• Implement automated and human-in-the-loop evaluations, including LLM-as-a-Judge approaches where appropriate.
• Analyze failure patterns and recommend improvements to prompts, models, retrieval pipelines, and agent orchestration.

3. AI Governance, Safety & Risk Management

• Implement guardrails to mitigate hallucinations, prompt injection, jailbreaks, sensitive-data leakage, and unsafe tool execution.
• Establish validation, access-control, human-review, escalation, and safe-fallback mechanisms appropriate to the use case.
• Support AI risk assessments, responsible AI practices, auditability, traceability, and documentation of model limitations and known failure modes.
• Align engineering controls with applicable frameworks and requirements, including NIST AI RMF, the EU AI Act where relevant, and organizational policies.
• Apply privacy-by-design principles for confidential, personal, and healthcare-related information.

4. Python Engineering & Production Monitoring

• Develop modular, reusable, and testable Python components for AI integrations, evaluation libraries, benchmarking, and reporting.
• Apply software engineering best practices, including Git, code reviews, automated testing, API development, and CI/CD.
• Support reproducible experiments through versioned datasets, prompts, models, and evaluation configurations.
• Monitor AI quality, agent failures, latency, cost, and safety; conduct root-cause analysis and support continuous improvement.

5. Client Consulting & Collaboration

• Translate client requirements into technical specifications, evaluation criteria, and measurable acceptance thresholds.
• Collaborate with cross-functional engineering, product, domain, and leadership teams to deliver scalable AI solutions.
• Communicate findings and recommendations through technical documentation, evaluation reports, demonstrations, and client presentations.

• Bachelor's or Master's degree in Computer Science, Data Science, AI/ML, Biomedical Informatics, or a related quantitative discipline; advanced degrees are preferred.
• Strong proficiency in Python, modular software design, Git, testing, API integration, and software engineering practices.
• Practical experience with LLMs, prompt engineering, RAG, embeddings, vector search, and agentic workflows.
• Hands-on experience developing or implementing LLM evaluation, benchmarking, regression testing, or AI reliability frameworks.
• Understanding of evaluation metrics for answer correctness, groundedness, retrieval effectiveness, hallucination, and agent/tool performance.
• Experience implementing AI guardrails, output validation, safe fallbacks, and security controls for LLM applications.
• Familiarity with cloud AI platforms, preferably AWS Bedrock, and deployment, logging, monitoring, and CI/CD practices.
• Strong analytical, problem-solving, documentation, and stakeholder communication skills.

• Experience in healthcare, life sciences, pharmaceuticals, clinical information, or other regulated domains.
• Familiarity with evaluation tools such as Ragas, DeepEval, MLflow, or promptfoo.
• Knowledge of responsible AI, AI risk management, privacy, and applicable governance frameworks.
• Exposure to healthcare terminology standards such as FHIR, SNOMED CT, UMLS, or RxNorm, where relevant.

• Working AI agent and RAG prototypes with documented limitations.
• Reusable evaluation frameworks, datasets, automated benchmarks, and regression pipelines.
• AI quality scorecards, guardrail implementations, and documented risk-mitigation recommendations.
• Evidence-based recommendations for improving AI accuracy, reliability, safety, and production readiness.

A hands-on GenAI professional who combines strong Python engineering with rigorous evaluation practices and practical AI governance. The candidate should be able to demonstrate how an AI system performs, identify where and why it fails, assess associated risks, and recommend measurable improvements for reliable enterprise deployment.

Chryselys is proud to be an Equal Employment Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.