GenAI Engineer – Test Agent / CI Integration

GenAI Engineer – Test Agent / CI Integration

GenAI Engineer – Test Agent / CI Integration

Location: Gurugram, India ( hybrid)

Experience: 4–8 Years

Employment Type: Full-Time

Job Description:

We are looking for a highly skilled GenAI Engineer – Test Agent / CI Integration to build and scale intelligent AI-powered testing systems for next-generation applications. The ideal candidate will work on automated test-agent frameworks, synthetic data generation, evaluation harnesses, and CI/CD-integrated AI testing pipelines.

This role requires strong expertise in Python backend development, LLM evaluation frameworks, retrieval-grounded testing systems, and modern DevOps practices.

Key Responsibilities:

AI Test Agent Development

• Design and develop autonomous AI-driven test agents for validating GenAI and LLM-powered applications

• Build systems for:

• Synthetic data generation

• Test-case synthesis

• Scenario generation

• Adversarial and edge-case testing

• Develop reusable evaluation harnesses for benchmarking model quality, accuracy, safety, and reliability

Context-Aware Test Generation

• Integrate test agents with BLK’s knowledge/context graph for retrieval-grounded testing

• Enable contextual test generation using RAG pipelines and graph-based retrieval systems

• Ensure generated tests align with enterprise knowledge sources and real-world workflows

CI/CD & Automation

• Integrate AI test agents into CI/CD pipelines as first-class pipeline jobs

• Automate regression testing, evaluation runs, and quality scoring during deployments

• Build scalable validation workflows for continuous model monitoring and release gating

Evaluation Frameworks & Quality Engineering

• Work with LLM evaluation frameworks such as:

• DeepEval

• Ragas

• Custom evaluation frameworks

• Develop automated scoring mechanisms for:

• Hallucination detection

• Faithfulness

• Relevance

• Toxicity

• Response quality

• Integrate with pytest and existing QA ecosystems

Backend & Infrastructure

• Build and maintain Python backend services powering evaluation workflows

• Optimize distributed evaluation execution for scalability and performance

• Collaborate with platform, MLOps, and DevOps teams for production deployment

Required Skills & Qualifications:

Technical Skills

• Strong proficiency in Python

• Experience with:

• Pytest
• CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI, etc.)
• REST APIs & backend development

• Hands-on experience with:

• LLM evaluation frameworks (DeepEval, Ragas, LangSmith, custom evaluators)
• RAG systems and retrieval pipelines
• Synthetic dataset generation
• Prompt engineering and evaluation strategies

AI/ML & GenAI Expertise

• Strong understanding of:

• Large Language Models (LLMs)
• Agentic systems
• AI evaluation methodologies
• Context grounding and knowledge retrieval

• Familiarity with vector databases, embeddings, and knowledge graphs

DevOps & Automation

• Experience integrating AI workflows into CI/CD environments

• Understanding of automated quality gates and testing orchestration

Preferred Qualifications

• Experience working with knowledge graphs or graph databases

• Exposure to LangChain, LlamaIndex, or similar orchestration frameworks

• Familiarity with Kubernetes, Docker, and cloud platforms (AWS/GCP/Azure)

• Experience in enterprise-scale AI platform engineering

Important Note:

This is a niche requirement and not a regular GenAI developer role. We are specifically looking for candidates with experience in:

• AI validation/testing

• QE automation

• Python backend

• CI/CD integration

• LLM evaluation frameworks

• RAG and retrieval-grounded systems

About Turing

Turing is the world's leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. They connect elite AI professionals with frontier model training projects, offering competitive compensation for domain expertise across coding, STEM, creative writing, and more.

How To Apply

Click the Apply button to view the full job details on Turing and submit your application.