Software Engineer – LLM Evaluation & Repository Validation

Software Engineer – LLM Evaluation & Repository Validation

We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.

Key Responsibilities

• Review and triage issues in open-source GitHub projects.
• Prepare repositories for evaluation, including containerisation and environment automation.
• Assess how thorough and well written the unit tests are.
• Edit and run code locally to see how LLMs perform on bug-fixing tasks.
• Help pick repositories and issues that are hard for LLMs.
• Write up findings clearly for researchers and reviewers.
• Follow project guidelines, take part in calibration and review sessions, and incorporate feedback to keep quality consistent.

Requirements

• 3+ years of software engineering experience.
• Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby.
• Proficiency with Git, Docker and basic pipeline setup.
• Ability to navigate complex code bases and run real-world projects locally.
• Fluent written English.
• Reliable, detail-oriented, and able to work independently and meet deadlines in a remote setting.

Preferred Qualifications

• Prior LLM research or evaluation projects.
• Experience building or testing developer tools or automation agents.
• Open-source contribution or evaluation experience.

• Submit only work that you created yourself or that you have the right to share.
• Do not submit material covered by an employer, client or third-party confidentiality agreement, or material you have no right to share.
• Do not submit export-controlled, military or defence-related technical data.
• Remove credentials, secrets and personal data from anything you submit.
• Rex may ask for evidence of ownership or licence, may verify submissions, and may reject or remove any submission.

Compensation

USD $50–100 per hour, adjusted by country of residence.

About RemoExperts

RemoExperts (Rex.zone) is a platform connecting domain experts with AI companies for RLHF (Reinforcement Learning from Human Feedback), data annotation, and AI training tasks. They specialize in matching subject matter experts with AI projects that require deep domain knowledge, offering competitive pay for remote, flexible work.

How To Apply

Click the Apply button to view the full job details and submit your application on the RemoExperts portal.