AI Evaluation and Expert Projects
Software Engineer, Coding-Agent Evaluator & AI Task Author
Use real software-engineering judgment to create, solve and evaluate realistic tasks for coding agents and AI development tools.
A real current or expected client need is being sourced now.
Join This Career CommunityAbout this path
This is expert evaluation work for experienced engineers. It may include repository-based task creation, gold-standard behavior, comparison of solutions, failure-mode analysis and defensible technical rationales. It is not generic data labeling.
You will use genuine software-engineering experience to create and evaluate realistic tasks for coding agents, including repository-based problems, gold-standard solutions and failure-mode analysis. This is expert evaluation work, not generic data labeling, and it rewards engineers who can write a defensible rationale for why one solution is better than another.
What you would own
- Creation of realistic, repository-based coding tasks for agent evaluation
- Authoring of gold-standard solutions and expected behavior
- Comparison and scoring of candidate solutions against defined criteria
- Failure-mode analysis of coding-agent outputs
- Written, defensible technical rationales for evaluation decisions
- Collaboration with evaluation leads on rubric and calibration questions
You are likely a strong match if
- You have professional software-engineering experience
- You can write and reason about nontrivial, realistic codebases
- You can clearly explain why one technical solution is better than another
- You are comfortable with detailed, sometimes repetitive evaluation work
- You have experience with the languages or frameworks relevant to the project
- You take rubric and instruction-following seriously
Helpful, not required
- Experience with automated testing and CI pipelines
- Familiarity with AI coding assistants or agents
- Open-source contribution history
- Experience mentoring or reviewing other engineers' code
What success looks like
- Tasks you author are realistic and expose meaningful agent failures
- Your evaluation rationales are clear enough for others to audit
- Calibration with other evaluators stays consistent over time
- Clients trust the quality bar behind your submitted work
Practical proof that helps
- Links to public code repositories or a portfolio of past engineering work
- A writing sample showing technical reasoning about a coding tradeoff
- Prior experience with code review or technical evaluation
What being in the network gives you
- Remote-first work with clients across the United States, Canada and Latin America.
- Human review of your profile, automation organizes information, people decide.
- One profile considered across current and future opportunities.
- Referral rewards when someone you refer directly is successfully placed.
- Full control over availability, matching and your data at any time.
One profile, many opportunities
Applying here creates a single reusable profile. If this path is not the right fit, you remain eligible for other suitable opportunities across the network.
Refer someone for this path
Know a professional who fits this career path?
Share this path directly. Referring is always optional, and candidate review stays merit based.
Apply and join the network
Free to join • Start in about 60 seconds • One profile for multiple opportunities • No advanced AI experience required for many roles
Joining the network does not guarantee immediate work or placement. Opportunities depend on professional fit, location, availability and client demand.