AI Evaluation and Expert Projects
Software Engineer, Coding-Agent Evaluator & AI Task Author
Use real software-engineering judgment to create, solve and evaluate realistic tasks for coding agents and AI development tools.
About this path
This is expert evaluation work for experienced engineers. It may include repository-based task creation, gold-standard behavior, comparison of solutions, failure-mode analysis and defensible technical rationales. It is not generic data labeling.
You will use genuine software-engineering experience to create and evaluate realistic tasks for coding agents, including repository-based problems, gold-standard solutions and failure-mode analysis. This is expert evaluation work, not generic data labeling, and it rewards engineers who can write a defensible rationale for why one solution is better than another.
What you would own
- Creation of realistic, repository-based coding tasks for agent evaluation
- Authoring of gold-standard solutions and expected behavior
- Comparison and scoring of candidate solutions against defined criteria
- Failure-mode analysis of coding-agent outputs
- Written, defensible technical rationales for evaluation decisions
- Collaboration with evaluation leads on rubric and calibration questions
You are likely a strong match if
- You have professional software-engineering experience
- You can write and reason about nontrivial, realistic codebases
- You can clearly explain why one technical solution is better than another
- You are comfortable with detailed, sometimes repetitive evaluation work
- You have experience with the languages or frameworks relevant to the project
- You take rubric and instruction-following seriously
Helpful, not required
- Experience with automated testing and CI pipelines
- Familiarity with AI coding assistants or agents
- Open-source contribution history
- Experience mentoring or reviewing other engineers' code
What success looks like
- Tasks you author are realistic and expose meaningful agent failures
- Your evaluation rationales are clear enough for others to audit
- Calibration with other evaluators stays consistent over time
- Clients trust the quality bar behind your submitted work
Practical proof that helps
- Links to public code repositories or a portfolio of past engineering work
- A writing sample showing technical reasoning about a coding tradeoff
- Prior experience with code review or technical evaluation
What being in the network gives you
- Remote-first work with clients across the United States, Canada and Latin America.
- Human review of your profile, automation organizes information, people decide.
- One profile considered across current and future opportunities.
- Referral rewards when someone you refer directly is successfully placed.
- Full control over availability, matching and your data at any time.
One profile, many opportunities
Applying here creates a single reusable profile. If this path is not the right fit, you remain eligible for other suitable opportunities across the network.
Apply and join the network
Free to join • Start in about 60 seconds • One profile for multiple opportunities • No advanced AI experience required for many roles
Joining the network does not guarantee immediate work or placement. Opportunities depend on professional fit, location, availability and client demand.