Software Engineering Evaluation Specialist 

Engagement :- Part-Time
Work Mode :- Fully Remote
Expected Commitment :- Approximately 15 hours per week
Schedule :- Flexible

Compensation :- ₹6,000–₹11,000 per hour

Role Overview

We are looking for an experienced Software Engineering Evaluation Specialist to contribute to the development and evaluation of advanced AI-driven software engineering systems.

This role sits at the intersection of software engineering, automated evaluation, developer tooling and AI systems. The selected professional will work on technically challenging engineering scenarios designed to assess how effectively AI agents can understand unfamiliar codebases, interact with development tools, diagnose problems and implement reliable solutions.

You will be working with structured engineering environments where software agents are required to investigate repositories, reason through technical dependencies, modify existing implementations and demonstrate that their solutions satisfy predefined engineering requirements.

Prior professional experience in Artificial Intelligence or Machine Learning is not mandatory. Strong software engineering fundamentals, problem-solving ability and experience working with real-world codebases are more important.

Key Responsibilities :-

Design and develop realistic software engineering scenarios for evaluating AI-based coding agents.
Create controlled development environments in which complex engineering tasks can be reproduced consistently.
Develop tasks involving bug diagnosis, feature implementation, refactoring, optimization and codebase maintenance.
Establish reliable validation mechanisms to determine whether an AI-generated solution satisfies functional and technical requirements.
Develop deterministic testing and verification workflows wherever applicable.
Prepare reference implementations against which generated solutions can be evaluated.
Analyze software repositories and identify appropriate points of intervention for engineering tasks.
Structure tasks that require an agent to locate, interpret and utilize information available through developer-facing tool interfaces.
Evaluate not only the final code output but also the effectiveness of the engineering process used to arrive at the solution.
Identify edge cases, hidden dependencies, regression risks and performance bottlenecks.
Improve task reproducibility, reliability and evaluation accuracy.
Work with complex codebases while maintaining clean engineering practices and documentation.
Review implementations for correctness, maintainability, scalability and performance.
Collaborate asynchronously with technical and project stakeholders in a distributed environment.

Technical Expertise Required :-

Candidates should have strong hands-on software engineering experience in one or more of the following :-

Python 3
Java
C++
Rust
Go / Golang
TypeScript

Strong proficiency in at least one programming language is required, while exposure to multiple languages is highly desirable.

Core Engineering Competencies

Advanced understanding of algorithms and data structures
Strong debugging and root-cause analysis capabilities
Experience diagnosing complex software defects
Feature development within existing codebases
Codebase refactoring and modernization
Performance profiling and optimization
Understanding of computational complexity
Writing maintainable and testable production-quality code
Regression analysis and failure investigation
Ability to reason about unfamiliar repositories quickly
Understanding of software scalability and reliability considerations

Engineering Evaluation & Tooling :-

The role may involve working with modern developer-tool interfaces and structured tool-access mechanisms through which an AI agent interacts with software repositories and development environments.

You should be comfortable thinking beyond simply “does the code run?” and evaluating :-

Whether an implementation satisfies the underlying specification
Whether the solution introduces regressions
Whether edge cases are handled correctly
Whether the implementation is deterministic and reproducible
Whether the solution is appropriately optimized
Whether the changes are consistent with the architecture of the existing codebase
Whether an automated agent was able to effectively navigate available technical information and tooling

Experience with AI systems, agentic coding systems, machine learning workflows, or developer-tool protocols is advantageous but not a prerequisite.

What We're Looking For

The ideal candidate is someone who can take an ambiguous engineering problem, investigate the surrounding code and dependencies, determine the correct implementation strategy and produce a solution that can withstand rigorous automated validation.

You should demonstrate :-

Strong analytical and logical reasoning
Excellent software debugging skills
Ability to work independently with limited supervision
High attention to technical detail
Strong written technical communication
Ability to understand unfamiliar systems quickly
Discipline around reproducibility and testing
Ability to distinguish between superficial fixes and robust engineering solutions
Experience working collaboratively within distributed or remote engineering teams

Preferred Background :-

The following will strengthen your application :-

Experience with large or distributed software repositories
Experience contributing to open-source or large-scale engineering projects
Exposure to CI/CD and automated testing frameworks
Experience with performance profiling or benchmarking
Familiarity with code review and engineering quality standards
Experience designing test cases or evaluation frameworks
Exposure to AI-assisted development tools or autonomous coding agents
Familiarity with structured developer tooling and API-driven workflows
Experience working on systems where reliability and reproducibility are critical