Remote | senior software engineer – llm evaluation

We are sharing a specialised part-time consulting opportunity for experienced Senior Software Engineers to contribute to advanced large language model evaluation, coding benchmark development, and AI-assisted software-engineering research. Selected professionals will curate and evaluate code, develop verification mechanisms, assess AI-generated software across multiple programming languages, and help researchers understand where advanced models succeed or fail throughout the software-development lifecycle. The work combines senior-level software engineering judgement with structured evaluation and benchmark design.

Key Responsibilities

• Code Curation & Solution Development
  • Curate high-quality code examples for model training and benchmarking
  • Develop precise reference solutions to software-engineering tasks
  • Correct and improve code across multiple programming languages
  • Work with Python, JavaScript, ReactJS, C/C++, Java, Rust, and Go
  • Maintain strong standards for correctness, clarity, and maintainability
• AI-Generated Code Evaluation
  • Evaluate AI-generated code for technical correctness
  • Assess solutions for efficiency, scalability, and reliability
  • Identify implementation weaknesses and recurring error patterns
  • Review code quality against professional engineering standards
  • Provide structured rationales supporting evaluation decisions
• Verification & Automated Assessment
  • Design mechanisms that automatically verify software-engineering solutions
  • Build agents capable of assessing code quality
  • Develop approaches for identifying common model failure patterns
  • Create deterministic or structured checks where appropriate
  • Support reliable evaluation across repeated coding tasks
• Software Engineering Lifecycle Evaluation
  • Evaluate model capabilities across the software-development lifecycle
  • Assess reasoning around prototyping and architecture design
  • Review API design and production implementation decisions
  • Evaluate launch, experimentation, monitoring, and operational scenarios
  • Identify where models struggle with real-world engineering workflows
• Research & Benchmark Collaboration
  • Collaborate with researchers and cross-functional technical teams
  • Contribute to datasets used for training and benchmarking
  • Help define engineering evaluation strategies and quality standards
  • Compare model performance against professional engineering expectations
  • Support iterative improvements to coding-focused evaluation systems

Ideal Profile

• Several years of professional software-engineering experience
• At least 2 years of continuous full-time experience at a top-tier product company
• Strong full-stack development capabilities
• Experience building scalable, production-grade software
• Deep understanding of software architecture and system design
• Strong knowledge of debugging and code-quality assessment
• Experience reviewing and improving complex production code
• Proficiency in one or more of Python, JavaScript, C/C++, Java, Rust, or Go
• ReactJS experience is valuable for frontend-oriented assignments
• Strong understanding of API design and production implementation
• Familiarity with software monitoring and operational maintenance
• Ability to reason across the complete software-engineering lifecycle
• Strong analytical and problem-solving capabilities
• Excellent written and verbal communication skills
• Ability to produce clear and structured evaluation rationales
• Comfortable collaborating with research and technical teams in a remote environment

Engagement Details

• Part-time independent contractor engagement
• Fully remote
• Minimum commitment: 10 hours per week
• Flexible workload of up to 40 hours per week
• Partial overlap with PST working hours is required
• Initial project duration is approximately 1 month
• Project is expected to begin the following week
• Extension may be available depending on performance and project fit
• No medical or paid-leave benefits are included under the contractor arrangement
• Work will involve coding evaluation, benchmark development, verification systems, and software-engineering research
• Assignments may span Python, JavaScript, ReactJS, C/C++, Java, Rust, Go, and related technologies
• Work must be completed without using confidential, proprietary, unreleased, employer-restricted, client-restricted, or otherwise protected code, datasets, architecture materials, or technical information belonging to any employer, client, institution, or other third party

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy