Machine Learning Engineer – Evals

Location: Brooklyn, New York

Work Arrangement: Onsite, five days per week

Employment Type: Full-time

Base Salary: $220,000–$300,000 annually, plus equity

Relocation Assistance: $5,000 stipend available

About the Opportunity

Our client is an engineering-driven AI research company developing infrastructure for AI memory, identity modeling, and personalization. Its work combines machine learning and cognitive science to help AI applications understand users and adapt their behavior over time. Join a small, research-focused engineering team where you will report directly to the CTO, contribute to open-source work, and help shape how advanced language models are evaluated and improved.

Position Overview

We are seeking a Machine Learning Engineer specializing in evaluations to design, build, and maintain complex evaluation frameworks, simulators, and environments for reinforcement learning (RL) post-training. This role requires both demonstrated research ability and strong software engineering skills. The ideal candidate has created novel LLM evaluations from scratch, produced high-quality published or open-source research, and developed production-quality Python systems.

Key Responsibilities

• Design and build novel evaluation frameworks that measure LLM capabilities, behavior, and performance.
• Develop complex simulators and RL environments to support model evaluation and post-training.
• Build and validate LLM-as-judge methods, including scoring criteria and assessment reliability.
• Create simulated environments and synthetic user interactions to assess memory, identity modeling, and personalization.
• Collaborate with researchers and engineers to translate research questions into reproducible experiments and actionable findings.
• Develop and maintain production-quality Python code, including tests, documentation, and ongoing maintenance.
• Use experiment tracking and evaluation results to identify model limitations and guide improvements.
• Integrate evaluation workflows into model development and the team’s regular release cycle.
• Contribute to open-source research, evaluation tools, and benchmarks.

Required Qualifications

• At least three years of experience designing and building LLM evaluations.
• Demonstrated experience designing, building, and operating complex evaluations or simulators for RL post-training, such as LLM-as-judge systems, RL environments, or simulations.
• A strong research record demonstrated through first-author or co-author publications at NeurIPS, ICML, ICLR, or equivalent venues, or self-published open-source research of comparable quality.
• Production-quality Python skills, including testing, maintainability, and experience supporting regular software releases.
• Hands-on experience with PyTorch, Hugging Face, and Weights & Biases.
• Experience designing original evaluation methods beyond running existing benchmarks.
• Ability to communicate research findings clearly and collaborate closely with a small engineering team.
• Availability to work onsite in Brooklyn five days per week.

Preferred Qualifications

• Master’s degree or PhD in computer science, mathematics, statistics, or a related quantitative field.
• Experience building simulated environments or synthetic users to evaluate or train LLMs.
• Research or engineering experience involving LLM memory, identity, or personalization.
• Familiarity with data pipeline and configuration tools such as Hydra, Kafka, and SQL.
• Meaningful contributions to open-source AI research or benchmarks.

What Success Looks Like

You bring research depth and the engineering ability to turn novel evaluation methods into reliable systems. Your experience extends beyond standard benchmark execution, RAG evaluation, or applied agent testing to include original evaluation design and research relevant to RL post-training.

Compensation and Benefits

• Annual base salary of $220,000–$300,000, plus equity.
• Fully covered premiums for medical, dental, and vision insurance.
• Starter 401(k) plan.
• $5,000 annual lifestyle stipend.
• Unlimited paid time off.
• Office located in a building with an Equinox fitness club.
• $5,000 relocation stipend.

Why Join

Help address foundational research challenges in AI memory and personalization. Work directly with the CTO in a flat team structure. Own consequential evaluation work from research through implementation. Contribute to visible, citable open-source work. Join a fast-growing AI company with a biweekly release cadence.

Application Materials

Please submit your resume or CV and links to relevant publications, open-source research, or evaluation projects. Highlight your contributions to original evaluation design, RL environments, simulators, or LLM-as-judge systems. By submitting your application, you authorize Lovefreedom Solution Workforce to represent your candidacy and present your qualifications to its client for the Machine Learning Engineer – Evals position, subject to disclosure of the client’s identity and verification that no prior application or representation exists for this specific opportunity.