AI Operations Specialist – LLM, Prompt Engineering & Conversation Systems

Position Type: Full-Time, Remote

Working Hours: U.S. Business Hours

Industry: Artificial Intelligence / SaaS / Telehealth Technology

Department: AI Operations / Conversational AI

Focus: Production LLM Systems, AI Conversation Optimization & Conversion Performance

About the Role

At Pavago, one of our clients is hiring a highly skilled AI Operations Specialist to manage, monitor, troubleshoot, and continuously optimize production AI conversation systems. Our client operates an AI-powered SMS lead recovery platform serving telehealth brands, processing millions of messages every month. These AI systems communicate directly with customers, making conversation quality, reliability, and conversion performance critical to business success.

This is a hands-on, execution-focused role for someone with proven experience operating Large Language Model (LLM) systems in production, rather than simply experimenting with AI tools. You’ll be responsible for onboarding new clients, configuring AI conversation workflows, managing knowledge bases and integrations, monitoring live customer interactions, identifying AI failure modes, optimizing prompts, and measuring the impact of changes on conversion performance.

The ideal candidate combines technical AI knowledge, prompt engineering expertise, analytical thinking, and operational ownership with the ability to work independently in a fast-moving environment. You’ll collaborate directly with the founder to ensure AI systems consistently deliver accurate, effective, and conversion-focused customer conversations.

What You’ll Own

Client Onboarding & AI System Configuration

• Onboard new clients onto the AI-powered conversation platform.
• Configure client-specific AI conversation systems and workflows.
• Set up and troubleshoot webhook integrations, APIs, and knowledge bases.
• Configure SMS campaigns and conversation workflows.
• Ensure AI systems are aligned with client requirements and objectives.
• Test conversation flows and integrations before deployment.
• Identify and resolve configuration issues during onboarding.
• Verify that new client systems are functioning correctly before going live.
• Maintain accurate documentation of client configurations and setup processes.

Production AI Operations & Monitoring

• Monitor live AI-generated conversations and customer interactions.
• Review conversation transcripts to identify errors, inconsistencies, and missed conversion opportunities.
• Diagnose AI behavior issues and determine their underlying causes.
• Identify patterns affecting conversation quality, accuracy, and performance.
• Detect unexpected AI responses, workflow failures, and system inconsistencies.
• Investigate issues involving conversation logic, prompts, knowledge bases, or integrations.
• Escalate technical issues when necessary and follow through until resolution.
• Maintain consistent quality standards across customer-facing AI systems.
• Proactively identify opportunities to improve production reliability and conversation effectiveness.

Prompt Engineering & LLM Optimization

• Write, edit, and optimize system prompts and customer-facing AI messaging instructions.
• Improve AI conversation logic based on real customer interactions.
• Identify weaknesses in existing prompts and system behavior.
• Adjust instructions to improve response quality, accuracy, and consistency.
• Test prompt changes systematically before deployment.
• Evaluate AI behavior before and after implementing changes.
• Document prompt updates, testing results, and system modifications.
• Continuously refine AI-generated responses to improve customer engagement.
• Ensure prompt changes support both conversation quality and conversion objectives.

AI Conversation Analysis & Quality Assurance

• Review large volumes of AI-generated conversation transcripts.
• Identify recurring response errors and behavioral patterns.
• Analyze missed opportunities within customer conversations.
• Evaluate whether AI responses appropriately address customer needs.
• Investigate inconsistencies between expected and actual AI behavior.
• Identify opportunities to improve conversation flows and response handling.
• Conduct structured testing of prompts and conversation workflows.
• Maintain quality standards across deployed AI systems.

Conversion Performance & Data Analysis

• Analyze conversation performance and customer interaction data.
• Measure the impact of AI system changes on lead conversion and engagement.
• Identify patterns that influence customer responses and conversion outcomes.
• Track performance before and after prompt or workflow adjustments.
• Evaluate whether changes deliver measurable improvements.
• Use conversation data to prioritize optimization opportunities.
• Communicate performance findings and recommendations to leadership.
• Support data-driven decisions regarding AI conversation strategies.

Founder Collaboration & Operational Ownership

• Work directly with the founder on daily AI operations and improvement initiatives.
• Provide clear updates on system performance, issues, and optimization opportunities.
• Identify operational problems and take ownership of resolving them.
• Recommend improvements based on production data and customer interactions.
• Prioritize urgent issues affecting customer-facing AI systems.
• Coordinate technical escalations when necessary.
• Document findings, system changes, and testing outcomes.
• Move quickly while maintaining high standards for AI reliability and accuracy.

Requirements

Must-Have Qualifications

• Proven experience operating LLM or AI systems in production — essential.
• Hands-on experience monitoring, troubleshooting, and improving live AI-generated outputs.
• Strong understanding of Large Language Models (LLMs) and their behavior in real-world applications.
• Practical experience with prompt engineering and system prompt optimization.
• Experience identifying and resolving common LLM failure modes.
• Understanding of iterative testing and AI system improvement.
• Experience reviewing and analyzing AI-generated conversations or other customer-facing AI outputs.
• Ability to work with webhooks, APIs, knowledge bases, or similar integrations.
• Strong analytical and problem-solving abilities.
• Ability to evaluate AI performance using real-world interaction data.
• Excellent written English for reviewing and improving customer-facing AI messages.
• Strong documentation and communication skills.
• Exceptional attention to detail.
• Ability to work independently and take ownership of production systems.
• Comfortable working directly with a founder in a fast-paced environment.
• Availability to work during U.S. Business Hours.

Important Experience Requirement

This role requires real-world production AI experience. Candidates should have hands-on experience managing or improving deployed AI systems that interact with actual users or customers. Experience limited to experimenting with ChatGPT, creating basic prompts, or exploring AI tools without production responsibilities will not meet the requirements of this position.

Preferred Qualifications

• Experience managing conversational AI systems or AI-powered messaging platforms.
• Experience optimizing LLM-generated responses for customer engagement or conversion.
• Familiarity with SMS-based customer communication workflows.
• Experience working with AI-powered lead generation or lead recovery systems.
• Experience supporting SaaS platforms or customer-facing AI products.
• Familiarity with telehealth or healthcare-related customer communication environments.
• Experience conducting structured prompt evaluations and regression testing.
• Experience analyzing large volumes of conversation transcripts.
• Understanding of conversion funnels and customer engagement metrics.
• Experience configuring client-specific AI knowledge bases and workflows.
• Experience working directly with founders or small technical teams.

Tools & Technical Skills

Large Language Models & Prompt Engineering

• LLM-based applications
• System prompt development
• Prompt optimization
• Conversation logic
• AI response evaluation
• Iterative prompt testing
• LLM failure analysis
• Production AI troubleshooting

AI Integrations & Configuration

• Webhooks
• APIs
• Knowledge bases
• AI conversation workflows
• Client-specific system configuration
• Campaign setup
• Integration troubleshooting

Conversation Monitoring & Performance

• Conversation transcript analysis
• AI quality assurance
• Customer interaction analysis
• Conversion tracking
• Performance measurement
• Root cause analysis
• System change documentation

Core Technical Focus

Production LLM Operations | Prompt Engineering | Conversational AI | API & Webhook Integrations | AI Quality Assurance | Conversion Optimization

What Makes You a Strong Fit

You’ll likely thrive in this role if you:

• Have actually operated LLM-powered systems in production, not just experimented with AI tools.
• Understand how AI systems behave when interacting with real customers.
• Can review hundreds or thousands of AI conversations to identify meaningful patterns.
• Know how to investigate unexpected AI responses and determine their root causes.
• Enjoy improving prompts and conversation workflows through structured testing.
• Evaluate AI performance based on measurable business outcomes.
• Understand that a well-written AI response is only valuable if it helps achieve the intended objective.
• Can identify issues affecting customer engagement and conversion.
• Are methodical about testing changes before and after deployment.
• Document system changes clearly and consistently.
• Communicate technical findings in straightforward language.
• Take ownership of problems from identification through resolution.
• Are comfortable managing multiple operational priorities.
• Enjoy collaborating directly with a founder.
• Can move quickly without compromising quality or reliability.

What a Typical Day Looks Like

Your day begins by reviewing the performance of live AI conversations, checking for unusual patterns, system failures, or opportunities to improve customer engagement. You may investigate conversation transcripts to understand why certain leads failed to convert, identify problematic AI responses, or determine whether an issue originates from a prompt, knowledge base, or integration. Later, you might onboard a new telehealth client by configuring their knowledge base, setting up webhooks, and testing AI conversation workflows before launch. Throughout the day, you’ll update system prompts, refine conversation logic, and conduct structured tests to evaluate whether your changes improve response quality and conversion performance. You’ll also analyze performance data, document your findings, and communicate recommendations directly to the founder. Before wrapping up, you’ll review the results of recent changes, monitor any unresolved issues, and prioritize the next set of improvements. In short: You’ll operate, troubleshoot, test, and optimize production AI conversation systems to improve reliability, customer engagement, and lead conversion.

Key Metrics for Success

Production AI Performance

• AI conversation quality and consistency
• Frequency of recurring AI errors
• Identification and resolution of production issues
• Reliability of deployed conversation workflows
• Accuracy of customer-facing AI responses

Prompt & System Optimization

• Effectiveness of prompt improvements
• Quality of system instructions
• Reduction in repeated AI failure patterns
• Accuracy of conversation logic
• Testing and documentation completeness
• Measurable improvement following system changes

Conversion & Business Outcomes

• Lead recovery performance
• Conversation-to-conversion rates
• Customer engagement rates
• Reduction in missed conversion opportunities
• Performance improvements following prompt updates
• Ability to identify and act on conversion trends

Client Onboarding & Operations

• Accuracy of client configurations
• Successful webhook and API integrations
• Completion of pre-launch testing
• Resolution of onboarding issues
• Quality of knowledge base configurations
• Timely completion of operational priorities

Ownership & Collaboration

• Responsiveness to production issues
• Quality of root cause analysis
• Clarity of documentation
• Effectiveness of communication with leadership
• Ability to independently resolve problems
• Consistency in monitoring and improving AI systems

Why This Role Stands Out

This opportunity offers the chance to work directly with the founder of an AI-powered SaaS platform processing millions of customer messages each month. Unlike traditional AI research or experimentation roles, this position focuses on the practical challenges of managing and improving AI systems that are already operating at scale. You’ll have the opportunity to:

• Take ownership of production AI conversation systems.
• Work with real customer interactions and large-scale conversation data.
• Improve LLM prompts and system behavior.
• Configure client-specific AI workflows and integrations.
• Identify and resolve complex production issues.
• Measure the business impact of AI optimization.
• Collaborate directly with the founder on operational priorities.
• Contribute to the reliability and effectiveness of customer-facing AI technology.

Working Hours & Availability

Position Type: Full-Time

Location: Remote

Working Hours: U.S. Business Hours

Candidates must be comfortable collaborating with the founder during U.S. business hours and responding to operational priorities involving production AI systems. Strong communication, independent problem-solving, and consistent availability are essential.

Interview Process

• Initial Application
• Spark Hire One-Way Video Interview
• Initial Recruiter Screening
• Client Interview
• Offer Stage
• Spark Hire Video Interview – Required

As part of the application process, all candidates are required to complete a one-way video interview through Spark Hire. After completing the first step of your application, you’ll receive a Spark Hire invitation by email with instructions to record and submit your video responses. Completion of the Spark Hire video is required to be considered for the next stage. Please check your inbox as well as your spam or junk folder for the invitation.

Apply Now

If you’re an experienced AI Operations Specialist, LLM Operations Specialist, Conversational AI Specialist, Prompt Engineer, or AI Systems Operations Professional with proven experience managing production AI systems, we’d love to hear from you. This is an opportunity to take ownership of real-world AI conversation systems, improve LLM performance, troubleshoot complex production issues, and directly influence customer engagement and conversion outcomes. Candidates with hands-on production LLM experience, strong prompt engineering skills, and the ability to independently optimize customer-facing AI systems are encouraged to apply.