LLM Ops Engineer (Raleigh) - HirexHire - Anywhere
HirexHire
About the position
We are seeking an experienced LLM Engineer to join our client's newly established LLM Ops Team in their Raleigh, NC office. In this role, you will be responsible for managing the complex lifecycle of Large Language Models from development to deployment, monitoring, and continuous improvement.\n\nThis role is hybrid to the Raleigh, NC area.
Responsibilities
Fine-tune pre-trained models for specific use casesCurate and prepare datasets for trainingManage training infrastructure, resources, and computational environmentsImplement optimization techniques to improve model performanceDevelop and manage APIs for model servingScale infrastructure to handle varying demand loadsBuild and maintain the GenAI middleware/sidecar layerIntegrate LLMs with existing systems and data sourcesTrack performance metrics including latency and throughputMonitor quality metrics such as hallucination rates and accuracyOptimize costs associated with model inference and trainingCreate and maintain dashboards for real-time performance insightsCreate and maintain golden datasets for benchmark testingImplement statistical validation methods for model outputsSet up similarity matching criteria for response evaluationDevelop confidence score thresholds for production systemsDesign and implement user feedback collection systemsEstablish continuous improvement processesCreate A/B testing frameworks for model and feature evaluationConduct trace analysis to identify areas for performance optimizationImplement content moderation systemsDetect and mitigate bias in model outputsEnsure regulatory compliance in AI systemsDevelop output validation frameworksVersion and store prompts systematicallyCreate and maintain prompt templatesSet up playground environments for prompt testingAbstract prompts from application code for better maintainabilityRequirements
Experience with LLM development, fine-tuning, and deploymentStrong programming skills, particularly in PythonExperience with Kubeflo