AI Evaluation Engineer

As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside. You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive edge cases) as well as classic product/QA issues — so that what we ship to customers handling real securities and compliance data is trustworthy every time. This is a hands-on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.

• Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (e.g., document extraction, compliance checks, automated workflows), covering accuracy, consistency, hallucination rate, and safety.
• Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively.
• Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time.
• Extend evaluation coverage beyond the model layer into full product/QA testing — functional, regression, and end-to-end testing of AI-powered features and the surrounding product.
• Partner with product managers and engineers to translate ambiguous quality bars ("is this good enough to ship?") into measurable, repeatable evaluation criteria.
• Investigate failures and edge cases, perform root-cause analysis across the model/product boundary, and drive fixes with engineering.
• Maintain traceability and reporting on eval/QA results for compliance-sensitive workflows, given the regulated nature of the data we handle.

• 5 plus years of experience in QA/test engineering, ML evaluation, or a related quality-focused engineering role.
• Hands-on experience testing or evaluating AI/LLM-powered features — building eval sets, scoring rubrics, or benchmark harnesses (or strong adjacent automation/QA experience with a demonstrated interest in AI evaluation).
• Solid automation/testing fundamentals: scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration.
• Comfort working with LLMs and AI tooling directly — prompt engineering, RAG pipelines, or AI-assisted development tools — either as a builder or a rigorous evaluator of them.
• Strong analytical mindset: comfortable defining metrics for fuzzy, subjective quality questions and defending them with data.
• Clear written communication — you'll be documenting failure modes and quality bars for both engineers and non-technical stakeholders.
• Bonus: experience in fintech, compliance, or another regulated domain where correctness and auditability matter.

• Experience with vector databases / semantic search (FAISS, ChromaDB, or similar) and RAG evaluation.
• Familiarity with eval tooling/frameworks (e.g., Ragas, DeepEval, promptfoo, custom LLM-as-judge pipelines).
• Background in Agile/Scrum environments and defect-management tooling (Jira or similar).

• Competitive compensation based on experience
• Equity participation
• Comprehensive health insurance (self + family)
• Paid leave and wellness benefits

To apply: send your résumé to careers@vinylequity.com.