AI Product & Evals Analyst (Voice Agents)

Company: Bluemachines.ai

Location: Bangalore / 5 days WFO

Experience: 3–4 years

Function: Analytics

We’re hiring an AI AI Product & Evals Analyst Analyst to own evaluation, quality measurement, dashboarding, and enterprise analytics for Bluemachines’ AI voice agents.

You will work at the intersection of AI Engineering, Product, and Customer Delivery to define how voice agent quality is measured, monitored, and improved at scale. This includes building evaluation frameworks, detecting regressions, analyzing business impact, and enabling enterprise teams with clear, actionable insights.

Beyond evaluation, we’re looking for someone with an AI-first mindset who can leverage modern AI tools and automation workflows to improve analytics, reporting, and operational efficiency. The role will also contribute toward building reusable analytics frameworks, internal intelligence tooling, and self-serve analytics capabilities for enterprise clients.

Your work will directly influence launch quality, customer trust, operational transparency, and how enterprise AI systems are measured and scaled.

What will you do?

Business & Enterprise Analytics

Track and analyze business KPIs tied to voice agent deployments (automation, containment, conversion, CSAT, cost savings, etc.)

Support enterprise billing, audit, and reconciliation workflows through reliable usage and conversation analytics

Translate AI behavior into clear operational and business impact insights for enterprise clients

AI-First Analytics & Automation

Leverage AI tools and agentic workflows to improve speed, quality, and scalability of analytics, reporting, and RCA workflows

Automate repetitive analytics and reporting tasks using LLMs, copilots, and workflow tooling

Drive AI-native ways of working across SQL, Python, dashboards, and insight generation

Analytics Platform & Internal Tooling

Build reusable analytics frameworks, standardized dashboards, and internal tooling for enterprise deployments

Contribute to self-serve analytics and intelligence capabilities such as text-to-data, AI-powered insights, and conversational analytics interfaces

Help evolve Bluemachines’ analytics and observability layer for enterprise AI voice agents

Voice Agent Evaluation

Design and own AI eval frameworks specifically for voice agents

Define conversation- and turn-level metrics

Build and maintain golden voice datasets across accents, languages, and noise conditions

What We’re Looking For

2–5 years of experience in Product Analytics, Data Analytics, AI Quality, or related analytical roles

Strong analytical thinking, metric design, and problem-solving skills

Hands-on proficiency in SQL and Python for analysis, evaluation, and data investigation

Ability to reason deeply about conversational, voice, and workflow-driven systems

Experience building dashboards and enterprise-grade visualizations using tools like Tableau, Power BI, Looker, or equivalent

Strong communication and storytelling skills with the ability to translate AI/system behavior into business insights

Comfort working cross-functionally with AI Engineering, Product, Operations, and customer-facing teams

AI-first mindset with strong adoption of modern AI tools, copilots, and automation workflows to improve productivity and scale analytics

Good-to-Have

Experience with voice systems such as STT, TTS, call analytics, or conversational AI platforms

Exposure to LLM-based products, prompt engineering, AI evals, or agentic workflows

Experience with enterprise analytics, client-facing reporting, billing/reconciliation, or operational analytics

Interest in building internal analytics tooling, reusable frameworks, or self-serve intelligence systems

Experience working in fast-moving AI or startup environments