When Should an AI Company Build Several Models?

In 2023, the obvious way to build our live speech-to-speech translation platform, Palabra.ai, was to buy access to existing speech recognition, translation, and voice generation models. Mature providers already offered all three. We chose to build all three models ourselves, accepting the cost of separate datasets, research, evaluation, and inference infrastructure.

We made that choice because the live translation experience depended on three connected timing decisions:

• Speech recognition decides when partial speech is stable enough to release.
• Translation decides whether to act on incomplete meaning.
• The voice model decides when it has enough context to begin speaking naturally.

Because each decision changes what the next model receives and when, optimizing one model for its own benchmark can make the complete conversation slower or less natural.

By mid-2026, Palabra had crossed one million minutes of live translation, while annual recurring revenue had grown from $60,000 to more than $1 million in six months. For us, those were strong signals that the decision was paying off.

That experience raised the question this article examines: when can AI models be evaluated and sourced separately, and when do several models need to be built and improved together?

Separate sourcing works only when models are interchangeable

The standard approach is to make a separate build-versus-buy decision for each model. That approach works when replacing one model does not require changes to the models around it.

Consider a voice agent that combines speech recognition, a language model, and voice synthesis. If the team can replace its speech-recognition provider without changing what the language model receives or when the voice model responds, speech recognition remains an independent sourcing decision.

That independence should be tested rather than assumed. A replacement may divide speech differently, release text later, or provide less context. Those changes affect what the next model receives and when it can act.

When replacing one model changes the performance of adjacent models, the company is no longer choosing among independent components. It is managing a connected system that may need to be optimized and owned together.

Four signs that several models may need joint control

Before deciding what to build, determine whether the product’s performance genuinely depends on several connected models. The following signals suggest that evaluating and sourcing each model separately may hide the source of the product’s advantage.

These signals are not requirements that all need to apply. One may justify investigating the connections among models, while several appearing together provide stronger evidence that joint optimization is necessary.

Your key metric depends on several technical layers

Component benchmarks matter, but they may not explain the product-level result you are trying to improve. Teams can track word error rate, translation quality, and time to first audio separately, yet none of those metrics captures conversational flow on its own.

To date, Coval’s independent benchmark ranks Palabra TTS v1 first, with a time to first audio of approximately 105 milliseconds. The next systems take about 113 and 155 milliseconds.

Note: because Coval’s benchmark is live, rankings may change as new results come in and can vary with model versions, infrastructure, and test conditions.

That result depended on more than fast TTS inference. Our architecture reaches about 35 milliseconds before network overhead, starts from partial text, and stops when interrupted. TTS performance therefore depends partly on how other layers supply and manage its input.

The leaderboard result cannot establish that proprietary TTS will always outperform external services. It does show that a product-level result may depend on more than one model. When the metric that matters depends on several connected layers, teams need to evaluate and optimize those layers together.

A better component can make the product worse

This is less obvious and gives leadership a practical warning sign. A model can score better on its own benchmark yet perform worse once inserted into the product because its outputs, timing, or behavior do not fit adjacent layers.

For example, a more accurate speech-recognition model might release partial results more slowly. Translation then receives better text, but later, increasing end-to-end latency. Similarly, a higher-quality translation model might produce output in larger chunks that delay speech synthesis.

The signal becomes: if replacing one component with a better-performing alternative does not reliably improve the product, your advantage may depend on how several layers work together.

Handoffs can discard information the next layer needs

Text is a narrow representation of speech. Depending on the architecture, converting audio to text can discard timing, hesitation, pace, emphasis, and emotional cues.

An external pipeline can preserve some of that information through metadata or additional models. The decision turns on whether providers expose the signals the next component needs. If they do, orchestration may solve the problem. If they do not, ownership becomes more defensible.

External providers may not provide the controls you need

Managed APIs expose settings for common use cases. A specialized product may need control over segmentation, confidence thresholds, partial-result policies, glossary behavior, output queues, model size, or deployment location.

Needing more controls does not immediately justify training a model. Self-hosting, fine-tuning, or a better orchestration layer may provide enough flexibility. Consider joint model ownership only after those less expensive options fail against a measured requirement.

Use the four signs to decide which models need joint control

The four signs reveal where separate sourcing may limit the product: several models determine the target metric, stronger individual components fail to improve the complete system, handoffs lose required information, or external providers withhold necessary controls.

Trace each sign to the models responsible for the constraint. Include a model in the ownership scope when its output, timing, or behavior needs coordinated changes with adjacent models. Stop where a model can be replaced independently without affecting the target product outcome.

At Palabra, the signs pointed to speech recognition, translation, and voice synthesis. All three affected latency and conversational flow, their handoffs determined what information reached the next model, and external services did not provide the required control over those interactions.

Other speech systems may require fewer proprietary models. A peer-reviewed ACL 2025 study reused Whisper and Llama 3.1 while building a custom streaming speech model. The researchers achieved their latency target without modifying every model in the pipeline.

The ownership decision should cover the smallest connected set of models needed to remove the measured product constraint.

Use six questions to decide whether ownership can earn its cost

Before building, leadership teams should answer six questions.

• Is AI behavior central to the product’s value? If a model supports a standard internal workflow, a mature service will often outperform an internal effort on time, cost, and reliability.
• Does the differentiation come from one model, the application workflow, or interactions among models, data pipelines, orchestration, and infrastructure? Keep the ownership scope narrow when one component creates the advantage. Expand it only when evidence shows that several components constrain the same product outcome.
• Can external providers expose the required information and controls? Better interfaces, streaming architecture, or observability may solve the problem without model ownership.
• Can the company measure the product advantage it expects to gain by owning these components rather than buying them? Define latency percentiles, stable-output time, critical-error rates, cost per transaction, or another product-level result. “Greater control” has little value until it changes something measurable.
• Can the company build and sustain ownership? Capability matters alongside technical need. Before founding Palabra, I had spent years in machine learning research and had grown an ML team from one person to 60. That experience made the broader ownership scope feasible, although it did not remove the cost or risk. After launch, data maintenance, evaluation, retraining, inference optimization, monitoring, security, and specialist hiring continue.
• Can the internal architecture remain modular? External models will improve, so the team should preserve its ability to compare and replace each internal component. Clear interfaces, component-level tests, shadow deployments, and fallback paths allow the team to replace a component when its advantage no longer justifies its cost.

Own the smallest connected set of models that creates the advantage

Begin with a measurable product requirement. Test whether external models can meet that requirement independently and whether orchestration, self-hosting, fine-tuning, or changing providers can close any remaining gap. The evidence may come from prototypes, controlled evaluations, or production data.

Consider building several models when those tests show that their outputs, timing, or behavior need to change together. The expected improvement should also justify the continuing costs of data collection, evaluation, retraining, infrastructure, and specialist talent.

At Palabra, latency and conversational flow depended on speech recognition, translation, and voice synthesis. That dependence made all three models part of the same ownership decision. Another speech product may reach its target by building only one model or by keeping the entire model stack external.