Tech Meridian ← LIVE FEED
PROMY MERIDIAN RU

RESEARCH · RESEARCH · #3586

Together AI publishes Together Compiler (TKCC) for continual AI inference optimization

Together AI announces Together Compiler (TKCC), a continual agent-driven compilation system for AI inference that uses a Tiered Persistent Engine and a cost model to balance measurement throughput and fidelity. The authors report TKCC outperforms hand-written Megakernels on Llama 1B by up to 1.7x and the best open-source engines on several recent frontier models by up to 2.6x TPS, and say TKCC is deployed in production serving trillions of tokens.

KEY POINTS

  1. Together AI announces Together Compiler (TKCC), a continual agent-driven compilation system for AI inference that uses a Tiered Persistent Engine and a cost model to balance measurement throughput and fidelity.
  2. The authors report TKCC outperforms hand-written Megakernels on Llama 1B by up to 1.7x and the best open-source engines on several recent frontier models by up to 2.6x TPS, and say TKCC is deployed in production serving trillions of tokens.
  3. If validated, continual compilation that reduces engine-launch overheads and improves throughput across models could materially lower latency and cost for inference at scale.

WHY IT MATTERS

If validated, continual compilation that reduces engine-launch overheads and improves throughput across models could materially lower latency and cost for inference at scale.

SOURCES & TIMELINE

1