RELEASE · CODING · #3038
Google launches LiteRT Tensor API for building and optimizing on-device ML in C++
Google introduced the LiteRT Tensor API, a lightweight C++ API that lets developers build computation graphs operation-by-operation for on-device execution across CPU, GPU, and NPU backends, with features like direct weight ingestion (including .safetensors), zero-copy buffer interoperability, and compatibility with AI coding agents. Google demonstrated the API with a SAM 2 multi-object segmentation and tracking app and reported cross-platform benchmarks (e.g., about 1.5x faster than Apple MLX on a MacBook Pro in one test) and listed planned enhancements such as NPU-friendly ops and automatic full-graph quantization.
KEY POINTS
- Google introduced the LiteRT Tensor API, a lightweight C++ API that lets developers build computation graphs operation-by-operation for on-device execution across CPU, GPU, and NPU backends, with features like direct weight ingestion (including .safetensors), zero-copy buffer interoperability, and compatibility with AI coding agents.
- Google demonstrated the API with a SAM 2 multi-object segmentation and tracking app and reported cross-platform benchmarks (e.g., about 1.5x faster than Apple MLX on a MacBook Pro in one test) and listed planned enhancements such as NPU-friendly ops and automatic full-graph quantization.
- The API gives developers low-level, portable C++ graph control and performance features (direct weights, zero-copy buffers, agent compatibility) that can reduce platform-specific rewrites and speed up on-device ML development and execution.
WHY IT MATTERS
The API gives developers low-level, portable C++ graph control and performance features (direct weights, zero-copy buffers, agent compatibility) that can reduce platform-specific rewrites and speed up on-device ML development and execution.