NEWS · MODELS · #310
Google describes private on-device LM training for Gboard using federated learning and differential privacy
Google researchers report that all next-word-prediction neural LMs in Gboard are now trained with federated learning and formal differential privacy guarantees, with over 30 on-device models across 7+ languages and 15+ countries. The deployed models satisfy (ε,δ)-DP with δ=1e-10 and ε ranging from 0.994 to 13.69; the post describes practical techniques used (DP-FTRL, secure aggregation, adaptive clipping, pretraining on C4) and states this is, to their knowledge, the largest production deployment of user-level DP and the first announced ε<1 for models trained directly on user data.
KEY POINTS
- Google researchers report that all next-word-prediction neural LMs in Gboard are now trained with federated learning and formal differential privacy guarantees, with over 30 on-device models across 7+ languages and 15+ countries.
- The deployed models satisfy (ε,δ)-DP with δ=1e-10 and ε ranging from 0.994 to 13.69; the post describes practical techniques used (DP-FTRL, secure aggregation, adaptive clipping, pretraining on C4) and states this is, to their knowledge, the largest production deployment of user-level DP and the first announced ε<1 for models trained directly on user data.
- This matters because it demonstrates production-scale deployment of federated learning with formal user-level differential privacy (including ε<1) for on-device language models, advancing practical privacy protections for user data.
WHY IT MATTERS
This matters because it demonstrates production-scale deployment of federated learning with formal user-level differential privacy (including ε<1) for on-device language models, advancing practical privacy protections for user data.