NEWS · MODELS · #280
Exphormer: using expander graphs to scale graph transformers
Exphormer is a sparse-attention framework for graph transformers proposed by Google Research (presented at ICML 2023) that replaces the dense full-attention interaction graph with a sparse d-regular expander graph (using a randomized Friedman near-Ramanujan construction). The approach combines expander edges with the input graph and virtual nodes to achieve linear edges and reports strong empirical results across diverse graph datasets; an implementation is available on GitHub.
KEY POINTS
- Exphormer is a sparse-attention framework for graph transformers proposed by Google Research (presented at ICML 2023) that replaces the dense full-attention interaction graph with a sparse d-regular expander graph (using a randomized Friedman near-Ramanujan construction).
- The approach combines expander edges with the input graph and virtual nodes to achieve linear edges and reports strong empirical results across diverse graph datasets; an implementation is available on GitHub.
- Reducing the quadratic cost of full attention on graphs by using expander-based sparse interaction graphs makes transformer architectures practical for much larger graph-structured datasets.
WHY IT MATTERS
Reducing the quadratic cost of full attention on graphs by using expander-based sparse interaction graphs makes transformer architectures practical for much larger graph-structured datasets.