TEFormer
Paper Code

TEFormer

Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers

ICML 2026

Sicheng Shen, Mingyang Lv, Bing Han, Dongcheng Zhao, Guobin Shen, Feifei Zhao, Yi Zeng

The first bidirectional temporal enhancement framework for Spiking Transformers.

TEFormer couples parallel forward temporal fusion in attention with gated backward temporal tracing in the MLP, yielding a structured brain-inspired path for stronger spatiotemporal representations.

TEFormer paper overview figure showing the forward TEA stream and backward T-MLP stream
Figure 1. TEFormer pipeline with forward TEA and backward T-MLP temporal fusion.
TEFormer pipeline with forward TEA stream and backward T-MLP stream TEFormer Block x L Input Spikes Patch Embedding ... Output Spikes Prediction Head Class t t Forward Temporal Stream (TEA) parallel forward fusion Temporal Enhancement Attention Backward Temporal Stream (T-MLP) gated backward tracing Temporal MLP Enhance Feedback TEA forward T-MLP backward

Method

Temporal Enhancement Attention

TEA applies a learnable exponential temporal mask to the value pathway, enabling synchronized forward fusion with a single scalar.

  • Parallel forward temporal fusion
  • Hyperparameter-free temporal decay
  • Preserves efficient QK attention
alpha = 0.5 + 0.5 sigma(theta) M maps historical time steps V enhanced = M V

Temporal MLP

T-MLP replaces conventional upsampling with a gated reverse-time update that carries future context backward through the sequence.

  • Gated backward temporal tracing
  • Single-gate recurrent structure
  • Bidirectional temporal consistency
h[T-1] = LIF(W_in X[T-1]) h[r] blends h[r+1] and X[r] Y[t] = LIF(BN(W_o h[t]))

Results

CIFAR10 CIFAR100 CIFAR10-DVS N-CALTECH101 NCARS SHD UCF101-DVS
96.24 79.84 81.90 78.50 95.95 90.19 63.16

Top-1 Accuracy (%)

Encoding

Direct

96.24

Phase

89.92

Rate

84.74

TTFS

87.46
Foundation

Implemented as the official TEFormer codebase, built on STEP and BrainCog for unified Spiking Transformer evaluation.

Scope

Evaluated on CIFAR10/100, SVHN, CIFAR10-DVS, N-CALTECH101, NCARS, SHD, HMDB51-DVS, UCF101-DVS, sCIFAR, and sMNIST.

Efficiency

Temporal enhancement is concentrated in shallow layers, where low-level spatiotemporal patterns benefit most from fusion.

Citation

@inproceedings{shen2026teformer,
  title={TEFormer: Structured Bidirectional Temporal Enhancement Modeling in Spiking Transformers},
  author={Shen, Sicheng and Lv, Mingyang and Han, Bing and Zhao, Dongcheng and Shen, Guobin and Zhao, Feifei and Zeng, Yi},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  year={2026}
}
BibTeX copied