---
title: Transformer Circuit Faithfulness Metrics are not Robust
url: https://www.ml-quant.com/papers/arxiv/2407.08734/
site: ML-Quant (https://www.ml-quant.com)
updated: 2026-09-26
license: Summaries CC BY 4.0; links go to the original sources
index: https://www.ml-quant.com/llms.txt
identifier: arXiv:2407.08734
source_url: https://arxiv.org/abs/2407.08734
featured: 2024-07-17
citations: 19
topic: ML & AI Methods
---


# Transformer Circuit Faithfulness Metrics are not Robust

The authors explore the difficulties in evaluating the performance of neural network 'circuits', emphasizing the sensitivity of current methods to changes in the ablation methodology and the need for clearer claims about circuits.

- Source: https://arxiv.org/abs/2407.08734
- Identifier: arXiv:2407.08734
- Released: 2024-07-11
- First featured: Quant Letter No. 57 (2024-07-17): https://www.ml-quant.com/issues/2024-07-17/
- Citations (Semantic Scholar): 19
- Published in: not yet
- Topic: ML & AI Methods

## Related

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://www.ml-quant.com/papers/arxiv/2312.00752/): Sequence Modeling: Mamba, a neural network architecture that doesn't use attention or MLP blocks, provides faster inference and better performance in language, audio, and genomics than Transformers.
- [Transaction Fraud Detection via Spatial-Temporal-Aware Graph Transformer](https://www.ml-quant.com/papers/arxiv/2307.05121/): A new graph neural network, STA-GT, is proposed for transaction fraud detection, effectively learning and incorporating spatial-temporal and global information.
- [MLBased CT Saturation Detection](https://www.ml-quant.com/papers/ssrn/4979169/): The study introduces a machine learning method for detecting saturation in Current Transformers using artificial neural networks and long short-term memory networks, enhancing anomaly detection in complex datasets.
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://www.ml-quant.com/papers/arxiv/2405.21060/): The research identifies a link between state-space models and Transformers in deep learning, leading to the creation of a faster language modeling architecture, Mamba-2.
- [Octo: An Open-Source Generalist Robot Policy](https://www.ml-quant.com/papers/arxiv/2405.12213/): Octo is a large transformer-based policy for robotic manipulation, trained on a vast dataset, that can be instructed via language or images and adapted to new domains.
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://www.ml-quant.com/papers/arxiv/2404.02905/): The article discusses Visual AutoRegressive modeling (VAR), a new image learning method that outperforms diffusion transformers in terms of speed, image quality, and scalability.