Skip to content

Commit bffff65

Browse files
fel-thomasfel-thomas
committed
docs: introduce jekyll doc
1 parent 67d4fa8 commit bffff65

3 files changed

Lines changed: 171 additions & 1 deletion

File tree

‎.github/workflows/docs.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: docs
2+
3+
on:
4+
push:
5+
branches: ["main"]
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: read
10+
pages: write
11+
id-token: write
12+
13+
concurrency:
14+
group: "pages"
15+
cancel-in-progress: false
16+
17+
jobs:
18+
build:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- uses: actions/checkout@v4
22+
- uses: actions/configure-pages@v5
23+
- uses: actions/jekyll-build-pages@v1
24+
with:
25+
source: ./docs
26+
destination: ./_site
27+
- uses: actions/upload-pages-artifact@v3
28+
with:
29+
path: ./_site
30+
31+
deploy:
32+
runs-on: ubuntu-latest
33+
needs: build
34+
environment:
35+
name: github-pages
36+
url: ${{ steps.deployment.outputs.page_url }}
37+
steps:
38+
- id: deployment
39+
uses: actions/deploy-pages@v4

‎README.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66

77
[![tests](https://github.com/KempnerInstitute/raptor/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/KempnerInstitute/raptor/actions/workflows/tests.yml)
88
[![arXiv](https://img.shields.io/badge/arXiv-1234.56789-b31b1b.svg)](https://arxiv.org/abs/1234.56789)
9-
[≫ Raptor Preprint](https://arxiv.org/abs/2510.08638)
109

1110
[**Mozes Jacobs**](https://scholar.google.com/citations?user=8Dm0KfQAAAAJ&hl=en)$^{\star1}$   [**Thomas Fel**](https://thomasfel.me)$^{\star1}$   [**Richard Hakim**](https://richhakim.com/)$^{\star1}$
1211
<br>

‎docs/index.md‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
<div align="center">
2+
3+
# Block-Recurrent Dynamics in ViTs (Raptor)
4+
5+
<img src="assets/raptor_logo.png" width="33%" />
6+
7+
[![tests](https://github.com/KempnerInstitute/raptor/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/KempnerInstitute/raptor/actions/workflows/tests.yml)
8+
[![arXiv](https://img.shields.io/badge/arXiv-1234.56789-b31b1b.svg)](https://arxiv.org/abs/1234.56789)
9+
10+
[**Mozes Jacobs**](https://scholar.google.com/citations?user=8Dm0KfQAAAAJ&hl=en)$^{\star1}$ &nbsp; [**Thomas Fel**](https://thomasfel.me)$^{\star1}$ &nbsp; [**Richard Hakim**](https://richhakim.com/)$^{\star1}$
11+
<br>
12+
[**Alessandra Brondetta**](https://alessandrabrondetta.github.io/)$^{2}$ &nbsp; [**Demba Ba**](https://scholar.google.com/citations?user=qHiACEgAAAAJ&hl=en)$^{1,3}$ &nbsp; [**T. Andy Keller**](https://akandykeller.github.io/)$^{1}$
13+
14+
<small>
15+
16+
$^1$**Kempner Institute, Harvard University** &nbsp; $^2$**Osnabrück University** &nbsp; $^3$**Harvard University**
17+
18+
</small>
19+
20+
</div>
21+
22+
---
23+
24+
**tl;dr** Our work introduces the Block-Recurrent Hypothesis (BRH), by noticing that foundation models like DINOv2 can be rewritten using only two recurrent blocks to recover 96% of the original accuracy. We leverage our framework and explore a Dynamical Interpretability approach where we interpret token evolution through layers as trajectories and show that they converge into class-dependent angular basins while late-stage updates collapse into low-rank attractors.
25+
26+
Ultimately, the study reveals that Vision Transformers seems to naturally converge toward compact, iterative programs instead of unique layer-by-layer transformations (indicating a lower algorithmic complexity / Kolmogorov complexity).
27+
28+
---
29+
30+
## Setup
31+
32+
### Environment
33+
To run the code, you will need to create a mamba (or conda) environment from the `environment.yml` file.
34+
Create and activate the environment with
35+
```bash
36+
mamba env create -f environment.yml
37+
mamba activate raptor
38+
```
39+
40+
### Paths
41+
Edit `src/paths.py` to have the correct absolute paths to different datasets.
42+
43+
### Extracting DINOv2 Activations for ImageNet-1k
44+
For ImageNet, we precompute the DINOv2 activations so that `Raptor` can train faster.
45+
We provide a script to extract the activations from the ImageNet-1k dataset. This script is available in the `data` directory.
46+
This script takes around 5 hours to run on 1 H100 GPU, and storing the activations requires a lot of disk space.
47+
```bash
48+
cd data
49+
python precompute_dinov2_act.py
50+
```
51+
52+
### Download Pretrained Classifiers
53+
Download the DINOv2 linear heads from Meta's [repository](https://github.com/facebookresearch/dinov2).
54+
These are used during training of `Raptor`.
55+
56+
```bash
57+
cd src
58+
wget https://dl.fbaipublicfiles.com/dinov2/dinov2_vitb14/dinov2_vitb14_reg4_linear_head.pth
59+
wget https://dl.fbaipublicfiles.com/dinov2/dinov2_vits14/dinov2_vits14_reg4_linear_head.pth
60+
cp dinov2_vitb14_reg4_linear_head.pth imagenet_probes/dinov2_vitb14_reg4_linear_head.pth
61+
cp dinov2_vits14_reg4_linear_head.pth imagenet_probes/dinov2_vits14_reg4_linear_head.pth
62+
```
63+
64+
## Usage Example
65+
`Raptor` training follows 4 main steps. Here, we show example usage for a 3-block `Raptor`:
66+
67+
1. Determine max-cut segmentations. This has been done for you in src/000_max_cut_dinov2_base.ipynb.
68+
2. Train each block independently.
69+
```bash
70+
cd src
71+
python trainer.py --teacher_force --mse --sigma 0 --lr 3e-4 --wandb --t_scale --swiglu --start_layer 0 --end_layer 7 --seed 100
72+
python trainer.py --teacher_force --mse --sigma 0 --lr 3e-4 --wandb --t_scale --swiglu --start_layer 7 --end_layer 10 --seed 101
73+
python trainer.py --teacher_force --weighted --sigma 0 --lr 3e-4 --wandb --t_scale --swiglu --start_layer 10 --end_layer 12 --seed 104
74+
```
75+
3. Train the full model with the pretrained blocks.
76+
```bash
77+
cd src
78+
BP1="final_weighted_False_autoregressive_False_distillation_False_teacher_True_mse_True_cosine_False_t_scale_True_swiglu_True_sigma_0.0_start_0_end_7_lr_0.0003_cls_weight_0.34_reg_weight_0.33_patch_weight_0.33_seed_100_step_312500.pt"
79+
BP2="final_weighted_False_autoregressive_False_distillation_False_teacher_True_mse_True_cosine_False_t_scale_True_swiglu_True_sigma_0.0_start_7_end_10_lr_0.0003_cls_weight_0.34_reg_weight_0.33_patch_weight_0.33_seed_101_step_312500.pt"
80+
BP3="final_weighted_True_autoregressive_False_distillation_False_teacher_True_mse_False_cosine_False_t_scale_True_swiglu_True_sigma_0.0_start_10_end_12_lr_0.0003_cls_weight_0.34_reg_weight_0.33_patch_weight_0.33_seed_104_step_312500.pt"
81+
python trainer.py --raptor3 --autoreg --weighted --sigma 0 --lr 3e-4 --wandb --t_scale --swiglu --start_layer 0 --end_layer 12 --cls_weight 0.45 --reg_weight 0.10 --patch_weight 0.45 --bp1 $BP1 --bp2 $BP2 --bp3 $BP3 --seed 1101
82+
```
83+
4. Train linear probes on the frozen pretrained checkpoints.
84+
```bash
85+
cd src/imagenet_probes
86+
python train_probe.py --variant raptor3 --model_seed 1101 --seed 4005
87+
```
88+
```bash
89+
cd src/ade20k_probes
90+
python train_probe.py --variant raptor3 --model_seed 1101 --seed 5005
91+
```
92+
```bash
93+
cd src/nyud_probes
94+
python train_probe.py --variant raptor3 --model_seed 1101 --seed 6005
95+
```
96+
97+
## Reproducing Foundation Models Results (Section 3)
98+
To reproduce the results for the foundation models section (Table 1 and Figure 7), do the following:
99+
100+
1. Determine max-cut segmentations. This has been done for you in src/max_cut_dinov2_base.ipynb.
101+
2. Train each block independently.
102+
```bash
103+
cd src/runs
104+
sbatch blocks.sh
105+
```
106+
3. Train the full model with the pretrained blocks.
107+
```bash
108+
cd src/runs
109+
sbatch 002_raptor2_pretrained.sh
110+
sbatch 003_raptor3_pretrained.sh
111+
sbatch 004_raptor4_pretrained.sh
112+
```
113+
4. Train linear probes on the frozen pretrained checkpoints.
114+
```bash
115+
cd src/ade20k_probes
116+
sbatch run_all.sh
117+
```
118+
```bash
119+
cd src/imagenet_probes
120+
sbatch run_all.sh
121+
```
122+
```bash
123+
cd src/nyud_probes
124+
sbatch run_all.sh
125+
```
126+
5. Table 1
127+
```bash
128+
cd src
129+
python aggregate_results.py
130+
```
131+
6. Figure 7
132+
Run the notebook in src/imagenet_probes/101_eval_error_bars.ipynb.

0 commit comments

Comments
 (0)