Contribution
It puts SFA’s slowness loss inside a variational objective with a nonlinear generative model, where earlier probabilistic SFA (as the authors describe it) needed linear emissions. With a random-walk prior and an instantaneous Gaussian encoder, the KL term of the ELBO is the unordered slowness loss, and the reconstruction term takes over from SFA’s unit-variance and decorrelation constraints. The paper does not claim equivalence with SFA and does not solve for any optimum.
Key insight
Slowness is the price of the encoder disagreeing with a random-walk prior. If the prior says z_t ≈ z_{t−1} with unit noise and the posterior approximation is N(g(x_t), I) at each step separately, then the KL cost of each step is half the squared jump of the encoder’s mean. In VAE terms the roles swap: the KL “regularizer” is SFA’s objective, and the reconstruction “objective” is what stops the features collapsing to constants. What keeps them from collapsing is the fixed encoder noise: a feature shrunk below the noise is lost to reconstruction.
Assumptions
- Time series x_{1:T}, x_t ∈ ℝⁿ; latents z_{1:T}, z_t ∈ ℝ
- Prior: Markov, p(z_t | z_{t−1}) = N(z_{t−1}, I) (Eq. 13). No decay (λ_i = 1 in the paper’s contrast with Turner and Sahani), so no stationary distribution, and the latent dimensions are independent and equally slow. p(z_1) is not specified.
- Forward model: p_θ(x_{1:T} | z_{1:T}) = Π_t p_θ(x_t | z_t) (Eq. 14), differentiable in θ, otherwise free.
- Variational family: instantaneous, q(z_{1:T} | x_{1:T}) = Π_t q(z_t | x_t), with q(z_t | x_t) = N(g_φ(x_t), I), unit covariance fixed (Eqs. 19–20). This is narrower than the posterior’s structure (Eq. 15) and than mean-field or causal alternatives (Eqs. 17–18), which the authors name and set aside.
- Linear case: q(z | x) = N(Wᵀx + b, I), p(x | z) = N(Vz + o, I) (Eqs. 43–44), and “for large T” the lagged second moment equals the unlagged one.
- The ordering of SFA’s features is discarded from the start (Eq. 2).
Key results
- The ELBO (Eq. 40). L = Σ_t E_{z_t~q} log p_θ(x_t | z_t) − ½Σ_t ‖g_φ(x_{t−1}) − g_φ(x_t)‖². Re-derived: the exact KL is ½Σ_{t≥2}‖Δg‖² + (T − 1)d/2, plus a t = 1 boundary term the paper does not write (THEORY-775, Step 1). Holds when: the assumptions above, for any g and any forward model.
- Diagonal covariances (§6). For any diagonal Σ_p and Σ_q only the Σ_p⁻¹-norm changes, which reweights the features’ contributions. True of the mean term; with Σ_q ≠ Σ_p the trace and log-determinant terms also enter, as constants in g.
- Linear objective (Eq. 41). L = c − (T/2)tr(VᵀV) − ½Σ‖x_t − x̃_t‖²
− ½Σ‖Wᵀx_{t−1} − Wᵀx_t‖², with x̃_t = VWᵀx_t + Vb + o. The paper prints
- before the reconstruction sum; Appendix Eq. 54 has the correct −.
- Necessary condition (Eq. 42, Eq. 92). WᵀĊW = VᵀV and o = x̄ − VWᵀx̄ − Vb, with Ċ the second moment of the step differences. Re-derived (THEORY-775, Step 3). Necessary only.
- The optimum, which the paper does not give (the record’s, THEORY-775, Steps 4–6). On whitened input the maximizers are W = [e_1 … e_k] diag(s_i*) Rᵀ, e_i the bottom eigenvectors of Ċ (SFA’s slow features), s_i* = (μ_i^{−1/2} − 1)^{1/2} for μ_i < 1 and 0 otherwise, R any orthogonal matrix. μ_i < 1 is lag-one autocorrelation above ½. On raw input a feature of variance c is kept iff c > 2(1 − ρ), so the solution depends on the input’s scale.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | Under the stated prior and variational family, the ELBO is per-point reconstruction minus half the unordered slowness loss | strong (up to a dropped constant) | derivation Eqs. 23–40; re-derived and checked by Monte Carlo |
| C2 | Unlike earlier probabilistic SFA, the slowness hypothesis is “less clearly woven into the prior” and “instead emerges from inference” | moderate | Eq. 13 has no decay, so slowness is not ordered by the prior; but the prior still prefers small steps, and Step 1 shows the slowness loss is exactly the prior’s penalty |
| C3 | The reconstruction term plays the role of the decorrelation constraint, through the bottleneck | weak; wrong as stated in the linear case | informal argument, §7; the optimal linear features are correlated for R ≠ I (THEORY-775, Step 5) |
| C4 | The probabilistic reconstruction term plays the role of the unit-variance constraint, by forbidding indefinite down-scaling | moderate | informal argument, §7; confirmed in the linear case, where the scale is fixed at s* and slow-enough features survive, but features with ρ ≤ ½ are scaled to zero, which SFA’s constraint would not do |
| C5 | With linear Gaussian encoder and decoder, ∂L = 0 “are equivalent to” WᵀĊW = VᵀV and the condition on o | weak as stated; strong as a necessary condition | Appendix A; the combination of stationarity equations loses information (a random W meeting it is far from optimal) |
| C6 | Turner and Sahani recover linear SFA as the ML emission model, exactly only in the limit of zero observation noise and white-noise latent dynamics, the latter “mainly for scale correction” | reported, not checked here | §2, one sentence and a footnote; LIT-1287 is Unavailable |
Method
Not an algorithm with experiments. The paper derives an objective and proposes training g_φ and p_θ jointly by stochastic variational inference with sampling from q (§7). It reports no implementation and no experiments.
Concepts
- Unordered slowness objective: ⟨‖g(x_{t+1}) − g(x_t)‖²⟩_t for a vector g, the sum of SFA’s per-feature losses without its sequential order (Eq. 2).
- PSFA: the authors’ name for Turner and Sahani’s probabilistic SFA, a linear Gaussian state-space model with linear emissions.
- Instantaneous factorization: q(z_t | x_t) only, mimicking SFA’s instantaneous feature map (Eq. 19, Fig. 4c).
- DVAE: dynamical variational autoencoder (Girin et al. 2021), the family this model belongs to.
Connections
It follows Turner and Sahani (2007) and the extensions it lists (Omori 2013 with observation noise; Fan et al. 2018 with t-distributed noise; Tsujimoto and Omori 2020 with switching dynamics; Puli and Huang 2024 with nonlinearities) and places SFA among dynamical VAEs (Girin et al. 2021). Bengio and Bergstra (2009) are cited for SFA with the constraints as soft penalties; Schüler et al. (2019) for SFA trained by gradient through differentiable whitening. The related-work paragraph cites “Kingma & Ba, 2014” (the Adam paper) for variational autoencoders where Kingma and Welling (2014) is meant.
Bearing on the record
-
What the paper says about Turner and Sahani, exactly. §2: “Almost all subsequent work in this direction … is based on an initial treatment of probabilistic SFA (later called PSFA) (Turner & Sahani, 2007) in which SFA was derived from a linear Gaussian state space model … with linear emissions. Linear SFA was recovered as the maximum-likelihood estimate of the emission model, with exact equivalence only in the limit of zero observation noise and annealing of the internal dynamics towards white noise”, with the footnote “Although the latter mainly for scale correction.” §4: Turner and Sahani “assumed the next step to be centered on a linear transformation Λz_{t−1} using a diagonal Λ with ordered diagonal elements λ_i”. That is all. It is a summary by a later author and is not a reading of LIT-1287, which stays Unavailable.
-
The record’s own check of that kind of model (not the paper’s, and not Turner and Sahani’s text). Take x_t = A z_t with A square and invertible (no observation noise) and independent latents z_{i,t} = λ_i z_{i,t−1} + (1 − λ_i²)^{1/2} η_{i,t}, λ_i ∈ (0, 1) distinct. With W = A⁻¹ (rows w_i) and edge terms ignored, the log-likelihood per step is log|det W| − Σ_i [a_i w_iᵀCw_i + b_i w_iᵀĊw_i], a_i = (1 − λ_i)/(2(1 + λ_i)), b_i = λ_i/(2(1 − λ_i²)), using ⟨(y_t − λy_{t−1})²⟩ = (1 − λ)²⟨y²⟩ + λ⟨(Δy)²⟩. Stationarity gives 2w_jᵀ(a_iC + b_iĊ)w_i = δ_ij. For i ≠ j this and its swap are two linear equations in w_iᵀCw_j and w_iᵀĊw_j whose determinant vanishes only if b_i/a_i = b_j/a_j; b/a = λ/(1 − λ)² is strictly increasing on (0, 1), so both cross terms vanish, and the ML rows are generalized eigenvectors of (Ċ, C), SFA’s pencil (THEORY-746), with no limit taken. The pairing that maximizes the likelihood sends the largest λ to the smallest eigenvalue, since log(1 + κμ) is supermodular. The output variances are 1/(2(a_i + b_iμ_i)), which is 1 when the data fit the model and tends to 1 as λ_i → 0 for any data. That matches the paper’s footnote that the white-noise limit is for scale. Checked: with λ = (0.9, 0.7, 0.3), a random 3 × 3 mixing and 10⁵ steps, the ML rows found by BFGS had C-cosines with the SFA eigenvectors of 1 to 10⁻⁵, in the predicted order.
-
THEORY-733, THEORY-257: slowness and closure, the record’s derivation. For a stationary Gaussian VAR(1) x_t = Ax_{t−1} + ε_t with covariance C, the lag covariance is AC and Ċ = 2C − AC − CAᵀ. If the process is time-reversible (AC = CAᵀ), Ċ = 2C(I − Aᵀ), so Ċw = μCw iff Aᵀw = (1 − μ/2)w: every SFA direction is a left eigenvector of A, any span of them is Aᵀ-invariant, and so it is exactly dynamically independent by THEORY-733. The reversible Gaussian VAR(1)s are exactly the noise-free square mixtures of independent AR(1) latents above, with λ_i allowed in (−1, 1) (put A = C^{1/2}SC^{−1/2} with S symmetric and diagonalize S). Off reversibility it fails: for A = [[0.8, 0.5, 0], [0, 0.5, 0.4], [0.1, 0, 0.3]] the slowest SFA direction w had ‖wᵀA − (wᵀAw)wᵀ‖ = 0.26 and |cos| 0.88 with the nearest left eigenvector, against 0.0015 and 1.000 in a reversible example. So in the linear Gaussian case, slowness selects closed coarse-grainings exactly when the process is reversible; otherwise the slowest feature is generally not closed and carries memory of the rest. Neither paper states this. It deserves a demonstrated CLAIM, with this NOTE as its ground.
-
QUESTION-217. NOTE-971 asked under which generative assumptions slow features are coarse, predictively sufficient or quickly learned.
- Generative assumptions: two are now on the record, the paper’s (random walk, any emission, instantaneous Gaussian encoder) and the noise-free linear AR(1) one checked above. In both, slowness is placed in the prior.
- Coarse: in the linear whitened variational case the optimal spans nest by slowness as the latent dimension grows, so a slowness order is a refinement order of spans there, up to a cut at ρ = ½; within a span there is no order (THEORY-775). This is the commuting case again, as in THEORY-746.
- Predictively sufficient: the paper does not address it, and its encoder sees only x_t. In the linear reversible case the slow features are closed, which is self-sufficiency, not sufficiency for predicting x.
- Quickly learned: not addressed.
-
QUESTION-222. By the reversibility result, a slow linear feature of an irreversible Gaussian VAR(1) is generally not Markov on its own: slowness does not buy freedom from memory. The paper’s suggestion to condition q on the past (§8) is the variational counterpart of a Markovian augmentation, and it is not developed.
-
QUESTION-218. Nothing directly.
-
CLAIM-331. Both probabilistic readings write slowness into the prior: Turner and Sahani by an ordered Λ (as reported), this paper by a random walk. A probabilistic SFA cannot count as evidence that higher levels are slower. It is consistent with THEORY-746’s conclusion that in SFA “higher is slower” is imposed by the objective.
-
CLAIM-327. Nothing on endogeneity.
-
Anthology. No instruction for machine-learning practice; §8’s research directions are suggestions, not findings. The anthology-candidate flag is for the representation-learning content.
Limitations
- No experiments, no optimum. The paper derives an objective and argues about it. Whether the variational objective learns anything SFA-like with a nonlinear g is not shown.
- Slips. Eq. 35’s spurious −d and the dropped constant; the sign in Eq. 41; “equivalent” for a necessary condition in §7; the unwritten t = 1 term and unstated p(z_1); the Adam citation for VAEs.
- The decorrelation argument fails in the linear case. The optimal features are an arbitrary rotation of scaled slow features, so they are correlated unless R = I. The reconstruction term secures a non-redundant span, not decorrelated outputs.
- A threshold SFA lacks. On whitened input, features with lag-one autocorrelation ρ ≤ ½ get zero weight, whatever k is. On raw input the threshold is c > 2(1 − ρ), which depends on scale.
- The model is improper as a stationary process. A random walk with unit steps has no stationary distribution, which the authors note. How this interacts with stationary data is not discussed.
- Turner and Sahani are summarized, not reproduced. The record’s check above is of a noise-free model of that kind, not of their paper.
Open questions
- What does the variational objective learn with a nonlinear encoder, and does the ρ > ½ cut have a nonlinear counterpart? A solution for a polynomial or kernel encoder on a known generative process would answer it.
- With q conditioned on the past (Eq. 18), does the optimum become a predictive state, sufficient for the future of x? A linear Gaussian computation, comparing it with the Kalman filter’s state, would answer it and bear on QUESTION-222.
- With observation noise, how do the ML directions of the linear AR(1) model depart from SFA’s pencil? Turner and Sahani’s paper is the place to look once a copy is reached.