Misra et al. (2019) — ARXIV-1908.08681
Key takeaways
- Proposes
f(x) = x·tanh(softplus(x))— smooth, non-monotonic, and explicitly positioned relative to the Swish familyx·σ(βx)rather than as an unrelated construction - The offered explanation is that the first derivative’s behaviour acts as a regularizer; the paper calls the function “self-regularized” on that basis
- +2.1 AP50 over Leaky ReLU on YOLOv4 / CSP-DarkNet-53 / MS-COCO, and ≈1% top-1 over ReLU on ResNet-50 / ImageNet-1k, “while keeping all other network parameters and hyperparameters constant” — the right comparison, and the one activation papers most often fail to make
- BMVC 2020, and CNN-vision throughout
Standing in the anthology
Skimmed, not read — ar5iv has no rendering for arXiv:1908.08681, so the
reading NOTE-054 is from the abstract, which under
ADR-025 is not enough to source a practice from. The second use of
Skimmed in this record, for the same reason as the first: record the gap
where the next reader finds it rather than imply a reading that did not happen.
Here that costs nothing. Nothing is sourced to this paper and nothing should
be — the record’s activation practice is SwiGLU-shaped, and the gated-linear
line is where language modelling went. Mish is a CNN-vision activation from the
pre-gating era, and Active is already generous.
Of the takeaways it carried, “better gradient flow” is not a claim the abstract makes; the others restate the title and the paper’s own adjective.
Audited by #121; unsourced because it has not been read.
Skimmed, no recommendations extracted. The audit’s determination is that the
absence here carries no information: nobody has assessed whether this paper
should source anything.