Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Existence, Stability and Scalability of Orthogonal Convolutional Neural Networks

Abstract : Imposing orthogonality on the layers of neural networks is known to facilitate the learning by limiting the exploding/vanishing of the gradient; decorrelate the features; improve the robustness. This paper studies theoretical properties of orthogonal convolutional layers. \indent We establish necessary and sufficient conditions on the layer architecture guaranteeing the existence of an orthogonal convolutional transform. The conditions prove that orthogonal convolutional transforms exist for almost all architectures used in practice for 'circular' padding. We also exhibit limitations with 'valid' boundary condition and 'same' boundary condition with zero padding. \indent Recently, a regularization term imposing the orthogonality of convolutional layers has been proposed, and impressive empirical results have been obtained in different applications (Wang et al. 2020). The second motivation of the present paper is to specify the theory behind this. We make the link between this regularization term and orthogonality measures. In doing so, we show that this regularization strategy is stable with respect to numerical and optimization errors and that, in the presence of small errors and when the size of the signal/image is large, the convolutional layers remain close to isometric. The theoretical results are confirmed with experiments, the landscape of the regularization term is studied and the regularization strategy is validated on real datasets. \indent Altogether, the study guarantees that the regularization with L_{orth} (Wang et al. 2020) is an efficient, flexible and stable numerical strategy to learn orthogonal convolutional layers.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03315801
Contributor : El Mehdi ACHOUR Connect in order to contact the contributor
Submitted on : Tuesday, February 8, 2022 - 11:34:09 AM
Last modification on : Wednesday, June 1, 2022 - 5:15:57 AM

Files

main.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03315801, version 2
  • ARXIV : 2108.05623

Citation

El Mehdi Achour, François Malgouyres, Franck Mamalet. Existence, Stability and Scalability of Orthogonal Convolutional Neural Networks. 2022. ⟨hal-03315801v2⟩

Share

Metrics

Record views

84

Files downloads

97