The loss landscape of deep linear neural networks: a second-order analysis - IRT Saint Exupéry - Institut de Recherche Technologique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2022

The loss landscape of deep linear neural networks: a second-order analysis

Résumé

We study the optimization landscape of deep linear neural networks with the square loss. It is known that, under weak assumptions, there are no spurious local minima and no local maxima. However, the existence and diversity of non-strict saddle points, which can play a role in first-order algorithms' dynamics, have only been lightly studied. We go a step further with a full analysis of the optimization landscape at order 2. We characterize, among all critical points, which are global minimizers, strict saddle points, and non-strict saddle points. We enumerate all the associated critical values. The characterization is simple, involves conditions on the ranks of partial matrix products, and sheds some light on global convergence or implicit regularization that have been proved or observed when optimizing linear neural networks. In passing, we provide an explicit parameterization of the set of all global minimizers and exhibit large sets of strict and non-strict saddle points.
Fichier principal
Vignette du fichier
main.pdf (749.2 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03299887 , version 1 (28-07-2021)
hal-03299887 , version 2 (24-02-2022)

Identifiants

Citer

El Mehdi Achour, François Malgouyres, Sébastien Gerchinovitz. The loss landscape of deep linear neural networks: a second-order analysis. 2022. ⟨hal-03299887v2⟩
147 Consultations
288 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More