Finding the ground-state configuration of point defects in semiconductors is in general a high-dimensional optimization problem. However, their adiabatic potential energy surfaces (APES) can be described with few effective coordinates by exploiting symmetry arguments within the Jahn-Teller (JT) theoretical framework. In this paper, we propose a general framework, that combines group theory and ab initio calculations, to build a full picture of common defects in diamond lattices (vacancies and substitutional impurities) as instances of the general T ⊗ (e + t 2) APES. Starting from the original tetrahedral symmetry, we consider all possible JT symmetries and we numerically investigate the shape of the APES by targeting the minimum energy path between equivalent JT distortions taking as examples the vacancy (V) and the metallic impurities Pd, Pt, and Au in crystalline silicon.