This paper addresses an NP-hard design optimization problem in a multibeam satellite communication system. This problem consists in designing irregular beam layouts to satisfy non-uniform user traffic demands over the satellite service area, under antenna constraints, satellite payload constraints and a telecommunication mission criterion. Efficiently solving this problem is of crucial importance due to its impact on the system performance and cost. We compare three mixed-integer linear programming formulations. The first one, issued from previous work, is based on a linearization of both convex and non-convex Euclidean distance constraints. The two other aim at reducing the solution space size and at breaking symmetry inherent to the first formulation. For that purpose, we introduce a new process to interface k-means clustering with mixed-integer linear programming. We examine an exact and a heuristic approach for exploiting these principles that yield two new formulations. The heuristic approach outperforms the others based on our tests on a set of large-scale realistic problem instances, allowing to use mixed-integer linear programming in the industrial context.