Fenchel-Young duality is widely used in convex optimization and relies on the conjugacy operation for convex functions; however, alternative notions of duality relying on parallel operations exist as well. In particular, gauge and perspective duality are defined via the polarity operation on gauge functions. We present a perturbation argument for deriving gauge duality, thus placing it on equal footing with Fenchel-Young duality. This approach also yields explicit optimality conditions (analogous to KKT conditions), and a simple primal-from-dual recovery method based on existing algorithms. Numerical results confirm the usefulness of this approach in certain contexts (e.g. optimization over PLQ functions).