In this work we investigate the use of deep inverse models (DIMs) for designing artificial electromagnetic materials (AEMs) - such as metamaterials, photonic crystals, and plasmonics - to achieve some desired scattering properties (e.g., transmission or reflection spectrum). DIMs are deep neural networks (i.e., deep learning models) that are specially-designed to solve ill-posed inverse problems. There has recently been tremendous growth in the use of DIMs for solving AEM design problems however there has been little comparison of these approaches to examine their absolute and relative performance capabilities. In this work we compare eight state-of-the-art DIMs on three unique AEM design problems, including two models that are novel to the AEM community. Our results indicate that DIMs can rapidly produce accurate designs to achieve a custom desired scattering on all three problems. Although no single model always performs best, the Neural-Adjoint approach achieves the best overall performance across all problem settings. As a final contribution we show that not all AEM design problems are ill-posed, and in such cases a conventional deep neural network can perform better than DIMs. We recommend that a deep neural network is always employed as a simple baseline approach when addressing AEM design problems. We publish python code for our AEM simulators and our DIMs to enable easy replication of our results, and benchmarking of new DIMs by the AEM community.