This paper presents a methodology for automatically estimating the energy consumption of buildings from aerial imagery using data from Gainesville, Florida. By detecting buildings in the imagery using convolutional neural networks and extracting features from those building annotations, we use only imagery-derived features to estimate building energy consumption using random forests regression. For individual buildings, we achieve a predictive R 2 value of 0.26, and with spatial aggregation over an area of 400m×400m our predictive R 2 value increases to 0.95. We also explore the sensitivity of these estimates to errors in the building estimation process. Our results indicate that information limited to the size and shape of buildings, provides substantial predictive potential for the energy consumption of buildings.