Ground Penetrating Radar (GPR) is a remote sensing modality that has been researched extensively for buried threat detection. For this purpose, algorithms can be developed to automatically determine the presence of such threats. To train such algorithms, small 2-dimensional images can be extracted from the larger image, or volume, of GPR data. One thread of research in the buried threat detection literature is to use visual descriptors from the computer vision literature. One recent, very successful approach in that field is the use of deep convolutional neural networks (CNNs). Applying CNNs requires a large number of design choices which complicate their use. In this work, we investigate their application to GPR data and adapt several recent advances from the CNN literature to improve detection performance on GPR data. In particular, we investigate the initialization step of pretraining and propose a dataset augmentation protocol. The efficacy of these approaches are evaluated on several architectures with a relatively similar number of network parameters to learn. The results indicate that both pretraining and dataset augmentation help achieve higher detection performance.