We develop a nonlinear model of chatter in drilling that incorporates friction of the material on the cutting tool and its interaction with the axial-torsional mode of vibration seen in twist drills. Stability criteria are determined for both regenerative and non-regenerative chatter, and the effect of tool parameters and the friction law itself on the results is analysed. Our analysis shows that the exact form of the friction law is not critical in the stability calculation, only the size of the friction coefficient for steady cutting, and the slope of the friction law at the steady cutting state. However, its interaction with the geometry of the vibration is crucial. In the laboratory, drilling vibrational instabilities can occur at frequencies less than the natural frequency of the excited mode of the drill, and we find this result depends critically on the shape of the vibration mode projected onto the cutting edge of the drill.