Continuous driver authentication is useful in the prevention of car thefts, fraudulent switching of designated drivers, and driving beyond a designated amount of time for asingle driver. In this paper, we propose a deep neural network based approach for real time and continuous authentication of vehicle drivers. Features extracted from pre-trained neural network models are classified with support vector classifiers. In order to examine realistic conditions, we collect 130 in-car driving videos from 52 different subjects. We investigate the conditions under which current face recognition technology will allow commercialization of continuous driver authentication.