Human detection is a significant and challenging task with applications in various domains. In real-time systems, the speed of detection is crucial to the performance of system, while the accuracy is also taken into consideration. In this work, a human detection approach based on Histograms of Oriented Gradients (HOG) feature and differential evolution (DE), termed as HOG-SVM-DE, is proposed to achieve both fast and accurate detection. The proposed method considers the problem of locating an objective detection window as a search problem, and speeds up the detection stage by solving the search problem with DE. DE is chosen as the optimizer as it is characterized by fast and global convergence. The proposed system trains only one linear-SVM, and allows tradeoffs between the detection rate and the detection time to satisfy different applications by simply tuning one parameter. Experiments are conducted on a set of images from the INRIA Person Dataset, and the results validate that the proposed HOG-SVM-DE is promising in terms of both speed and accuracy.