In the area of home security and protection, vision-based home automation systems (HAS) play a significant role. In this paper, we present the design and implementation of a smart HAS with incorporated intrusion detection to minimize damages caused by burglary. In addition, the proposed HAS integrates a web server to home appliances in order to remotely access and control their status. Intrusion detection, on the other hand, uses Histogram of Oriented Gradients (HOG) feature descriptors and a Support Vector Machine (SVM) classifier for accurate human detection by smartly rejecting false alarms arising from pets. This system comprised of a simple architecture and requires no human intervention. It allows for prompt accessibility, efficient usage of electricity and provides user convenience.