One of the major problems within eye blink detection algorithms is the data insufficiency and subsequent over-fitting on existing datasets. We collected new, more challenging datasets during an event called Researcher’s night where more than 100 unique people were recorded and 1849 blinks annotated. Another problem is the evaluation procedure that is often not specified within published algorithms. We propose an evaluation based on intersection over union metric to define the detected blink. We extend an annotation of individual videos with the face and eye corner positions. Consequently, performance of an eye blink detection algorithm is measured without an influence of used face and eye detection method.