This was the problem that took me longest to twig, but voices that are clearly distinct to a native speaker can be completely confusing for a non-native speaker. I haven't quite worked out why those problems occur on some occasions and not on others, but the native speaker could be identifying a lisp, an accent or a difference in range of tone that escapes a student. You can avoid these problems by using texts with one woman and one man, or you can practice them with tasks where the students only have to count how many times the speaker changes.