I had an email from another collector who said that she started to collect them "because pencils were small and cheap, and buying them was an easy way to give in to that urge to buy some tacky souvenir when I went someplace new. Pencils are also good items for traveling friends and relatives to get me - I get a souvenir from somewhere I've never been, and the travelers feel like they got off cheap"
I couldn't put it better!
Now my interests have changed - I've switched from souvenir to brand name writing, and I've started to get interested in pencil ephemera as well as pencils. But still the attraction is the same - they're cheap, there's enough variety to keep you interested, but not so much that can't keep track. They're fun, and there aren't many of us about.