The constraints of your project, such as budget and timescales, will generally dictate how many users you test with more than any rules of thumb. Remember that user testing is predominantly used as a form of qualitative research, so you needn't worry about statistical significance.