Make sure your goals and objectives are clearly defined and operationalized. Expectations of students should be written down.
Match your assessment measure to your goals and objectives. Additionally, have the test reviewed by faculty at other schools to obtain feedback from an outside party who is less invested in the instrument.
Get students involved; have the students look over the assessment for troublesome wording, or other difficulties.
If possible, compare your measure with other measures, or data that may be available.