The Quality slider ranges from (Low Quality) 51 to 0 (High Quality) For good quality output at reasonable file sizes, the following values are recommended:
Standard Definition (e.g DVD's)
Use an RF value of around 20 +/- 1
As an example using the AppleTV2 preset at RF20, with 20 different sources, the average size was 925MB per hour of video. (Min: 625MB/h Max:1,503MB/hr)
High Definition (e.g Blurays 720/1080)
Use an RF value of 22 +/- 1
Since HD sources are typically quality, you can get away with a slightly higher RF value than SD content without any perceived difference in quality.
You should note that the quality slider is not linear for x264. It is logarithmic (like the Richter or pH scales.) This means small movements in the numbers can have large results.
You can control the granularity of the slider in the preferences / options.
You can encode a few chapters of a source to experiment with different quality values if the above recommendations are not suitable. Encoding a 10minute sample in the middle of your source should give you a good idea of what to expect.