The middle and late-innings relievers don't sit on the bench with the rest of their team, but normally sit in a warm-up area known as the "bullpen". It generally takes a pitcher five to ten minutes to get warm, so when the manager thinks he may need a pitcher out of the bullpen, you'll see a telephone call made to the bullpen to get a pitcher warming up.
Quite often a pitcher will warm up in the bullpen because the pitcher on the mound is getting into trouble and may need replacing. Then the pitcher gets himself out of the jam and the bullpen pitcher sits down again. There's only so many times a pitcher can warm up without risking injury though, and it may be that after a couple of false alarms he's no longer available to pitch in that game.
Which pitchers a manager will use from the bullpen is a decision made on the spot. Sometimes he may be restricted by fatigue (if a bullpen pitcher pitched two or three innings the day before, he may not be available for that game), sometimes it may be dependent on the hitter coming up (a left handed pitcher is brought in to face a key left handed hitter), sometimes it may depend on how many innings it's hoped he'll be able to cover.
Replacing a Pitcher