Using the Direct Selection Tool (A), reselect your rectangle, move to the control panel and enter "5px" in the Corners box. If you happen to use an older version of Illustrator you will have to use the Rounded Corners effect. Make sure that your rectangle is selected and go to Effect > Stylize > Rounded Corners. Enter a 5px Radius, click OK and go to Object > Expand Appearance. In the end your shape should look like in the following image.