The Handy AR presents a vision-based user interface that tracks a user's outstretched hand to use it as the reference pattern for augmented reality (AR) inspection, providing a 6-DOF camera pose estimation from the tracked fingertip configuration. A hand pose model is constructed in a one-time calibration step by measuring the fingertip positions relative to each other in presence of ground-truth scale information. Through frame-by-frame reconstruction of the camera pose relative to the hand, we can stabilize 3D graphics annotations on top of the hand, allowing the user to inspect such virtual objects conveniently from different viewing angles in AR.