Crop productivity in Rwanda is declining as a result of intensive farming on steep slopes, which leads to soil loss and declining soil fertility particularly in the northern highlands. Slow-forming terraces have been widely adopted in the northern highlands of Rwanda to control soil erosion however not much been done to evaluate their efficiency. We hypothesized that slow-forming terraces reduce soil loss and soil fertility gradients compared with non-conserved land. A field experiment compared the soil erosion rates and fertility gradients of 20+ year old terraces where sole grass strips (Pennisetum purpureum) or grass strips combined with infiltration ditches were used with those of land where no soil conservation technologies were applied. The experiment was conducted in the year 2010 on three landscape positions (Upperslope, Hillslope and Footslope) along a representative toposequence using farmers’ fields where potato and maize were grown in two consecutive cropping seasons. The highest annual soil loss (41.5 t ha−1) was recorded with plots receiving no soil conservation practices while the lowest annual soil loss (18 t ha−1) was recorded with grass strips combined with infiltration ditches. The slow-forming terraces showed a marked “within” spatial difference in both soil quality and crop yield. The soil in the lower part of the terraces showed as much as 57% more organic carbon content and 31% more available phosphorous than the soil in the upper part. Organic carbon (OC) was higher (OC = 2.1%) on the upperslope than on hillslope (OC = 1.9%). Less than 2.3 t ha−1 of the mean potato yield (23 t ha−1) and only 0.5 t ha−1 of the mean maize yield (5 t ha−1) were recorded on the uppermost third of the terraces on all three landscape positions. The marked soil fertility gradients indicate that the sustainability of slow-forming terraces is threatened, unless a site-specific fertilizer strategy is developed. For the sustainability of these terraces, the current practice of “harvesting” the fertile soil from the lower edge of the grass strip and using it as fertilizer for the nutrient deficient upper parts of terraces needs to be stopped.