The relationship of the diffuse N with the non-metals (AN, TN, SOC, and S) displayed similar pattern in both rice paddy and upland sub-basins. The diffuse N in the paddy rice sub-basin was positively correlated with AN and TN contents, while the correlations in the upland were strongly negative (R2 = 0.597). The relationships of diffuse N with the metals (Fe, Mn, Cd, and Pb) showed the opposite characteristics of the non-metals. The correlations were negative in the paddy rice sub-basin and positive in the upland sub-basin. The higher content of AN and TN contributed to the N discharge in the paddy rice sub-basins, but did not increase the watershed loading. Most diffuse N pollution transferred as Org-N under dry conditions, which indicated that the NO3–N surplus in the upland sub-basins did not cause huge risk on water quality. The absorption between soil particles and N was one of the principal factors for the transference.