Accurate forecasts of water demand are required for real-time control of water supply systems under normal and abnormal conditions. A methodology is presented for quantifying, diagnosing and reducing model structural and predictive errors for the development of short term water demand forecasting models. The methodology (re-)emphasises the importance of posterior predictive checks of modelling assumptions in model development, and to account for inherent demand uncertainty, quantifies model performance probabilistically through evaluation of the sharpness and reliability of model predictive distributions. The methodology, when applied to forecast demand for three District Meter Areas in the UK, revealed the inappropriateness of simplistic Gaussian residual assumptions in demand forecasting. An iteratively revised, parsimonious model using a formal Bayesian likelihood function that accounts for kurtosis and heteroscedasticity in the residuals led to sharper yet reliable predictive distributions that better quantifies the time varying nature of demand uncertainty across the day in water supply systems.