Small ruminants (i.e., sheep and goats) are ubiquitous, and contribute significantly to the subsistence, economic and social livelihoods of a large human population in low-input, smallholder production systems in developing countries. Increasing human population, urbanization and incomes, coupled with changing consumer preferences are creating more demand for these animals and their products. This demand can effectively be met by substantially increasing the productivity of these animals. Integrated effort in terms of management, health, genetic improvement and product technology to enhance production and decrease wastage is, therefore, desirable. Efficient genetic improvement programs can boost output and profitability for the smallholders. However, there is a lack of information on sustainable conventional genetic improvement programs under smallholder production circumstances. Consequently, methods for implementing and the factors influencing the success of genetic improvement programs should be studied. This review focuses on the technical and infrastructural issues affecting the genetic improvement of small ruminants in low-input, smallholder production systems. It is concluded that a key step is to identify existing structures, institutions, and indigenous breeding practices, and, to build upon these foundation programs where there are opportunities for sustainable genetic improvement.