Grain legumes play a key role in smallholder farming systems in sub-Saharan Africa (SSA), in relation to food and nutrition security and income generation. Moreover, because of their N2-fixation capacity, such legumes can also have a positive influence on soil fertility. Notwithstanding many decades of research on the agronomy of grain legumes, their N2-fixation capacity, and their contribution to overall system productivity, several issues remain to be resolved to realize fully the benefits of grain legumes. In this paper we highlight major lessons learnt and expose key knowledge gaps in relation to grain legumes and their contributions to farming system productivity. The symbiosis between legumes and rhizobia forms the basis for its benefits and biological N2-fixation (BNF) relies as much on the legume genotype as on the rhizobial strains. As such, breeding grain legumes for BNF deserves considerably more attention. Even promiscuous varieties usually respond to inoculation, and as African soils contain a huge pool of unexploited biodiversity with potential to contribute elite rhizobial strains, strain selection should go hand-in-hand with legume breeding for N2-fixation. Although inoculated strains can outcompete indigenous strains, our understanding of what constitutes a good competitor is rudimentary, as well as which factors affect the persistence of inoculated rhizobia, which in its turn determines whether a farmer needs to re-inoculate each and every season. Although it is commonly assumed that indigenous rhizobia are better adapted to local conditions than elite strains used in inoculants, there is little evidence that this is the case. The problems of delivering inoculants to smallholders through poorly-developed supply chains in Africa necessitates inoculants based on sterile carriers with long shelf life. Other factors critical for a well-functioning symbiosis are also central to the overall productivity of grain legumes. Good agronomic practices, including the use of phosphorus (P)-containing fertilizer, improve legume yields though responses to inputs are usually very variable. In some situations, a considerable proportion of soils show no response of legumes to applied inputs, often referred to as non-responsive soils. Understanding the causes underlying this phenomenon is limited and hinders the uptake of legume agronomy practices. Grain legumes also contribute to the productivity of farming systems, although such effects are commonly greater in rotational than in intercropping systems. While most cropping systems allow for the integration of legumes, intercropped legumes provide only marginal benefits to associated crops. Important rotational benefits have been shown for most grain legumes though those with the highest N accumulation and lowest N harvest index appear to demonstrate higher residual benefits. N balance estimates often results in contradictory observations, mostly caused by the lack of understanding of belowground contributions of legumes to the N balance. Lastly, the ultimate condition for increased uptake of grain legumes by smallholder farmers lies in the understanding of how legume technologies and management practices can be tailored to the enormous diversity of agroecologies, farming systems, and smallholder farms in SSA. In conclusion, while research on grain legumes has revealed a number of important insights that will guide realization of the full potential of such legumes to the sustainable intensification of smallholder farming systems in SSA, many research challenges remain to be addressed to realize the full potential of BNF in these systems.