Rural households in sub-Saharan Africa earn a substantial part of their living from rain-fed smallholder agriculture, which is highly sensitive to climate change. There is a growing number of multi-level assessments on impacts and adaptation options for African smallholder systems under climate change, yet few studies translate impacts at the individual crop level to vulnerability at the household level, at which other livelihood activities need to be considered. Further, these assessments often use representative household types rather than considering the diversity of households for the identification of larger-scale patterns at sub-national and national levels. We developed a framework that combines crop suitability maps with a household food availability analysis to quantify household vulnerability to climate-related impacts on crop production and effects of adaptation options. The framework was tested for Uganda, identifying four hotspots of household vulnerability across the country. Hotspots were visually identified as areas with a relatively high concentration of vulnerable households, experiencing a decline in household crop suitability. About 30% of the households in the hotspots in (central) southwest were vulnerable to a combination of 3 °C temperature increase and 10% rainfall decline through declining suitability for several key crops (including highland banana, cassava, maize and sorghum). In contrast only 10% of the households in West Nile and central northern Uganda were negatively affected, and these were mainly affected by declining suitability of common beans. Households that depended on common beans and lived at lower elevations in West Nile and central north were vulnerable to a 2 to 3 °C temperature increase, while households located at higher elevations (above 1100–2000 m.a.s.l. depending on the crop) benefited from such an increase. Options for adaptation to increasing temperatures were most beneficial in northern Uganda, while drought-related adaptation options were more beneficial in the southwest. This framework provides a basis for decision makers who need information on where the vulnerable households are, what crops drive the vulnerability at household level and which intervention efforts are most beneficial in which regions.