Efforts to increase agricultural productivity, adapt to climate change, and reduce the carbon footprint of agriculture are reflected in a growing interest in climate-smart agriculture (CSA). Specific indicators of productivity, adaptation and mitigation are commonly used in support of claims about the climate smartness of practices. However, it is rare that these three objectives can be optimized simultaneously by any one strategy. In evaluating the relative climate smartness of different agricultural practices, plans and policies, there is a need for metrics that can simultaneously represent all three objectives and therefore be used in comparing strategies that have different benefits and trade-offs across this triad of objectives. In this context, a method for developing a Climate Smartness Index (CSI) is presented. The process of developing the index follows four steps: (1) defining system specific climate smartness; (2) selecting relevant indicators; (3) normalizing against reference values from a systematic literature review; and (4) aggregating and weighting. The CSI presented here has been developed for application in a systematic review of rice irrigation strategies and it combines normalized water productivity (WP) and greenhouse gas intensity (GHGI) The CSI was developed for application to data from published field experiments that assessed the impact of water management practices in irrigated rice, focusing on practices heralded as climate-smart strategies, such as Alternate Wetting and Drying (AWD). The analysis shows that the CSI can provide a consistent judgment of the treatments based on the evidence of water efficiency and reduced GHGI reported in such studies. Using a measurable and replicable index supports the aim of generating a reliable quantification of the climate smartness of agricultural practices. The same four step process can be used to build metrics for a broad range of CSA practice, policy and planning.