Estimating the Return on Investment (ROI) for AI/ML projects, especially Generative AI (Gen AI), presents unique challenges due to inherent uncertainties and intangible benefits. This article offers a structured framework, detailing eight essential steps, to guide businesses in effectively calculating the potential ROI for such projects, ensuring they make informed decisions before implementation.
Estimating ROI On High Uncertainty AI Projects
Navigating the intricate world of AI/ML projects, many organizations grapple with predicting their Return on Investment (ROI). Unlike conventional projects where assessing risks and returns is more straightforward, AI and specifically, Generative AI (Gen AI) projects, confront higher uncertainties (for a non-technical primer on Gen AI, see this post). These span from fluctuating costs to the myriad intangible benefits that can be challenging to quantify.
Recognizing ROI is paramount before plunging into the depths of any project. There are companies that jump in too hastily, only to later realize that their ROI isn't up to par. Conversely, others might miss great opportunities, being overly cautious and seeking pinpoint ROI accuracy before initiation. This article serves as a simple guide (with examples), offering a structured approach to evaluate the ROI of Gen AI and broadly, any AI/ML project.
Step 1: Set Crystal Clear Objectives
Every successful project commences with defined objectives. In the AI realm, these can range from operational efficiency enhancements and better customer engagement to product innovations or workforce development. Pinpoint a singular, primary objective. Spreading oneself too thin by setting multiple primary goals can dilute the project's potency. While secondary objectives are permissible, the primary one should take precedence.
Example: Company A wanted to leverage on Gen AI to identify and reach out to potential leads via hyper-personalized emails. Company A decides that its primary objective would be to increase the number of leads from its main targeted customer segment.
Step 2: Pinpoint Key Metrics
Once your objectives are in place, settle on the metrics that will chart your project's trajectory. This could be anything from cost savings, revenue upticks, customer contentment levels, or other pertinent performance yardsticks. Ensure these metrics resonate with your goals and are quantifiable.
Example: Company A decides to monitor:
Percentage Uplift in Qualified Sales Lead: A measure of growth in leads from targeted customer brackets.
Time To Value: The duration between initiating an investment and the realization of its value.
Step 3: Project Costs
Now, delineate the financial implications of your endeavour. This encompasses the upfront investment for kickstarting the project and the consistent costs that follow. Recognizing the potential variability in costs, create different scenarios like 'ideal', 'most likely', and 'worst case' to accommodate uncertainties.
One-Time Costs: These encompass software/hardware purchases, talent acquisition, data gathering, training, and execution costs.
Recurring Costs: These pertain to subscription/licensing fees, upkeep, and hosting charges.
Example: Company A lists down its cost estimates for each of the scenarios:
Ideal Scenario (20% Probability of Happening):
Implementation (One-Time): $30,000
Data Acquisition & Preparation (One-Time): $10,000
Cloud Hosting (Monthly): $1,000
Service & Maintenance (Monthly): $2,000
Total One-Time Costs: $40,000
Total Ongoing Costs: $3,000
Most Likely Scenario (60% Probability of Happening):
Implementation (One-Time): $45,000
Data Acquisition & Preparation (One-Time): $10,000
Cloud Hosting (Monthly): $1,000
Service & Maintenance (Monthly): $2,000
Total One-Time Costs: $55,000
Total Ongoing Costs: $3,000
Worst Case Scenario (20% Probability of Happening):
Implementation (One-Time): $60,000
Data Acquisition & Preparation (One-Time): $10,000
Cloud Hosting (Monthly): $1,000
Service & Maintenance (Monthly): $2,000
Total One-Time Costs: $70,000
Total Ongoing Costs: $3,000
Step 4: Gauge Benefits
To estimate the benefits of your project, consider the potential impact on the key metrics identified in Step 2. This could include factors such as increased revenue, cost savings, improved customer satisfaction, or productivity gains. Quantify these benefits in measurable terms and assign a value to each one. It is important to base these estimates on realistic assumptions and data to ensure accuracy. Intangible benefits such as improved employee satisfaction, operational flexibility, and brand enhancement might be difficult to quantify, but you can assign a relative importance to each or rank them.
Example: Company A estimates the benefits of its Gen AI project under different scenarios:
Ideal Scenario:
Increase in qualified sales leads: 50% (which translate to 10% increase in sales, $100,000 annually)
Time to value: 6 months
(Intangible) Better employee satisfaction due to reduced cold outreach.
(Intangible) Improved outreach to potentially viable but untested segments.
Most Likely Scenario:
Increase in qualified sales leads: 30% (which translate to 6% increase in sales, $60,000 annually)
Time to value: 9 months
(Intangible) Better employee satisfaction due to reduced cold outreach.
(Intangible) Improved outreach to potentially viable but untested segments.
Worst Case Scenario:
Increase in qualified sales leads: 20% (which translate to 4% increase in sales, $40,000 annually)
Time to value: 12 months
(Intangible) Better employee satisfaction due to reduced cold outreach.
(Intangible) Improved outreach to potentially viable but untested segments.
Step 5: Consolidate Costs & Benefits for Each Scenario
Establish a project lifecycle (commonly 3-5 years) and then compute the Total Benefits and Total Costs for each scenario.
Total Benefits = One-Benefits + Ongoing Benefits x Project Lifecycle
Total Costs = One-Time Costs + Ongoing Costs x Project Lifecycle
Example: Company A calculates the Total Costs and Total Benefits for the project under different scenarios:
Ideal Scenario:
Total Benefits: $100,000 x 3 = $300,000
Total Costs: $40,000 + $3,000 x 36 = $148,000
Most Likely Scenario:
Total Benefits: $60,000 x 3 = $180,000
Total Costs: $55,000 + $3,000 x 36 = $163,000
Worst Case Scenario:
Total Benefits: $40,000 x 3 = $120,000
Total Costs: $70,000 + $3,000 x 36 = $178,000
Step 6: Derive Expected ROI
Now, assign the probability of each scenario happening, and combine the Total Costs and Total Benefits for each scenario into Expected Total Costs and Expected Total Benefits respectively. Finally, you can calculate the Expected ROI.
Expected Total Costs = (Probability of Scenario 1 Happening x Scenario 1 Total Costs) + (Probability of Scenario 2 Happening x Scenario 2 Total Costs) + (Probability of Scenario 3 Happening x Scenario 3 Total Costs) + …
Expected Total Benefits = (Probability of Scenario 1 Happening x Scenario 1 Total Benefits) + (Probability of Scenario 2 Happening x Scenario 2 Total Benefits) + (Probability of Scenario 3 Happening x Scenario 3 Total Benefits) + …
Expected ROI = (Expected Total Benefits - Expected Total Costs) / Expected Total Costs x 100%
Example: Company A calculated the Expected ROI for the project to be 18% as per calculation below:
Expected Total Costs: (20% x $148,000) + (60% x $163,000) + (20% x $178,000) = $163,000
Expected Total Benefits: (20% x $300,000) + (60% x $180,000) + (20% x $120,000) = $192,000
Expected ROI: ($192,000 - $163,000) / $163,000 x 100% = 18%
Step 7: Stack Up Your Alternatives
Once you've detailed the costs and benefits, comparing your project to other investment alternatives (or doing nothing) is essential. The intangible benefits and costs (e.g. time to value) should also be compared alongside the Expected ROI with the other alternatives. By juxtaposing the project's Expected ROI with other options, businesses can make informed decisions about where to allocate resources.
Example: Company A calculated that the Expected ROI for the project is 18% over three years. Comparatively, investing in a new marketing campaign might yield a 10% ROI. In such a case, the project seems to be a better option.
Step 8: Keep Tabs on Outcomes
To ensure the accuracy of your ROI estimates, it is crucial to monitor the progress and outcomes of your project. Regularly review and update the key metrics identified in Step 2 to track the actual performance of your project. This will allow you to make any necessary adjustments and evaluate the effectiveness of your investment. Additionally, consider conducting periodic assessments to identify any potential risks or opportunities that may arise during the project's lifecycle.
With the above, you now can estimate the ROI of your AI/ML or Gen AI project in a systematic approach. This is a very crucial step in the implementation process which will determine the success or failure of the project. What other challenges do you face when estimating the ROI of your AI/ML project? Let me know in the comments below.
Comments