Before we start our review of Monte Carlo Simulation (MCS) Challenges, we have to clarify what MCS is and how it is applied in project management.

To be useful a model has to be dynamic and represent the true relationships between inputs and outputs.

In the project management model, inputs and outputs are:

Project Delivery Plan used a model for project planning and delivery.

I don’t like the term “schedule” in the context of MCS. For many project practitioners, this term is associated with Time management and it is only one component of project delivery.

Apart from the Golden Triangle, which includes:

• Scope
• Time
• Cost

a project delivery plan also integrates:

• Risks (including uncertainties)
• Resources (People, Equipment & Materials)
• Benefits

Full integration means that changes in one parameter are reflected in other parameters.

A Monte Carlo Simulation is performed to identify: 

• Project risks that require maximum attention
• Project targets that can be met with sufficient probabilities
• Contingency reserves that must be created to meet project targets
• Current probabilities of meeting project targets
• Resources required for the reliable achievement of project targets
• Project activities that require maximum attention

MCS is only one of the methods of Quantitative Risk Analysis (QRA). In this and future posts, if not specified, under QRA I mean the MCS method. If there is any interest, we can discuss other QRA methods later.

In project management Monte Carlo Simulation works the following way:

1. Integrated logically driven Project Delivery Plan developed and assessed.
2. People enter three estimates of initial data that are uncertain (optimistic, most probable and pessimistic) and define what probability distribution each uncertain parameter has.
3. Risk events are included in the project risk model with their probabilities and impacts.
4. The software calculates the model and accompanying parameters over and over, each time using a different set of initial data in accordance with their probability distributions. The number of iterations is usually defined by the risk management software user. Usually, this number is measured in thousands of distributions.

As the result, we get the distributions of possible outcome values.

This process has one important missing step: “Add corresponding corrective actions”. We will cover it later in a separate post.

Any process may look simple at a high level but to get the desired result detailed instructions, tools and practice are often required. Everyone, who tried to cook knows that cooking is more complex than “Prepare ingredients, mix them, apply temperature processing, and serve”. So, why do we accept the idea that the application of the scientific method in complex and complicated environments could be as simple as: “Prepare model and estimations, add risks and uncertainties, simulate, and report”? There are a few reasons for that:

1. A burned dish can damage a planned dinner and the reputation of the chef, but decisions made based on misleading project risk analysis are likely to impact someone else, not the person who runs the analysis, or even a project manager who is responsible for project delivery.

So many times we have been told that communication skill is the most important skill in project management. Now we have many project managers that could perfectly explain why their project is late and over budget, but don’t know how to develop an optimised project delivery plan and run quantitative risk analysis. Adding even a small financial incentive based on achieved results would motivate project managers to learn advanced project management methods, including MCS

Another problem is a disconnect between time, cost and benefits. Until project delivery plans will not have integrated with planned benefits, this challenge will always be there. Such integration helps to drive project decisions and understand how these decisions impact outcomes and benefits, not just time and cost. MCS also could be used to calculate the distribution of financial (NVP & ROI) and non-financial metrics.

An example of this kind of integration was discussed in this post:

https://saluteenterprises.com.au/project-success-criteria/

2. If companies that promote their MCS tool or QRA training say that MCS requires many hours of effort, no one will buy their tool or service.

There are four levels in the Conscious Competence Learning Matrix and companies that promote the MCS methods can help us with moving from ‘Unconscious Incompetent’ to ‘Conscious Incompetent’, but reaching the next, ‘Conscious Competent’ level will require much more effort to grow deep knowledge of how the MCS method works, how to develop a reliable project delivery plan and which features in risk analysis tools are important.

3. Popularity of QRA is growing. More and more companies and projects include requirements to run QRA periodically and even specify that it must be done with the MCS method. Unfortunately, well too often the QRA is performed only to “tick the box” and the result of the QRA doesn’t drive any project decisions.

There is a simple test to identify misuse of the MCS method. Ask the Project Manager about the current probabilities of meeting project targets, which is one of the outputs of the MCS process.

4. Precision creates an illusion of accurate estimation. MCS is a perfect tool to play a guesstimation game.

A knife could be used to cut cooking ingredients or as a weapon. MCS is also used to justify desired or already made decisions, not to calculate required contingencies and identify critical risks and activities. In that case, it is not critical if the MCS method was applied correctly or not, and which tool was used for the simulation.

The market is full of Monte Carlo Simulation software that primarily serves this purpose. The ability to generate colourful reports seems to be the most critical feature of this tool. If your project or portfolio runs MCS to satisfy someone’s need to say that the analysis is performed or it is performed to justify the desired decision, you may not worry about other MCS challenges.

On the other hand, for business owners and accountable project managers, the ability to run their own independent project quantitative risk analysis based on a reliable model and risk analysis tool can save money, reputation and even lives. ‘Covid-19’ projects are a great example of it.

Ones Niels Bohr, the Nobel laureate in Physics and a father of the atomic model, said: “It is difficult to make predictions, especially about the future”. Any project model is just a proxy and the result of risk analysis never could be precisely accurate. So, why are we still trying to make predictions then? Because even not 100% accurate forecasts still could be used for our decisions! However, misleading forecasts are dangerous, as they drive wrong decisions.

The challenge is to understand where to draw a line between “inaccurate” and “misleading” when MCS analysis is performed.

Probably you have already heard the famous quote of George Box: “All models are wrong, but some models are useful”. I prefer the full version:

All models are wrong, but some models are useful. So, the question you need to ask is not ‘Is the model true?’ (it never is) but ‘Is the model good enough for this particular application?’

In the application of project management, we can rephrase it as:

In the series of future posts, we will try to find the answer to this question.