Monte Carlo Simulation Challenges. Simulating the true source of uncertainty

Usually, duration and cost uncertainties are used as a base for the Monte Carlo Simulation (MCS). However, the duration and cost uncertainties are the results of other types of uncertainties.

Any project has Primary, Secondary and Project Level Uncertainties.

Duration and cost of any activity are secondary uncertainties that are the result of the true primary uncertainties. The primary uncertainties include Volume of Work, Resource Productivity Rate and Cost, Skills Quantity, Resource & Activity Calendars, Material Quantity and Cost, etc.

Is it important for the MCS analysis? Why, after the probabilities to meet project objectives (time & cost) are calculated, we can’t analyse the sources?

Such analysis must be done BEFORE the simulation as the result of the MCS depends on what is simulated: primary or secondary uncertainties.

Let’s consider two simple projects consisting of a single activity:

Project 1
• The volume of work for this activity is 800 units
• Activity duration depends on the assigned resource’s productivity, with the optimistic estimate of 1.25 u/h, the most likely estimate being 1 u/h, and the pessimistic estimate being 0.8 u/p.
• The workday is 8 hours
• Uncertainty Type: Resource Productivity (-20%, +25%)

Project 2
• The volume of work for this activity is 800 units
• Activity duration depends on the volume of work, with the optimistic estimate of 640 units, the most likely estimate being 800 units, and the pessimistic estimate being 1,000 units.
• The workday is 8 hours
• Uncertainty Type: Volume of Work (-20%, +25%)

Both projects have the same duration uncertainty: 80 /100 /125 days, and the uncertainty range: -20% / +25%. If the duration uncertainty is used as the base for MCS, the probability of delivering the project in 100 days will be the same for both projects. If the true uncertainties are used instead, the duration uncertain the result will be different.

Let’s review three scenarios simulating secondary uncertainty (duration) and primary uncertainties (Resource productivity and Volume of work).

If the work is represented by a single activity with three estimations (80d, 100d, 125d) and the activity duration is used as the base of uncertainty, as the result of MCS would be as following:
• 100d = 43%
• P75 = 108d
• P90 = 114d
• Distribution: Triangular
• Number of iterations: 50,000

If the work is represented by a single activity with three estimations (80d, 100d, 125d) and the resource productivity (0.8, 1, 1.25) is used as the basis of uncertainty, the result of MCS would be as follows:

• 100d = 55%
• P75 = 106d
• P90 = 112d
• Distribution: Triangular
• Number of iterations: 50,000

If the work is represented by a single activity with three estimations (80d, 100d, 125d) and the volume of work (640, 800, 1000) is used as the basis of uncertainty, the result of MCS would be as follows:

• 100d = 44%
• P75 = 108d
• P90 = 115d
• Distribution: Triangular
• Number of iterations: 50,000

Activity Duration Formula

Why is the result of Scenario 2 different from Scenarios 1 and 3?
As mentioned, activity duration depends on many parameters. These parameters impact the probability of completion in different ways. It is impossible (at least for now) to create a complete formula that shows how all primary uncertainties impact activity duration as some parameters (resource quantity uncertainty, etc.) also depend on other activities. Still, the simplified formula that only includes reviewed scenarios is:

As we can see above, it does matter if the Primary uncertainty is the Numerator or Dominator, as it impacts the distribution curve!

In the first case (scenario 2) the probability of achieving the Most Likely duration is >50%, and in the second case (scenario 3), it is <50%.

We reviewed a simple example with the uncertainty of one parameter to explain the challenge. It is clear that just for one activity without specialised risk simulation software that supports the simulation of primary uncertainties, it is very hard to understand how primary uncertainties impact activity duration uncertainty.
The real-life relationship is more complex as:

What would the Optimistic & Pessimistic duration and the distribution curve be if an activity has uncertainties in many primary parameters? Is it impossible to understand it accurately?

It is very hard, and there is a good chance that even experienced estimators make costly mistakes when they provide 3-point estimations.

Do demonstrate it before you read the result guess:
If the activity has both (Volume of work and Productivity) uncertainties,
– What would be the probability to complete the activity in 100 days?
– What would be the optimistic & pessimistic durations?

There is 47% chance to complete this activity in 100 days.
The most likely duration is 96 days and the Optimistic – Pessimistic range is: -32% , +52% (!).
Have you noticed that the curve is starting to get a “fat tale”? Take note, I will go back to the “fat tale” anomaly in future posts.
This was a simple example, as all required information was given  (even preliminary calculations!). Do you have this right? Probably not.

We have simulated only one activity. Projects have thousands of them. How does this challenge impact project-level uncertainties? This is not an easy question that requires separate deep analysis. However, practically we can see that:

Let’s review one more scenario to demonstrate that.

Assume the scheduler for this project decided that each activity has to be no longer than ten days and converted the schedule to 10 linked activities. Visually nothing is changed. The project has the same duration and uncertianties. Has this impact the probablity? 

If the work is represented by ten 10-day activities with three estimations (8d, 10d, 12.5d) and the activity duration is used as the basis of uncertainty, the result of MCS would be as follows:
• 100d = 28% (!)
• P75 = 104d
• P90 = 105d
• Distribution: Triangular
• Number of iterations: 50,000

It does not look logical. 

The project duration uncertainty depends on uncertainties of activities. However, as they are linked in a sequence, uncertainties compensate each, and simulation shows that only 105 days are required to achieve 90% probability.
The durations of all ten activities are strongly correlated, but this correlation is different. If a resource with low productivity is available for the assignments, it will impact all ten activities similarly. If the source of uncertainty (resource productivity) is simulated instead of the result (activity duration), this problem would not exist.

If the work is represented by ten 10day activities and the resource productivity (0.8, 1, 1.25) is used as the basis of uncertainty, the result of MCS would be as following:

• 100d = 55%
• P75 =106d
• P90 = 112d
• Distribution: Triangular
• Number of iterations: 50,000

We received the same result as it was in scenario 2 when the simulation was based on resource productivity, but the work was represented by one 100d activity!

– I used the triangular distribution to make it possible for others to verify the analysis. If PERT or Long-Normal distributions are used instead the difference in the results will be even bigger.
– This challenge alone has a significant effect, but when correlated with other discussed in previous (and future) posts issues, it has accumulated effect and the results of MCS are likely to be misleading. 

• Activity duration and cost uncertainties are the results of other types of uncertainties. They called secondary and primary uncertainties.

• The primary uncertainties impact activity duration and probability of completion activity in planned duration. However, without Project Management software that supports the simulation of primary uncertainties, it is not easy to understand the relationship between primary and secondary uncertainties.

• An activity may have more than one source of uncertainty. The correlation between them is complex. In some cases, primary uncertainties may compensate for each other and in other situations, the impact may be accumulated.

• On the activity level simulating the source and result of activity uncertainties may lead to different estimates. Sometimes substantially different and even experienced estimators are likely to make mistakes in providing 3-point estimations. It is much easier to provide reliable 3-point estimations for primary uncertainties.

• Projects have thousands of primary uncertainties. If simulating the result of uncertainty rather than its source, the deviation from the correct result will be substantial to say that it is misleading, not just inaccurate.

• Most project management Monte Carlo solution tools don’t work with primary uncertainties, and the results produced based on these tools are unreliable.