Monte Carlo Simulation Challenges. Data collection Challenge.

What do the terms optimistic, pessimistic, and most likely really mean?
Because they are very different for different people, when one estimator says pessimistic, that person is thinking the technology might prove to be a bit more difficult than expected and thus take 10% longer. Another person in the team may assume that ‘pessimistic’ means that the work might be interrupted by an earthquake, tsunami, and nuclear meltdown simultaneously (even if it has not happened in the past).

Popular scheduling tools, like Microsoft Project and Primavera, don’t have native features to capture ‘3 points estimations’, so planners usually don’t capture estimations provided by Subject Matter Experts. A risk manager or a scheduler applies a range based on a single deterministic estimation instead. This approach is based on two dangerous assumptions:

• • • All provided estimations are “Most Likely” estimations.

When SMEs are forced to provide a single estimation, they have to decide on how much contingency to include in the estimation. Some estimations in schedules have “optimism bias”, others, on the opposite, are too conservative.

• • •  All activities have the same level of uncertainty.

There are many reasons why some of the activities may have dramatically different uncertainties: new technology, new equipment, new contractor, etc.  

Probabilistic scheduling tools, in the opposite, give an opportunity to capture activity durations as ‘3-point estimations’. Spider Project, arguably, the most advanced project delivery tool gives the opportunity to capture any type of initial data as 3 points: durations, cost, volume of work, resource productivity, lags, etc.

 Primavera and Microsoft project have custom fields that could be used to capture at least the duration of activities. Then weighted durations could be calculated. Both tools have built-in formulas. PERT or any other method could be applied for that.

Microsoft Project used to have built-in 3 points estimation fields but it’s current version doesn’t have this feature any more.

The Monte Carlo method works well when a lot of historical data is available. Unfortunately, the unique nature of projects means that likely historical data is going to be limited. However, even a small volume of historical data could be beneficial and portfolios have to implement methods to collect historical project data in a systemic way.  

 When historical data is not available the only way to collect estimations is to interview Subject Matter Experts (SME) and collect subjective information. However, it needs to be done in a smart way.

 It’s always better to ask a SME to provide an optimistic estimation first. Then ask the interviewee to think about the worst-case scenario and only after that provide the most likely estimation.

The trick is when the interviewee thinks about different possibilities, the most probable estimation is likely to be more accurate.

I found it extremely useful to collect data as a range even though I am not going to use it for Quantitative Risk analysis. It actually doesn’t take much longer to do so but gives me an opportunity to identify hidden risks. After all estimations are collected, some activities may have an abnormal range, comparing to the rest of the activities. It’s worth discovering the underlying reasons as there is often a story behind each abnormal range. Some of these stories get converted into risks or even issues.