## Monte Carlo Simulation challenges. Level of details challenge.

As discussed in the previous posts, applying the Monte Carlo Simulation (MCS) CORRECTLY for a large project may take many hours. As a workaround, some risk consultants recommend running simulations based on a high-level schedule, known as a control schedule and a ‘Level 3’ schedule.

The results of MCS performed based on the detailed schedule and high-level schedule are likely to be different, sometimes significantly different! There are a few reasons for that. In this post, we review one of them: the missing level of complexity.

Work packages are the smallest unit of work that a project can be broken down into when creating your Work Breakdown Structure.

In this post, the term ‘Work package’ (WP) represents ‘A group of activities’. It could be the smallest unit of work or a group of units.

When WP in a high-level schedule is represented by a single activity (or a few activities), the schedule misses a level of complexity of each WP. Without this information, the result of the simulation could be dramatically different.

Let’s review a simple example: A project has two WPs. Each of them has five 10-day activities, but in 1st WP all activities are consecutive, and in 2nd WP, all activities are concurrent.

If we decide to create a high-level schedule, we could simplify it to two linked ‘50 days’ and ‘10 days’ activities. The overall duration of both schedules and the logic between WPs will be the same.

Let’s assume each activity in our project has +/-2 days of uncertainty, which is +/-20%.

Detailed schedule with uncertainties:

High-Level schedule with uncertainties:

Now, we can calculate the probability of delivering this project in 60 days (the deterministic Most Likely scenario) and use both schedules as the base for the simulation.

We are going to use a triangular distribution. As explained in the previous posts, the Beta / PERT distributions often better represent real-life probability, but they are unique to each risk analysis tool. I want to be sure that this experiment can be replicated with other risk analysis tools.

Also, due to the randomised nature of the Monte Carlo Simulation method, the result of any new simulation is always different. The more iterations performed, the less volatile results will be. I set the Spider Project to 50,000 iterations to minimise the volatility.

## Project Probability Distribution

The performed simulation based on the High-Level schedule gives us the following probability distribution:

The performed simulation based on the Detail schedule gives us a different result:

There is 49.8% probability that this project could be completed in 60 days if the High-level schedule was used for this analysis and 29.2% probability if it was performed based on the Detail schedule.

However, project activities in both cases are going to be executed the same way, and the probability  should be the same. To understand such an anomaly, we need to go to the next level and analyse the probability of each WP separately.

## 1st WP probability distribution

The probability distribution of the first WP based on High-Level schedule:

The probability distribution of the first WP based on Detailed schedule:

Uncertainties in a sequence with ‘Finish-to-Start’ dependencies compensate for each other, so the result of the First WP simulation is roughly the same, regardless of the applied schedule. It is around 50%.

## 2nd WP probability distribution

As the second WP has only concurrent activities, the probability distribution curves are dramatically different.

Based on the Detailed schedule it is only ~3% chance that ALL activities could be completed in 10 days. If any activity is late (>10 days), the WP will be late.

In the High-level schedule, this level of complexity is lost, and the probability complete the WP in 10 days is 50%.

Probability distribution of the second WP based on the High-level schedule:

Probability distribution of the second WP based on the Detailed schedule

There is a big difference between 3% and 50%! Depending on which result is presented to project decision-makers, it may lead to different actions.

## Level of complexity

The analysed example above may look extraordinary. Not all projects have five linked concurrent activities. However, if we analyse any schedule, we are likely to find that there are days with more than five planned activities. All these activities are logically linked via direct or indirect logic. If these activities have large Total Floats, the effect of uncertainties should be immaterial. However, if they are on a project critical path or near a critical path, the effect will likely be noticeable.

Without Quantitative Risk Analysis (QRA), it may not be possible to understand the effect of uncertainties on project delivery dates.

Logic in real-life schedules is much more complex. WPs have other than ‘Finish-to-Start’ types of dependencies, technical and resource constraints, activity and resource calendars, lags and leads. These factors impact the level of complexity of each WP.

Also, logic between WPs is more complex than just a ‘Finish-to-Start’ dependency between the last activity in one WP and the first activity in another WP. On top of this, there could be more than one dependency between WPs, and these dependencies could be somewhere inside each WP, not between the end and start points:

Example of complex logic between WPs

## The volatility of Critical Path

Another challenge is to demonstrate a Critical Path in a high-level schedule. A WP may or may not be on a critical path. If it is, it doesn’t mean that all activities in the WP are critical. The critical activities in the WP could be somewhere inside of WP. There also could be many critical fragments.

A scheduler may summarise critical and not critical activities in a high-level schedule. Unfortunately, there is no guarantee that the critical path is going to be stable during the life of the project. The volatility of the critical path means that a high-level schedule has to be constantly redeveloped.

## Detailed Schedule

The described challenge is relevant to detailed schedules, not just to Control schedules.

There are no commonly accepted rules that define the correct level of detail for each schedule. While there are some recommendations, practically the level of detail depends on portfolio standards, the experience of the master schedule, the capacity of the scheduling team, applied schedule and risk analysis tools, and some other factors.

Two schedules developed for the same project by two different project teams are unlikely to be identical. The difference may look immaterial but may have a significant impact to the results of Monte Carlo Simulation.

The probability of delivering this project based on the ‘Most Likely’ durations is only 29%. This example demonstrates ‘Schedule duration paradox’:

Schedule based on the most likely activity durations doesn’t have the most likely duration

This is one of the factors why projects are late. When project SMEs provide the ‘Most Likely’ estimations for each schedule activity, and a scheduler uses them to develop a deterministic (Critical Path Method) schedule, project stakeholders often assume that such a schedule has ‘most likely’ delivery dates. Calculated dates are used as project commitments. However, due to the complexity of logic and parallel streams, the probability of meeting these commitments is likely to be less than 50%. Sometimes significantly less.

Monte Carlo Simulation is one of the Quantitative Risk Analysis (QRA) methods that help to calculate achievable dates, but it has to be applied correctly. A high-level schedule doesn’t have the same level of complexity as it is in a detailed schedule. It impacts the project probabilities calculated based on the Monte Carlo Simulation method. The difference could be sufficient to drive the project in the wrong direction.

There are many other important MCS challenges. I will continue “unwrapping” them one by one in my future posts.

#### Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma

## Monte Carlo Simulation Challenges. Critical Path Volatility Challenge.

The ‘3 points estimation’ known as a PERT methodology is relatively simple to use for cost, but for the schedule, it’s more complicated. The fact that some activities are on the critical path means that the volatility of those specific activities, as opposed to the set of all project’s activities, can make a big difference for Monte Carlo Simulation.

## Risks and Critical Path

It is well accepted that risks are measured based on the probability of the event (or condition) occurring and the consequences.

Risk Exposure = Probability (Event) * Consequence

The same risk may have different consequences for Time, Cost, Safety, Reputation and other project objectives.

• The time consequence is highly dependent on the criticality of impacted activities. If impacted activities are on the critical path# [including delay], the project is going to be delayed. Otherwise, it will impact some parts of the project but not the overall delivery date(s).
• Risk consequences are not always a stable parameter. Some of them depend on project conditions.
• Risk consequence has to be measured based on future conditions when the risk is likely to occur, not the current situation.
• When the risk is likely to occur project critical path may be different from the current critical path. It could be due to the impact of project uncertainties, other risks, scope changes or any other reasons.
• At that time, there will be only two possibilities: the critical path impacted or not. In other words, the probability (CP) that the risk has a time impact will be 0% or 100%. Do not mix with the Probability of the event, which is not dependent on the critical path.

However, when the risk is assessed, it is not always possible to predict if the impacted activities are going to be on the critical path or not, so the probability (CP) could be different from 0% and 100%.

## Probabilistic vs Deterministic Delivery Model

If the Monte Carlo Simulation is performed based on a dynamic PROBOBALISTIC model (aka schedule) that includes data uncertainties and “conditional branches ”, the critical path will be recalculated during each calculation, and only the event probabilities need to be assessed prior to the simulation.

PROBABILISTIC project delivery systems, like Spider Project, could additionally calculate the probability of each activity being on the critical path, which is known as ‘activity criticality index’.

However, if the Monte Carlo Simulation is performed based on a DETERMINISTIC schedule (developed in Primavera, Microsoft Project or another CPM-based scheduling tool) and risk data is collected in a separate risk register, the probability that impacted activities are going to be on the critical path (CP) has to be assessed manually. The accuracy of such an evaluation would be significantly different from calculated probabilities and it will have following effect the results of the Monte Carlo Simulation.

Some risk management tools provide users with an option to define if the critical path is going to be impacted or not with a binary option (Yes/No). Users potentially could assess it’s based on the current critical path, but, with some rare exceptions, it is impossible to say whether the future critical path is going to be impacted or not as it depends on other materialised risks.

Some project Risk consultants just assume that the Critical Path is going to be the same

(Probability (CP) =100% or 0%) and only assess event probability even if the previous similar projects had critical path volatility.

## Risk Exposure

If the (time) consequence depends on the Project Critical Path, or to be precisely the probability that impacted activities are going to be on the critical path when risk is materialised, Risk Exposure also depends on this parameter:

Consequence (Net) = Probability (CP) * Consequence (Gross)

So:

Risk Exposure = Probability (Event) * Probability (CP) * Consequence (Gross)

or

Risk Exposure = [ Probability (Event) * Probability (CP)] * Consequence (Gross)

## Cost Impact

So far, we have discussed only time exposure. What about the cost exposure? Is it also dependent on the critical path? Yes, but in a different way.

Risk cost consequence has two parts: direct cost impact and cost of delay.

Direct cost impact only depends on the Probability (Event) and the cost of delay only depends on Probability (CP).

## Summary

The volatility of the Critical path means that risk consequence is not a stable parameter and depends on future project conditions when the risk is likely to occur. It is hard to accuratelly predict which activities are going to be on the Criticla Path at that time.

The results of Time and Cost probability distributions received based on deterministic the CPM-based schedule and predefined risk data could be misleading and should be used for project decisions with caution.

Risk consequences have to be calculated in each simulation. Dynamic probabilistic model with fully integrated risk data is required for reliable Monte Carlo Simulation.

#### Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma

## Risk vs Uncertainty

At project management conferences, LinkedIn discussions and even in one of the most recognised Project Risk Management book, I’ve found that uncertainty is just a risk with 100% probability. Not sure what is the origin of this statement, but a risk with 100% probability is an issue, not an uncertainty.

Managing project risks and uncertainties is essential for mature risk management and successful project delivery. Lack of understanding differences between these two concepts creates serious project planning and control issues and often results in project delays and cost overruns. In this post, we will review what is common between these two concepts and where the key differences are.

## Project Risk Management Standards

Risk and Uncertainties are interrelated. Risk arises out of uncertainty and generates uncertainty.

Project Management Institute (PMI) and Association for Project Management (APM) used these terms widely in their project management and risk management standards. Both organisations suggest dividing the term Risk into two terms: Risk Event and Overall Project Risk. PMI and APM don’t provide definitions of Uncertainty but mention that the source of uncertainties is complexity, ambiguity, and volatility.

### Project Event

Risk Event Definition by PMI:

Risk is an uncertain event or condition that, if it occurs, has a positive or negative impact on at least one project objective. (The Standard for Risk Management in Portfolio, Program, and Projects)

Risk Event Definition by APM:

A risk event is an uncertain event or set of circumstances that, should it occur, will have an effect on achievement of one or more of the project’s objectives. (Project Risk Analysis and Management Guide)

### Overall Project Risk

Overall Project Risk Definition by PMI:

Overall project risk is the effect of uncertainty on the project as a whole, arising from all sources of uncertainty. This includes individual risks and the exposure to the implications of variation in project outcome, both positive and negative. Overall risk is often a function of complexity, ambiguity, and volatility. (The Standard for Risk Management in Portfolio, Program, and Projects)

Overall Project Risk Definition by APM:

Overall project risk is joint effect of risk events and other sources of uncertainty. Project risk results largely from the accumulation of a number of individual risk events, together with other sources of uncertainty to the project as a whole, such as variability and ambiguity. (Project Risk Analysis and Management Guide)

Unfortunately, apart from the definitions, both standards don’t provide any methods and techniques to measure and control Overall Project Risk. There is no common acceptance among project and risk managers on what it is the “beast” and how to deal with it.

Success Driven Project Management (SDPM) suggest dividing risks into two major categories: uncertainties and events, where uncertainty is a property of project parameters. Based on SDPM, changes caused by risk are discreet events, while changes caused by uncertainties are continuous.

Let’s look at risks and uncertainties more closely.

## Risk

Less formally, risk is defined as the situation of winning or losing something worthy.

In projects, stakeholders are interested in understanding how such situations may impact project objectives, usually represented as the delivery of outputs (or critical deliverables) to the committed dates with the agreed budget. Depending on the type of project, objectives may also include quality, benefits, reputation and other important aspects of the business.

“Worthy” means that while risk events may impact other aspects of the project, these are the only important ones in the context of the overall delivery.

For example, a project team may raise a risk that a particular activity might be running late or next month’s forecast could not be met if a risk event materialised.
While this information still may be necessary for some stakeholders, a project risk usually is raised in a broader context:

• Does it mean that if the activity is late, the project will also be late?
• Does it mean that if the next month’s actuals are more than was forecasted, the project will be over budget?

‘Risk impact’ is a dynamic characteristic, not a stable parameter. A risk may have no impact on the project objective now, but the situation may be different tomorrow, and the risk would become the most critical risk in the project. Without Quantitative Risk Analysis methods, it is not easy to identify such changes timely.

There are many books and risk management standards with recommendations for risk control. While uncertainties are also considered in these books and standards, I think this area of project risk management is less developed and definitely less controlled.

## Primary Uncertainty

Uncertainty is a situation where multiple alternatives result in a specific outcome, but the probability of the outcome is not certain.

Any project has hundreds, if not thousands, of situations when the outcome is uncertain. However, any project has quite limited project parameters with uncertainties.
Examples of parameters that may have uncertainties:

• Volume of Work
• Resource Productivity Rate
• Resource Quantity
• Resource Calendar
• Activity Calendar
• Material Quantity
• Activity Duration
• Lag Duration
• Material Cost
• Unit Cost
• Currency rate
• Resource Rate
• Inflation Rate

There are examples of bottom-level uncertainties, also known as base, or primary uncertainties.

• Most, but not all project parameters, have primary uncertainties.

Example:
Office calendars, compared to construction calendars, don’t depend on weather conditions.

• Data collection challenges

It is easy to collect primary uncertainty data as each refers to only one project parameter. Three-point estimation is a common practical approach. Usually, the critical constraint for data collection is not the information itself but the lack of opportunity to store it in a project management system. Not many of them are designed to store and apply primary uncertainties to calculate project delivery plans. Spider Project is an excellent example of such a system when primary uncertainties are stored as corporate norms and used to calculate project delivery plans with probabilities to achieve committed project objectives: dates, costs, benefits, etc.

I believe there is a lack of recognition of the benefits of capturing and applying uncertainty data for the reasons explained below. Many organisations with such experience are in non-English-speaking countries and usually don’t publish results in recognised PM Journals.

## Secondary Uncertainties

The outcome of primary uncertainties may result in uncertainties of other project parameters, especially activity duration and cost uncertainties, which are secondary or middle-level uncertainties.

• The secondary uncertainty is the result of one or multiple primary uncertainties.

Example:

An activity has uncertainty in the volume of work and productivity of assigned resources. Uncertainty in the duration and cost of the activity is the result of both uncertainties.

• Activity duration and cost uncertainty may or may not depend on other primary uncertainties.

So, it could be a primary or secondary uncertainty. It is very confusing, but who said that project management has to be simple?

Example:

An external project review takes 24-40 hours. It means that the ‘volume of work’ of this task is measured in hours, and in the worst-case scenario, the report is going to be completed when 40 hours of effort, even if the quality of the report needs to be compromised.

• Not all primary uncertainties impact duration and cost of activities.

Example:

‘Resource Quantity’ uncertainty may impact the order of planned activities but not in their durations and cost.

• Result of Monte Carlo Simulation (MCS) highly depends on the type of uncertainties used as a base for analysis.

Example:

Results of MCS analysis performed based on secondary uncertainties may significantly vary from the analysis performed based on primary types, as important correlations between project parameters are missing. This is another big topic to discuss, and I’m planning to cover it in a separate post under the “Monte Carlo Simulation challenges” posts.

## Project Level Uncertainties

Multiple primary and secondary level uncertainties and related risks result in probability distributions of project commitments. These distributions could be considered as ‘top-level’ or tertiary uncertainties.

Examples of top-level uncertainties include:

• Delivery date of committed output
• End of the design phase
• Cost of project

• Probability distributions could be considered in the context of individual project objective or with the relationship to other objectives.

## Risk Uncertainties

Projects usually estimate risks with probability, as a percentage of the likelihood of the event or condition to occur, and the impact estimated in days and \$. However, this information is often subjective as different project stakeholders have different opinions about the probability and impact.

As a result, a risk may have uncertainties in the probability, the impact(s) or both at the same time.

## Cost of Knowledge

Both, uncertainties and risks depend on the knowledge of estimators. Sometimes it may be possible to gain this knowledge and reduce uncertainties but usually, it is associated with extra effort and additional cost.

Sometimes it is worth spending extra effort to gain missing information that could help with project decisions, sometimes not. In other situations, the required information may not exist and any effort to clarify the information is a waste of time and money. It is another separate big but interesting topic to discuss.

Portfolios with advanced project management collect historical data and apply more accurate estimations to outstanding delivery and future projects. It is important to mention that, if possible, historical data has to be collected with primary types of uncertainties. If only secondary (activity level) uncertainties are collected, the critical context would be missing, and the historical data could not be applied correctly.

Example:

An activity has uncertainty in duration due to the uncertainty in the productivity of the planned resource: new bulldozer. The uncertainty is clarified during project execution and, in theory, next projects could use this information to reduce volatility. However, if only duration uncertainty was documented and another bulldozer with a different productivity rate is assigned to perform the same type of work, the historical data would be misleading as the critical context is missing to understand that this type of work has a different duration now.

## Impacts of Risk and Uncertainties

• Not all uncertainties have an impact to project objectives.
Sometimes, under current conditions, regardless of the result of the uncertainty, it would be no impact to project commitments.

Example:

Next month, a project must pay for materials in a different currency. There is uncertainty about next month’s currency rate. This uncertainty could be an opportunity, treat or has no impact to project cost at all. It depends on:

• Which way does the currency rate go? Maybe it is an opportunity, not a threat?
• Currency rate the project forecasted. Maybe it was forecasted based on the worst rate?
• Conditions in the contract. Maybe the project has a fixed currency rate?
• A risk is always a threat or opportunity from one perspective (time, cost, etc). Uncertainties could be a threat and an opportunity to the same perspective simultaneously.

Example:

There is uncertainty in the volume of work that will impact the duration and cost of the activity. The uncertainty was estimated as a range (100 – 150 m2). If the project plan has 125 m2 as a base, it would be an opportunity to complete the activity ahead of schedule or a threat of delaying the completion.

• The same risk could be an opportunity for one perspective and a threat for another.

Example:

The bulldozer with certain productivity is planned to be assigned to work. There is a 20% chance that this bulldozer is not going to be available, and the project needs to hire another more expensive but also more productive bulldozer. If the risk is materialised, it will be a threat to the project cost, but also an opportunity to project delivery date (if planned work is on the critical path). So, this risk is a threat to cost and an opportunity to time. When a project uses a risk register and classifies all risks as a threat or opportunity, the mitigation actions may solve some potential issues but generate others.

## Creeping Normality

Primary uncertainties impact projects via very small changes, but commutatively, they may have a large effect. Such an effect could be even more severe than the impact caused by project risks.

Project control teams usually have less control over activities with sufficient contingency. Positive and negative changes caused by uncertainties slowly and in a non-linear way “eat” available contingency, and when suddenly activities start to be critical and impact project delivery dates, it is already too late to apply mitigation actions. The contingency already has been spent. It is even more applicable to project cost when a lot of minor cost increases slowly “eat” cost contingency.

I think this could be due to “Creeping Normality” cognitive bias, also known as “Landscape amnesia”.

Creeping normality is a process by which a major change can be accepted as normal and acceptable if it happens slowly through small, often unnoticeable, increments of change. The change could otherwise be regarded as remarkable and objectionable if it took place in a single step or short period.

It creates an illusion that ‘risk events’ is the biggest factor in project delay and cost overrun when, in reality, changes caused by uncertainties have a bigger effect caused by project risks.

Projects that document uncertainties and compare them with actuals can notice “patterns” and apply mitigation actions before it is too late.

## Relationship Between Risks and Uncertainties

While it is obvious that Risks and Uncertainties are interrelated, the relationship between them is not always obvious.

• Uncertainties in one part of a project may have a negative (or positive) impact on risks in another part of the project.

A risk associated with activities on the critical path is likely to severely impact project delivery dates. Changes caused by uncertainties in another part of the project may have an indirect impact on the risk activities total float and the risk mitigation actions should be changed as well. However, as projects usually don’t control uncertainties, it is hard to notice when the impact of risk is changed. In the future post, I am going to demonstrate this effect with a more detailed example.

## Summary

Risk and uncertainties are important elements of Project Risk management. They are interrelated but have critical differences that are not always understood correctly. While Project risk management standards recognise the importance of management of uncertainties and risk, projects across the Globe often don’t have mature control over uncertainties.
For project success, it is not sufficient to only control risks. Mature Risk Management also has to include the management of uncertainties.

#### Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma