Earned Schedule Metrics Challenge

Earned Schedule Metrics Challenge

It sounds like EVM is the topic of the week on LinkedIn 😊 It prompts me to
raise awareness of one popular myth that pops up whenever EVM challenges are discussed.

👺 Myth: “Earned Schedule addresses EVM methodological time management problems”.

One of LI’s comments: “A good alternative to SPI is the Earned Schedule created by Walter Lipke, while SPI makes measurements the schedule deadline in money, earned schedule makes in time (i.e. days) much more reliable and understandable.”

No, it doesn’t! Earned Schedule addresses one problem, but others are so big that time forecast is still unreliable.

🔮 Addressed challenge:
The original EVM method is based on the assumption that the remaining part of the project will be performed as it was planned on time of baseline. Walter pointed out that current progress is a better proxy and offered a set of metrics to apply the new ‘TPI – Time Performance Index ‘ proxy.

It is the right idea that the performance factor has to be considered, so Earned Schedule is an excellent addition to EVM and is a better alternative to SPI. However, it does make forecasts reliable! It doesn’t show when the project is likely to be delivered!

There are other challenges that should not be ignored.

💣 Critical path.
We can’t apply the same principle to forecasting cost and time!

A dollar spent on a project will be added to the overall cost regardless of where it was spent. However, a day of delay may or may not impact the delivery date, depending on whether the delay was on a critical path.

The completion date depends on critical path activities, not all activities.
Both EVM and ES ignore this fact.

💣 Risk profile.
Let’s assume a project progressed ideally; all works performed as per the original plan but the probability of one risk is changed from low to high. It means that if the risk impacts activities that are likely to be on a critical path, the expected completion date is changed, but both SPI and TPI can’t show that.

💣 Project stages
Projects progress through different stages, meaning different activity types drive critical paths. Each stage has its unique uncertainties, challenges, and opportunities.

💡 There are no reasons to assume that previously materialised risks will materialise again for the same project!

Even worse, a project may accelerate delivery, pushing risks to later stages of the project. Rush (to initiate/design/develop/etc.) compromising maturity & quality may help with good performance but will likely cause delays during testing.

Due to the unique nature of projects, it is impossible to predict completion dates just by analysing past performance. It must be done by analysing the remaining work and risks. Performance is a complementary factor, not the base for such analysis.

Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma

Project Resource Levelling and Risk Analysis

Project Resource Levelling and Risk Analysis

The results of Project Risk Analysis performed without considering resource limitations are not reliable.

Project practitioners are usually aware that a risk may impact cost, completion date, and resource demand. However, that there is an opposite correlation between resources and risks.

Resource limitations may impact risk consequences.

Let’s review the following scheduling quiz:

Project with resource constraints and a risk

A project has two streams of work: A->B and C->D and will be completed when both streams are completed.
The project has activity uncertainties, resource uncertainty and one risk.
Activity uncertainties
The duration of each activity is estimated as a range. Let’s consider 2-point estimations to keep it simple.
Resource uncertainty
Activities A and C require the same skill. So, two resources are needed to perform these activities in parallel. However, it is unclear if the second resource is going to be available or not.
Risk
The project has a risk associated with Activity D. Activity D will take one day longer if the risk materialises. There is an opportunity to fully mitigate the risk, but it requires additional expenses.
Project priorities and goal

The goal is to complete the project as soon as possible with the fewest expenses.
The project team needs to determine the overall duration (as a range) and decide whether it is worth mitigating the risk.

Optimistic project plan

Let’s review an optimistic scenario: each activity takes the least possible time, there are two resources ‘Resoucre 1’, and the risk is mitigated. This project takes 12 days.

Optimistic plan with the resource constraint

How could the plan be optimised if only one resource is available (to perform Activities A and C)? There are only two possible options: to either delay Activity A or Activity C.

Option 1a: delay Activity A

Option 1b: delay Activity C

It seems obvious to choose Option 1a, mitigate the risk, and complete the project in 12 days.

Pessimistic project plan

The project plan without resoucre constraints based on pessimistic durations doesn’t look dramatically different.
The critical path is the same as it was in the optimistic scenario. It still makes sense to mitigate the risk and complete the project in 14 days.

Pessimistic plan with the resource constraints

Let’s add the resource constraint to the pessimistic scenarion. There are still only two possible options: delay Activity A or Activity C:

Option 2a: Delay Activity A

Option 2b: Delay Activity C

It still makes sense to complete Activity C before Activity A. However, please pay attention to Activity D. This activity is no longer on the resource critical path, so incurring extra expenses for risk mitigation doesn’t make sense. If Activity D takes one day longer, it will not delay the project completion date. The consequence of the risk has changed!

Conclusion N1: A change in resource allocation in one part of the schedule may change risk consequences in another part.

This is a very simple example, and we can track the impact of each uncertainty on all project attributes and the overall result. Real-life projects are more complex. Even small projects have uncertainties, resource limitations and risk events. As a result, the Resource Critical Path is volatile. Without a dynamic delivery model, it is usually impossible to understand how a change in one project parameter impacts other project parameters and project objectives. 

If the consequences of evaluated risks may change after resource levelling, risk evaluation needs to be repeated after each levelling cycle. However, risk re-evaluation is a time-consuming task, and it would be beneficial if a project delivery system could automatically identify changes in the risk profile after re-levelling is performed. 

 

Quiz answers

How long does it take to deliver this project?

If the delivery plan is optimised and used to drive project decisions, it takes 12-16 days, depending on uncertainties. However, wrong decisions can extend the project to 20 days (25%).

Should the risk be mitigated?

This question is not as simple as it may sound and has no YES/NO answer.

Depending on uncertainties, the risk may impact the overall project duration and need to be mitigated, or in some circumstances, the mitigation would not bring any benefits but rather would cause extra expenses.

We found two answers based on edge-case scenarios:

– The optimistic scenario: definitely mitigate risk.

– The pessimistic scenario: definitely not mitigate risk.

These two extremes give an an opposite answer. Optimistic and Pessimistic scenarios have a nearly zero chance of occurring. The actual progress will be somewhere in between these two extremes.

Additional information is required to enable a data-driven decision .

Risk Period

The quiz specified that if the risk is materialised, it takes one day longer to complete Activity D but it doesn’t explain the risk period. A potential period could be:

  • Before Activity D start date;
  • Any time during Activity D period;
  • Specific day during Activity D.

Mitigation decision date

When is the latest date to decide on a risk mitigation strategy? It would be good if we could postpone our decision as we may be better informed by that time.

For example, the above analysis shows that if two resources are available, the risk has to be mitigated regardless of other uncertainties. So, when project commenced we may have some addional information that is not directly related to the project risk but couls help us to choose the best mitigation strategy.

Conclusion N2: Risk mitigation strategy is a variable parameter, not a stable risk characteristic.

Probabilistic approach
An alternative approach is to make a probabilistic decision. The project can calculate the chance the risk impacts the completion date by clarifying risk probabilities and building a probabilistic project delivery model. This approach can provide some vital information for project decisions. However, projects need to be extremely careful with this approach, as a low-quality model may show measling results and drive wrong project decisions.

Data-driven decisions require development of Dynamic Deliery Model that correctly shows how primary uncertenites and risks may impact project objectives (time, cost and benefits).

Could different project teams be using exactly the same initial information to develop a model that gives accurate and consistent outcomes? Could it be an another quiz?

Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma

Schedule Quality Metrics. Invalid Dates.

Schedule Quality Metrics. Invalid Dates.

One of the important questions, when a schedule quality analysis is performed, is: “Is the schedule review up-to-date?”. There are a few metrics that could help find the answer to this question and identify activities with potentially invalid dates. Some of them could be performed based on the current schedule and others require a comparison to the previous schedule snapshot. Identified issues may indicate that project has gaps in a schedule maintenance process that must be addressed.

Completed, in-Progress and Not Started activities

The status of each activity is defined by Project status date (data date), Start & Finish dates and ‘% Complete’. There is a correlation between these data points.

Incomplete activities in the past.

All activities with start and/or finish dates in the past have to be actualised (100% complete) or rescheduled.

Started or Completed activities in the future.

All activities with actual dates in the future have to be unactualised (0% if not Started and <100% if In-Progress) or actualised dates updated to the past.

Activates with a start date in the past and finish in the future but with 0% or 100% of progress.

In-progress activities have to have the start date in the past, the finish in the future, and ‘% Complete’ between 0% and 100%. All In-progress activities with 100% or 0% of Completion have to be reviewed. Dates or ‘% Complete’ need to be updated.

Incorrect Status of Milestones

When a milestone indicates completion of the event, the milestone has only to be marked as “Completed” when all predecessor activities are completed and, on the opposite, if all predecessors are completed, the milestone has to have a “100% Complete” status.

Actualised milestones with open predecessors.

Completed milestones with the “Not Completed” predecessor(s).

The status of the milestone and the status of predecessor activities have to be reviewed and updated.
This issue may indicate that the milestone has a strict dependency that must be removed.

Not Completed milestones with complete predecessors.

Not Completed milestones with ALL Completed predecessors.

This issue indicates that the schedule may have missing activities or dependencies.

Historical dates

Status, Start and Finish dates of already Completed activities should not change from version to version.

Unactualised activities

Activities that previously were reported as “Completed” but the status has changed to “In-progress” or “Not Started”.

Justification for each activity previously reported as completed, but not completed now must be provided.

Historical start and dates changed

Activities that previously were reported as “Completed” with changed Start and/or Finish dates.

Justification for each activity with changed actual start and/or finish dates must be provided.

The last two metrics are particularly important when a quality analysis of a contractor schedule is performed. Constant issues are a good indication that the contractor needs to mature their schedule maintenance process.

Also, projects with such issues may have a “double counting” issue when the same activity (or deliverable) is counted as being completed a number of times.  

Schedule Guidelines

GAO Schedule Assessment Guide, Planning & Scheduling Excellence Guide (DCMA-14) and NASA Schedule Management Handbook have metrics to identify invalid dates but they include only metrics to identify invalid activities in the past and the future, the first two checks in this post.

Schedule Tools

There is a philosophical question what to do when a schedule status update is not received?
Should a project team assume that all activities in past have not been started (or completed) and automatically push the activities to the earliest possible date: Project Status Update (known as Data Date) or they should assume that some of them may have started (or Completed) and not be updated dates until the clarification is received?
Probably developers of scheduling tools have different views on what is the correct answer to this question.
Primavera and Spider Project have the “Data Date” feature to automatically push activities with dates in the past to the Data date. Microsoft Project doesn’t have a such feature. In my observation, projects managed in Microsoft Project more often have “Incomplete activities in the past” issues and projects that are managed in Primavera more often have “Historical start and dates changed” issues.

Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma

Implementation of the Critical Path Methods in Different Scheduling Tools

Implementation of the Critical Path Methods in Different Scheduling Tools

Would the project dates calculated based on the critical path method be the same regardless of the applied scheduling tool? Not always!!!

Some time ago one large Telco company in Australia decided to replace their existing PPM system with ‘CA Clarity’. Clarity already was implemented in the organisation but used only as a corporate timesheeting system. The previous custom-made PPM system allowed to import Microsoft Project schedules and automatically retrieve dates for project status reports.

Clarity has integration with Microsoft Project, so the tool looked like a good alternative to the old PPM system. The portfolio initiated a project, which was a big disaster. The project manager made a dangerous assumption: “As both tools are based on the Critical Path Method, the schedules calculated in these tools are going to have the same dates”.

This was not the case. Imported and recalculated schedules had different delivery dates!!! The reality was even worse: the schedule imported back to Microsoft Project had different dates compared to the original schedule developed in Microsoft Project, even if the schedule was recalculated in Microsoft Project again!

In theory, the Critical Path Method is a well-defined method and a schedule calculated in any scheduling tool that supports this method should be the same.

In practice, projects consist of components: activities, calendars, constraints, dependencies, lags, resources, etc.

Scheduling tools have different features to manage and integrate project components. The way these features are implemented impacts the Critical Path Calculation.

When a schedule is imported between the tools some information could be completely lost as the new tool doesn’t have data fields to store this information.

In the described example, at that time (not sure if it is still the case) Clarity supported only resource calendars, not activity calendars. So, during the import, the information in activity calendars was lost and when the schedule was imported back to Microsoft Project, the project dates were different. Later, when I supported one of the Australian banks to intergrade Microsoft Project and Clarity, I found other differences between systems that impact schedule calculation.

Primavera, Microsoft Project, Spider Project and other project management tools support the critical path method they have unique features. The way these features are implemented may impact the Critical Path calculation!

Some examples of such features:

  • Primavera supports more than one dependency between the activities, and Microsoft Project doesn’t;
  • ‘Deadline’ in Microsoft Project impact Total Float calculation, Primavera doesn’t store ‘Deadline’ dates;
  • Primavera and Microsoft Project have different types of activity constraints;
  • Microsoft Project supports ‘% duration lags’, and Primavera doesn’t;
  • Spider Project takes into the calculation activity AND resource calendars, Primavera activity OR resource calendars;
  • Spider Project supports ‘Volume’ and ‘Duration’ types of activity, Primavera and Microsoft Project support only ‘Duration’ type;
  • Microsoft Project allows to link activities and summary tasks (WBS), Primavera only creat links between activities;
  • Primavera and Microsoft Project support Levell of Effort / Hammack activities, but this feature is implemented in a totally different way;
  • Primavera support the ‘WBS Summary’ type of activity, and Microsoft Project doesn’t;
  • Spider Project supports ‘Conditional’ scheduling, Primavera and Microsoft Project don’t.

Principles of the Critical Path Method could be explained based on a small simple schedule and calculation of such schedule in any scheduling tool should give the same result.

Real-life projects are much more complex. They have different calendars, hard and soft constraints, lags and leads, other than ‘Finish to Start type’ types of dependencies, etc. Implementation of these features impacts the way the critical path is calculated.

Some scheduling tools have methodological issues. Project’s real-life scenarios could not be properly simulated due to technical constraints. It is important to learn these constraints, find work-arounds or maybe replace the tool.

Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma

Monte Carlo Simulation Challenges. Data collection Challenge.

Monte Carlo Simulation Challenges. Data collection Challenge.

Terminology

What do the terms optimistic, pessimistic, and most likely really mean?
Because they are very different for different people, when one estimator says pessimistic, that person is thinking the technology might prove to be a bit more difficult than expected and thus take 10% longer. Another person in the team may assume that ‘pessimistic’ means that the work might be interrupted by an earthquake, tsunami, and nuclear meltdown simultaneously (even if it has not happened in the past).

Data Collection

Popular scheduling tools, like Microsoft Project and Primavera, don’t have native features to capture ‘3 points estimations’, so planners usually don’t capture estimations provided by Subject Matter Experts. A risk manager or a scheduler applies a range based on a single deterministic estimation instead. This approach is based on two dangerous assumptions:

      • All provided estimations are “Most Likely” estimations.

When SMEs are forced to provide a single estimation, they have to decide on how much contingency to include in the estimation. Some estimations in schedules have “optimism bias”, others, on the opposite, are too conservative.

      •  All activities have the same level of uncertainty.

There are many reasons why some of the activities may have dramatically different uncertainties: new technology, new equipment, new contractor, etc.  

Probabilistic scheduling tools, in the opposite, give an opportunity to capture activity durations as ‘3-point estimations’. Spider Project, arguably, the most advanced project delivery tool gives the opportunity to capture any type of initial data as 3 points: durations, cost, volume of work, resource productivity, lags, etc.

 Primavera and Microsoft project have custom fields that could be used to capture at least the duration of activities. Then weighted durations could be calculated. Both tools have built-in formulas. PERT or any other method could be applied for that.

Microsoft Project used to have built-in 3 points estimation fields but it’s current version doesn’t have this feature any more.

Objective vs Subjective

The Monte Carlo method works well when a lot of historical data is available. Unfortunately, the unique nature of projects means that likely historical data is going to be limited. However, even a small volume of historical data could be beneficial and portfolios have to implement methods to collect historical project data in a systemic way.  

 When historical data is not available the only way to collect estimations is to interview Subject Matter Experts (SME) and collect subjective information. However, it needs to be done in a smart way.

 It’s always better to ask a SME to provide an optimistic estimation first. Then ask the interviewee to think about the worst-case scenario and only after that provide the most likely estimation.

The trick is when the interviewee thinks about different possibilities, the most probable estimation is likely to be more accurate.

Hidden Risks

I found it extremely useful to collect data as a range even though I am not going to use it for Quantitative Risk analysis. It actually doesn’t take much longer to do so but gives me an opportunity to identify hidden risks. After all estimations are collected, some activities may have an abnormal range, comparing to the rest of the activities. It’s worth discovering the underlying reasons as there is often a story behind each abnormal range. Some of these stories get converted into risks or even issues.

Alex Lyaschenko

PMO | Portfolio Planning & Delivery | PMP | P3O Practitioner | AgilePM Practitioner | Six Sigma