I’m continuing with the series of posts related to my experience with different Hydroelectric Projects, in Venezuela and in Canada, from the point of view of a Project Manager, Plant Manager and Maintenance Engineer.

We already talked about the conceptualization and initial planning of a hydroelectric project. This second post is about the commissioning stages: what should be taking on count during the commissioning planning, some of the risks that should be identified and some of the mitigating measures you should take on count for those risk. As well, some good practices based on experience for commissioning tests and coordination with your operations’ stakeholders. I will talk about the generation aspect only, which is where my strength and experience is, the transmission aspect is a good subject for a dedicated post in the future. The idea of these post is to start conversations and debates around the topic. All suggestions and comments are welcomed!

Post #2: Time to test the machine

The commissioning phase of a hydro project (and in my opinion, of any big industrial project), is probably the most exciting phase. Test and seeing the different systems of your project being operating for the first time, and especially in hydro, that first turn of the generating unit, is something that definitively you must be there to really feel it. It is an extremely exciting experience!

But is not an easy path to get to this stage. It does require a lot of planning, a huge team effort, lot of coordination between different stakeholders with different priorities, and a very big focus on quality control. We will talk about those four aspects in this post.

1. Planning for commissioning

The first question that many people may ask is “when is the right time to start to plan for commissioning?”. Some may think that is just after the detailed engineering is completed, or after the basic engineering is completed, or after the long lead time equipment arrives to site and is ready to be installed. The answer is yes to all those questions.

Early planning stages

When the project is being conceptualized, and the basic engineering is completed and the gates to go to the next phase, detailed engineering, are passed, is a good time to start to talk about the commissioning stage on a high level. What should be discussed on this phase is:

• A first iteration of “chronologic order” of events in terms of what major systems require to be commissioned first, and what systems follow up. The first iteration serves as a start point, and inevitable will suffer for changes, and we will talk about change management.
• What will be the temporary, and permanent, site services available to start commissioning? When I talk about site services, I mean access to sources of energy (electricity, water and compressed air), as well control over those sources.
• At this point some equipment vendors are already selected, with others still on queue to be selected after detailed engineering is complete. It is good to get the vendors involved in this planning stage, to understand what their internal capacities are to go after commissioning activities, and what will be the tasks that they may want to bring specialists.
• A first approach to a commissioning budget can be done at this stage, with an accuracy of ±50%.
• A first iteration of the quality control plan for the entire project, and to each major contract of the project, should be completed and finalized before advancing to detailed engineering. This will serve as a baseline on how the quality will be manage during construction and installation and will help the designers to understand the quality requirements that the project requires on each component, adapting that way the detailed engineering to those requirements.
• Risk management plan should be updated at this stage (well, in fact, a good practice is to revise periodically the risk management plan, and update it as a recurrent activity, adding or removing risks as the project advances) to include potential risks during commissioning, and establish mitigation measures. A risk management plan focused only on commissioning activities is recommended, and at this stage, will be a high-level one.

Planning for commissioning during and after detailed engineering stages

Hydro projects are projects with so long lead times and spread about many years, that in fact many veterans in the industry take a “plan as we go” approach. That approach, even though is valid for certain activities, can lead to issues especially during commissioning.

When detailed engineering starts, and the equipment and vendors of the support equipment are selected (when I mean support equipment, I mean everything except the civil structure, the turbine, the shaft, the rotor and the stator, the wicket gates, the operating ring, the tail race and intake gates or valves, and the spillway gates and structure), it is time to build a detailed commissioning plan.

The first iteration of the commissioning plan will be your start point. Over that, you will buy the detailed plan for systems and subsystems. During this stage, you will understand the different interrelationships between systems and sub-systems, and prioritization will be a necessary and not easy task to accomplish. Some advice for that:

• Prioritize the simple systems that can allow or make easy to other more complex systems to be completed first. For example: service electric power and plant and generating unit compressed air. The most complex one, the service electric power, can be tackled down by areas, aligned with the schedule of commissioning of other systems, but definitively requires a high level of attention and prioritization.
• If your plant has a spillway with gates, or another mechanized reservoir spill control system, in your plan I suggest prioritizing the completion of its commissioning over the generating units itself in terms of resource assignment. This way, you will guarantee full control of the reservoir through the final stages of the project and ease the job of the generation unit’s commissioning team in terms of pond level control for several necessary tests.
• Prioritize quality control during installation. Don’t be afraid to invest money, time and people on quality control during installation of equipment, it will save you tons of hours, money and people on re-work during commissioning. Notice that I’m using the term “invest”, not “spent”, because quality control is an investment that pays off later.
• During detailed engineering stages, always keep in mind how you will install the equipment, how you will commission it, and how the customer will operate and maintain it. Prioritize commissioning and operation optimization over installation easiness and prioritize quality control and quality delivery over schedule.
• The risk management plan needs to be updated. At this stage, it could be beneficial to start to involve the vendors of the equipment itself and other stakeholders of the supply chain, and update the risk assessment of the supply chain, the availability of parts for commissioning, the availability of people, and establish realistic timelines. This could lead on a re-prioritization of activities, so a strong change management process needs to be already in place.

And here we will make a pit’s stop, to talk about change management. The only thing that is constant on a project is change, is the responsibility of the project team to manage that change, and to establish a strong change management process. In the first post of this series I talked about conceptualization, and that is during that stage where changes are way easier to do. When you are on commissioning, changes are very difficult, so it is very important for the project team to be very careful in terms of what changes (at the micro and macro levels) are introduced to the project, because it will impact greatly the final deliverables.

It is not uncommon to find issues during commissioning that may require a change on the specs of a system, change an equipment completely, or even disassemble a system to a certain point (for example, removing the runner from the unit after being already in position and ready to turn, that’s more common that you may think) to be able to correct a defect or make a system work as intended under specs. To try to avoid that (I repeat, try, because sometimes is just inevitable) is why you invest time and resources on quality control during detailed engineering design and during installation, so the changes, limitations, defects, etc. manifests themselves earlier in the project when is easier, and cheaper, to solve.

2. The coordination between stakeholders: contractors, project management team, and operations

The customer’s operations team should be involved in the coordination talks between the main contractors, the project management team, and other stakeholders as soon as the final commissioning plan starts to be developed. This will help to get alignment during the execution itself of the different tests that the commissioning activities requires. All parties will be on the same page since the planning stages, avoiding any kind of friction and helping to resolve potential conflicts even before they occur.

As well, this helps with the prioritization of activities. Let’s remember that each stakeholder has their own interests (completely valid), and their own priorities (also completely valid), but the project itself has one big priority: be completed on time, with quality, and on budget, complying with all regulations applicable. So, having all the stakeholders aligned with that global priority is a task that needs to start as early as possible in the planning stages, so during execution stages fully alignment may be possible.

Tests require coordination between parties, and vendors and contractors had a lot to say on it. It is important for the project team to get their inputs directly and constantly, and to facilitate the coordination between them and operations if it is the case. Prioritization is very important, try to follow the plan already in place with the order of activities to avoid or reduce possible conflicts.

Communication, communication, and more communication is the key to coordinate between all different groups and stakeholders in a project of this magnitude.

Another aspect in multi-unit plant’s projects is that a unit can be operational while other are being still under commissioning or even under construction. Clear delimitation of areas of authority (what is managed by the customer’s operations team, what is manage by the contractor) must be already in place way before the tests start. A clear protocol and a clear communication and coordination plan should be in place and agreed by all parties before reaching this stage. On turnkey projects, where the contractor has full authority until the very end of the commissioning phase, this is not an issue.

It is recommended for the units’ test, when load is being put on the grid, to coordinate with the transmission authority well ahead of time what will be the limits and consequence of the test to the grid, especially the load rejection tests. The very first tests should be coordinated to occur when the load of the grid is lower, to avoid negative impacts in case of a failure, which is very common.

3. Failures, Quality Control, and documentation of lessons learned during installation and commissioning

One of the things that will happen, and that you should expect to happen, during commissioning, is failures on systems and equipment. In fact, it is the best time during the entire project for failures to happen. Normally, during commissioning stages, you have the experts on site, and the vendor support very close and aligned with the objectives of the tests, to support and make the commissioning activities to go as smooth as possible. Well, part of the planning efforts are aimed to establish the presence of the best resources around the commissioning activities, including of course the experts that will perform the tests, sometimes the designers of some of the systems should be available to be consulted right away, and remote support should be fully available.

But again, this is the perfect time for failures to occur and manifest themselves. And this should drive a revision of the risk management plan as well (again, is a constant activity, that ends only when the project fully closes).

Why? Because you have all the support around, and the conditions of your systems and subsystems, including the grid itself, are “test conditions”, so a failure should not have the impact that it may have when you are on the commercial operations’ stage. Not only that, from the vendors’ and contractors’ technical support on site, your operations team will learn how to deal with some failures with the help of the experts, better than any training program, and at no additional costs.

Expect and embrace failures during commissioning, learn from them, document them, add them to your risk management plan.

Now, Quality Control is a critical part on all of this. Even though failures are common and expected, what should not happen is rework due to quality issues found at this stage. We already mentioned that Quality Control must be a priority since the design phases, and during installation. The better the Quality Control process is during installation, the smoother the commissioning phase will be, and the possibilities of rework are far reduced. We should not confuse failures due to tests and equipment configuration, with failures caused by poor quality control during installation and during design, the last ones are not acceptable. The way to mitigate that risk, is invest heavily on Quality Control since early stages of the project and intensify that investment during installation.

Quality Control records should be kept for future references and be part of the final delivery package for the customers’ operations team. Not only that, but it is also highly recommended to the customer to consider investing on a dedicated documentation and lessons learned engineering team. The job of that team is to organize and store for future reference all the information that is originated during the tests, on a document database. In my experience, especially back in Venezuela, the company invested on that, and it helped enormously in the diagnose of failures and issues years after commissioning and into commercial operations. It is not uncommon that failures that happened during commissioning, and are resolved, repeat themselves again years into operations, due to normal wear and tear of the equipment, or lost of accuracy of instruments after many years operational, or simply normal aging of the equipment. Having the information about how the commissioning team dealt with a similar failure saved a lot of time in diagnosis, repair and drives decisions on improvements and upgrades.

4. Final thoughts on Commissioning

As it was mentioned at the start of the post, the commissioning stage is probably one of the most exciting phases of a hydro project, and any industrial project in general. For it to go as smooth as possible, it requires proper planning, risk management, coordination between stakeholders, big focus on quality control, and proper follow up and documentation, to learn as much as possible from it.

And you, what do you think should be considered on the commissioning phases? Let’s chat!

Javier Ojeda