By Pat Dixon, PE, PMP
President of DPAS, (DPAS-INC.com)
This article is not directly focused on Industry 4.0 but is an example of what can happen in automation projects and how fundamental principles of engineering and ethics must not be overlooked. This is a true story, but I am not going to mention names or companies.
The project was to automate a production bay for a semiconductor chemical facility. Chemicals used in the processes can be rather exotic, with toxicity being severe to unknown; therefore, safety should be of the highest priority.
In this bay there was an existing distillation column and control system. The control system was antiquated but worked. It automated the sequence of distillation; so that from a start command, it would fill a flask, start the heater, wait for the vapor to rise, go through reflux cycles to separate the components, then turn off the heater and end the sequence when the product tank was full. The system also had a sequence to transfer from the product tank to customer bottles, but this sequence was not working so they did it manually.
The customer wanted to upgrade this system with a more modern offering that they had standardized on in their newer bays. They wanted the automation and software to match the design that had been used in these more recent projects.
In addition, they wanted to add another distillation unit in the bay for a different chemical. That required process design work and construction in addition to the automation.
The project included developing the user interface (HMI) displays for the operation of the process. There were the original displays from the old system and examples to go by from the newer bays; but given that they were now going to have two distillation columns in a bay that used to have only one, design decisions needed to be made. Putting too much into a single display is not good practice, but the customer wanted everything on one display. It was unclear whether the lines in the display should have animation to show where there was material in the pipe; so on the advice of the customer engineer advising on the HMI, it was decided not to implement animation.
There was no written sequence of operations for developing code to provide the required automation functionality. While the old code could be used for guidance, the customer recommended reusing code from an upgraded bay. While this did help provide a starting point, it did not confirm all required functionality. Therefore, to ensure the HMI and functionality met expectations, agile development was employed to show the customer weekly progress to confirm requirements were met.
The only feedback control (PID loop) was for the temperature of the distillate. The code for upgraded bays included this functionality, so the customer asked that it be implemented for the new bay.
The schedule to get both distillation columns yielding quality production in the new bay was aggressive. In nearly every project, automation becomes the critical path when mechanical completion is achieved.
With that as a background, the following issues arose during the project:
- The first week of onsite commissioning was unproductive due to unavailable power, resources, and incomplete construction. This compressed the commissioning schedule. Continued mechanical problems during commissioning put more pressure on the automation schedule.
- During startup of the first distillation column, the customer was shown a problem with the reliability of the mantle temperature measurement. The mantle temperature sensor is located between the heating mantle that provides the heat for distillation and the glass flask holding the chemistry. This was a critical measurement used in two safety interlocks. One of these interlocks would determine if the differential temperature across the glass flask was too high, which could cause a failure of the glassware and spill toxic chemical. The customer did not regard this as a priority item and was not addressed. The customer wanted the interlock inhibited to allow the unit to operate.
- During commissioning, it was discovered that several more inputs to safety interlocks were not functioning. Instead of fixing these problems, the customer wanted to run the bay with these safety interlocks inhibited.
- The customer was responsive for providing the safety interlock design. The original design of safety interlocks provided by the customer failed to include shutting down the heaters. This would have meant that when an unsafe condition was detected and the system went to shutdown state, the heater would still be cooking toxic chemicals in the flask. This was identified, the customer was informed, and the design change was made. Operators said this could have been a disaster.
- In commissioning it was found that tuning a PID loop for control of distillate temperature was impossible due to extreme deadtime in the dynamic response. Inspection of the implementation in other bays showed that all PID loops were locked in manual mode with both high and low outputs limits clamped at 40%. Instead of PID control, the other bays were manually stepping the % output to the heater in a sequence of timed phases. This meant that there had never been successful implementation of PID control at this facility. The customer never made the vendor aware of this. The control design was modified for the new bay to use a proper temperature measurement to reduce deadtime. This bay became the first and only one to have automatic PID temperature control.
- Given that there was no written sequence of operations, discussion with operators helped document these details and ensured the required automation functionality was delivered.
- When production began, operators had full face masks and respirators on and were warning people not to enter the bay. Despite these warnings, management personnel were ignoring the operators and walking right into the production bay with no protection.
- As the project neared closeout, the customer was more concerned in cosmetic issues in the HMI than in the safety interlocks. It was insisted that animation be added to the lines, overturning the prior decision to omit them. Despite having reviewed the HMI design 4 months earlier, last minute requirements for addition or re-arrangement of information of the HMI was demanded. The customer scrutinized pixel by pixel alignment of objects instead of addressing unresolved safety concerns.
- On the final day of the project, the customer insisted an online change be made to add a new interlock for a non-existent switch. This was while distillation was in the production. There was no urgency for this interlock since the switch would be added at some unspecified time later. Adding an interlock during operation is a highly inadvisable practice, especially when it can be safely done with the process shutdown.
Despite all these issues, the automation system has been in use for over a year, and operators confirm that it works as desired. While operators consider the automation system a success, management considers it a failure and blames the vendor.
This can happen when vendors are not treated as partners or team members. Some manufacturers treat vendors like competition; the customer tries to get the most benefit with the least responsibility for the least investment. The vendors behave in the opposite direction, and when problems arise everyone looks for a scapegoat. In such an environment, engineering or project management solutions become much more difficult to implement because of hostility and lack of trust.
The most successful projects are those where everyone is a team member. When problems arise, stakeholders work together to find the best solution.
Fundamentally, this is more a matter of ethics than engineering. There must be a commitment from all team members to do what is right and treat others with respect. Without that, a project becomes an internal war, and there will be casualties.
In Industry 4.0 the reliance on automation grows, and project such as these need to be successful. Ethics needs to be at the foundation of projects to keep us safe and sustainable.
