Reliability Problem in Pipeline Control System

The Situation:
A major Oil Company operating a pipeline serving many customers controlled the entire system using a Honeywell Distributed Control System (DCS) from a central facility over a TCP/IP network. There were frequent, but intermittent, setpoint failures across the network which could — if repeated across the A & B systems — have triggered a pipeline shutdown.

The Response:
YR20 deployed Network Reliability Probes on an extended basis to capture all the network traffic to identify the root cause of the setpoint failures. Analysis of the network traffic showed that there was chronic packet loss in both the A & B TCP/IP networks. At times there was acute repeat packet loss which caused the DCS setpoints to time out nd fail. Further analysis of the
configuration of the TCP/IP routers in the Wide Area Network (WAN) showed that the router configurations were sub-optimal for use by a TCP/IP based DCS.

The Result:
The WAN TCP/IP routers were re-configured to be optimal for the critical DCS traffic. The network traffic was analysed after the changes and showed no packet loss. The DCS setpoint process became reliable.

General Lessons from this Case Study:
TCP/IP is an extremely robust network protocol and
is well suited to use by DCS. However (there is always a “however”...) its behaviour must be fully understood and this must be incorporated into system design, engineering, commissioning and maintenance in order to maintain a reliable system.

In the presence of packet loss TCP/IP does not fail in a linear manner. TCP/IP will continue to operate in the presence of packet loss in an apparently normally manner showing only degraded latency but will then fail abruptly.

The principles of engineering, commissioning and maintaining critical TCP/IP systems are no different from other oilfield systems. The only difference is the specific skills, tools and processes which are new.