Performance Problems in Sub-Surface System

The Situation:
An Oil Company moved to a new PC-based sub-surface interpretation system with all the data stored on Microsoft servers using an entry-level Storage Area Network (SAN) for the hard-disks. The interactive performance was inconsistent and prone to briefly freezing during seismic interpretation. The application, PC, server and network teams all blamed each other. Users and management were getting frustrated.

The Response:
YR20 deployed Network Reliability Probes to capture the network traffic between the PCs and the Microsoft servers. A number of tests were run — by the software vendor — at different times of day and with different numbers of concurrent active users. Analysis of the test results and the network traffic showed a clear relationship between the number of concurrent users and the likelihood of poor performance. Associated network analysis showed that the response times of the data storage server became slower and unstable when
multiple concurrent users were active.

The Result:
The server and SAN systems were upgraded to improve performance but disputes continued. When the contract expired for the PC-based sub-surface system it was replaced with one of the proven large-scale products known to scale to large numbers of concurrent users.

General Lessons from this Case Study:
The PC-based sub-surface system involved has a long and reliable track-record in single-user situations and is widely used by independent sub-surface consultants. The system did not have a proven track-record in multi-user environments in OilCos using shared data storage on Microsoft servers.

The first lesson here is that the PC-based system was being deployed outside its proven “single user, data stored on the PC” configuration. In such a situation, the OilCo should have insisted on a full multi-user Factory Acceptance Test before concluding the contract.

The second lesson here is that because of the product’s history, the software vendor did not have proven known-to-work configurations for all parts of the system; the software, the PCs, the network and the data servers. The OilCo should have insisted on getting such information.

The third lesson is that the software vendor did not have a full set of stress-test software available which could be used to prove that the Customer environment of PCs, network and data servers was — or was not — appropriate for running an extremely demanding application. The vendor should have insisted on running its own audit and stress-test of the Customer environment.

In the absence of known-to-work configurations and pre-deployment testing of the Customer environment there was ample space for problems developing into disputes and a blame situation.