Capital Health Success Story

Overview
Capital Health provides core health services to 395,000 residents, or 40-percent of the population of Nova Scotia. It also provides specialized services to other residents of Nova Scotia as well as residents of New Brunswick and Prince Edward Island.

The organization employs 8,500 people across two main campuses and 13 other satellite facilities. Capital Health’s computer network includes approximately 165 servers as well as close to 5,000 desktop computers connected by a combination of Windows, Novell and UNIX network operating systems.

Maintaining precise time on the computer networks helps the hospital function properly. “We don’t necessarily have to be accurate (correct real time), but the time on our devices must be precise (identical to each other),” said David Devlin, Senior Systems Administrator for Capital Health. “Having servers and clocks synchronized is vital when it comes to assessing patient care. For example, Physicians must know precisely when drug and anesthesia administration occurs, and they need to know when specific treatments begin and end. This helps determine whether they need to alter how we treat certain patients.”

Another function that relies on precise time is the hospital’s transcription service, which time-stamps dictation notes for close to 700 doctors at Capital Health. “These recordings become legal medical documents,” Devlin said. “The transcription time comes from the computer system, and if two systems fall out of sync, it creates a problem comparing records when assessing patient care.”

Capital Health had relied on free time servers via the Internet to synchronize its internal servers. Desktops and network devices also utilized Internet time sources via root servers on the network. The hospital deployed low-cost shareware that pinged the various time servers on a periodic basis. If the application could not find a server, it simply looked for the next time server on a pre-determined list.

Challenge:
The free Internet-time solution seemed to work, but Capital Health ran into a major problem. “Many of the Internet time servers we had been pinging were no longer in service, so a large group of our servers and desktops defaulted to pinging a single time server via one of our root servers,” said Devlin.

To this external, Internet-based time server, traffic coming from many devices at one company looks like it’s all coming from one source. In reaction to this, the owners of the external time server set their system to transmit a time five years in the past, to December 31, 1999. In the owner’s opinion, the organization did not act maliciously — it just knew that a specific source, which could not be identified, was pinging their time server an unreasonable number of times per day.

Thus Devlin and his co-workers arrived at work one day and started receiving support calls that some of the servers displayed times significantly off from the correct time. “We determined that 29 of our 165 servers registered the wrong time,” Devlin said. It took close to three hours to discover the affected servers and correct the problem. “I didn’t get to many meetings that day,” Devlin said.

The problem originally began at about 4am. By the time Capital Health discovered the situation at 9am and fixed it, nearly eight hours had elapsed. Patient-care data entered during this period on those 29 servers appeared incorrectly on reports. “With the time set-back by five years, the system immediately archived some of the files,” Devlin explained. “We did not lose files, but the data for that time period became invalid, and many historical trace logs were lost. The tools we use to look at them simply do not see the time. We were lucky. If the time had been set back only three or four days, it would have been far worse — we could not have easily identified good data from bad data.”

Solution:
“We basically lost faith in the various organizations that deliver free time over the Internet,” Devlin said. “We don’t know the people who manage the servers on the other end. What if they have a disgruntled employee or someone who wants to play a joke? Precise time is outside of our control on the Internet.”

Devlin searched for a network appliance that would give control back to Capital Health. His research on the Internet and information gleaned from product manuals led to the clear conclusion that Symmetricom offered the best solution.

“We also looked at two other products, including one that cost less” Devlin said. “But we chose Symmetricom’s NTS-200 network time server because of its GUI (graphical user interface), precision capabilities, and failover features.”

The NTS-200 synchronizes clocks on servers and workstations across networks by combining a high-speed/high-capacity network interface, versatile GPS timing-receiver-technology, a web-enabled user interface, and a wide range of network protocol support.

The GUI interface allows Capital Health to easily manage and check on the NTS-200 when necessary, but the appliance has functioned so well that Capital Health simply set it up and has since left it alone. “Anyone can come up to speed on it immediately,” Devlin said.

But perhaps the feature that sold Capital Health most on the NTS-200 was its ability to failover to Internet time should the need arise. “If the antenna goes down due to a storm, we don’t want long holdover times,” Devlin said. “We prefer to temporarily failover to Internet time from multiple time servers, that we have negotiated an agreement with to provide failover time, rather than rely on the oscillator that many Symmetricom competitors require.” Devlin found that other solutions did not offer this feature, referred to as peering.

Capital Health installed the NTS-200 without any assistance from Symmetricom. Within minutes the appliance detected several GPS satellites and synchronized the time on Capital Health’s servers, routers and other network devices. When Devlin purposefully forced the antenna connection to fail, the system properly switched over to Internet time sources.

Results:
Approximately 200 Capital Health devices, including servers, routers and switches, now point directly to the NTS-200. The time on all the devices has remained precise since the installation, and Capital Health’s team has not had to spend any time monitoring the NTS-200 since initially watching over the system during the first week.

Once Capital Health’s IT department demonstrated the success of the new time server, other departments in charge of their own servers asked to also point their servers to the NTS-200 for time synchronization. “With the time-problem event that occurred and the features demonstrated by the NTS-200, all of our departments now understand the value of accurate and precise time,” Devlin said. “They also appreciate the ability of having control over precise time from within the company.”

The NTS-200 has proven to work effortlessly with all of Capital Health’s server operating systems — Novell, Microsoft and UNIX. The appliance also synchronizes Capital Health’s operating room clocks, paging systems, and telecommunication switches.

With accurate and precise time, Capital Health’s patient reports will present valid results for doctors to consider. This in turn helps to improve upon patient care. But for Devlin, the main benefit is peace of mind knowing that Capital Health now has full control over the time on its network devices. “Synchronized systems are absolutely critical,” Devlin said. “Without this we compromise our ability to provide a high level of patient-care related services. To leave this responsibility in the hands of an outside source is just too risky.”