I.T.@Kelley Drye & Warren L.L.P.
Taming the Data Dragon
How we slayed our backup nightmares.
By Judith Flournoy
IN 2000, Kelly, Drye & Warren L.L.P. faced a challenge: The amount of data stored on our New York office network had continued to grow at a rate that is one to three times faster than our former DOS-based environment. To manage the information glut now upon us, we had to re-engineer how data would be stored and backed up, while providing for continued growth. Our existing digital linear tape (DLT) tape backup systems used for the file and print, as well as our Microsoft Exchange e-mail system were laboring to complete backups in under nine hours.
We clearly needed a big change.
First, a bit of history. Kelley Drye is a general practice firm, heaquartered in New York City. We have seven domestic offices, two international offices and several international affiliates. Founded by two partners in 1836, we now have about 350 lawyers, and more than 20 practice groups, including litigation; project finance and infrastrucutre; telecommunications; venture capital and e-commerce; and real estate.
In 1997, in preparation for a Windows 95 roll-out, we upgraded all servers firm-wide. At that time the firm purchased Compaq Proliant Dual Pentium 200 MHz servers with 4 GB internal SCSI drives, configured as Raid 5. In our New York office, the design included three file and print servers running NT 4.0. This meant that every user profile, policy change, application update, and log-in script had to be replicated across the three servers.
Administering user accounts, moving users from one server to another as they changed floors became an administrative nightmare. The nightly back-ups ran eight to nine hours, often creating problems for users who needed to work on documents or access e-mail overnight or in the early hours of the day.
As luck or passing technology would have it, we really faced two challenges.
First, our two-year-old DEC Alpha, which was host to our CMS OPEN time and billing software, had become under-powered and short of disk space. Second, as previously mentioned, the increased storage requirements for the data generated in our new Windows-based environment was filling up (quickly!) the available drive space on our file and print servers. For this reason we had to decide if we should increase drive space, memory and CPU on the DEC Alpha, and increase drive space on our existing file and print servers for the third time. The Alpha question was answered when Microsoft Corp. announced its intent to phase out support of the RISC platform for its NT operating system. So, replacing the DEC Alpha was inevitable. But what to do about those three file and print servers?
I knew the firm would need to invest significant dollars in replacing and or upgrading the existing equipment. I wanted to make the best possible decision for all the right reasons. Total cost of ownership, initial investment and return on investment were three critical components of the decision making process.
I had been reading about "network attached storage" and "storage area networks" in various publications. NAS can be described as a storage appliance on the network, a turnkey product for client access to data. SAN can be described as a fibre channel network with SCSI storage devices using "I.P." Internet protocol (Note: Technologists spell "fibre" that way, to differentiate it from fiber as in fiber optic networks.)
I was intrigued by the idea of utilizing a high-performance tape backup system in conjunction with the ability to consolidate storage into one fully redundant storage system. If I could reduce administration of our existing file and print environment in our New York office, while increasing availability and redundancy, the transition from the three file and print servers to a single server would be worth the time it would take to migrate.
The added benefit of a multi-drive tape backup library would be to reduce backup time to such a degree as to all but eliminate the impact of nightly backups on the user population.
Timing is Everything
PC Expo was upon us in New York and Dell Computer Corporation was in town to show off its latest server, storage, and PC technology. I grabbed a cab and headed over to Javits Convention Center, where I felt as if I were walking into a car dealership showroom: everything was bright and shiny and the sales people were excited to tell me all about the power under the hood.
The Dell PowerEdge 8450 was one of those featured products. This eight-way Intel based server, with fully redundant hot-swappable components, was a high availability server and a potential replacement for the DEC Alpha.
Could the same model or similar model replace those three Compaq Dual Pentium 200MHz file and print servers?
After thoroughly researching the Compaq and Dell server products, I de-cided to replace the DEC Alpha with a Dell PowerEdge 8450 eight-way; and the three file and print servers with a Dell PowerEdge 8450, eight-way. This decision was not easy. In the end, when the final decision to go with the Dell server was made, I had the support of my staff as well as our integrator.
Storage Area Network
Storage area network technology (synonymous with fibre four channel) has been around in one form or another since 1988. Supporting up to 126 devices, the fibre channel adapter would provide ongoing expandability. At its inception, fibre channel was intended to be a high-speed Internet Protocol (I.P.)-based backbone, but was later seen as scalable and very flexible storage technology.
Up to the end of 1999, we increased our drive space on several servers, first from four GB drives to nine GB drives and then from nine GB drives to 18 GB drives. The technical limits of SCSI, i.e., 16 attached devices (including the host adapter), meant we would have to add servers to support more disk space. Of course we could replace the existing disks in the servers every time larger disks became available.
Anyone who has upgraded drive space in a server knows that the process is arduous. Backing up the data, swapping out the hardware and restoring the data requires downtime of the system, which means the resources are not available to the user during the upgrade. Like most I.T. departments, we try to minimize the impact of system maintenance and upgrades as much as possible by performing the work over long holiday weekends, early mornings or late evenings.
After looking at the Compaq and Dell storage products, the Dell PowerVault 650, and two PowerVault 630 drive enclosures which are fully redundant were selected. Each fully populated with 10 SCSI drives. Ultimately, each can support up to 4 TB of storage. To complete the design, a PowerVault 35F (optical bridge), a 51F (eight-port fiber channel switch), and a PowerVault 150T (tape backup library) with four drives was added. Now we had a storage infrastructure in place to support the burgeoning data storage needs the applications demanded.
Moving CMS Open off the Alpha and onto the new Dell eight-way was a straightforward project that was completed over a single weekend. We had a clear cut project plan developed by Tom Nohs and David Silvey of the KDW I.T. department, in partnership with our accounting department for post-installation testing. Their attention to detail essentially meant that our users returned to work on a Monday never knowing the Alpha had been retired.
There was an immediate gain in performance as evidenced by the SQL log dumps, which went from one hour to 18 minutes, and our backups now run at an average of 375 MB per minute. We were quite pleased with the outcome of the upgrade from the DEC Alpha to the Dell PowerEdge 8450 which was confirmation that the decision had been the right one.
The three file and print servers were a much larger and more complex project. Due in part to the way in which our original servers were set up, every batch file, login script, link file, user home directory, and user data directory, had to be moved and modified to adapt to a single server environment. All applications had to be moved or reinstalled. All documents had to be moved which meant the PC Docs user profiles had to be updated to reflect the new location.
For this project we enlisted the assistance of Union Square Technology Group, whose staff worked closely with James Barry, a member of the KDW I.T. department.
At the end of nine months we had successfully migrated the users and data from the three file and print servers in New York to one Dell PowerEdge 8450. We have since relocated those servers to regional offices, leveraging an existing investment to the benefit of all locations.
An added benefit: One of the companion products available with the Dell PowerEdge server line is the DRAC, a.k.a. Dell Remote Assistant Card. This card provides the capability to remotely manage the server via an Internet connection all the way through the server POST and Operating System load. For the first time, we could actually manage a server without having to walk into the data center.
The following project goals were accomplished:
* Replace DEC Alpha
* Increase database performance in CMS OPEN
* Increase storage capacity and extendibility
* Introduce fault tolerant 24x7 systems, provided by highly redundant components and a back-up system that does not affect users.
* Ease administration
* Reduce the cost of ownership
* Reduce backup time
Our experience has been so positive that we have just installed another SAN and high availability Dell server in our Washington D.C. office. More recently, our ability to add drive space to the SAN enabled us to expand storage for a litigation project, where in the past the project would have required multiple servers and storage. Not only did we solve our storage problem, we were also able to connect other servers including our Exchange server to the SAN and tape library for nightly backup.
Judith Flournoy is director of information technology for Kelley Drye & Warren L.L.P., and is based in New York City.