INNOVATION Data Processing Employs Cole Software's z/XPF Profiler to Enhance Product Performance

About Innovation Data Processing

When your business is providing software products that improve performance and total cost of ownership for your customers' computing, it's mission-critical to deliver highly-optimized tools that can achieve results quickly. This is the ongoing task for INNOVATION Data Processing (http://www.innovationdp.fdr.com), a leading international provider of enterprise data protection, business continuance and storage resource management solutions. Since its beginnings in 1972, INNOVATION's focus has been on combining high performance design with non-disruptive virtualization technology, easy to use and extremely reliable backup, recovery, data replication and privileged information protection that help eliminate downtime and data security concerns for enterprises around the world.

"All of our products are focused on improving application performance for our customers," said Rich Morse, INNOVATION Senior Software Developer, "The goals are to lower resource utilization by reducing CPU cycles and I//Os and by raising overall performance. If our clients can do that with the help of our products, they gain efficiencies and lower overall costs of operation."

Morse and fellow Senior Software Developer Terry Thomas collectively have 70 years experience at INNOVATION developing software products that solve enterprise computing performance and optimization issues. "We've worked on virtually every IBM mainframe operating system and language," said Thomas. "The IBM System z is an outstanding enterprise computing platform, and we pride ourselves in delivering solutions that assist sites in getting the most work they can out of the systems they have."

In its product development lab, INNOVATION runs an IBM System z9 machine that hosts various levels of z/OS, the primary System z operating system. "We use the System z9 for product development and test, and we also work hand in hand with other hardware and software vendors as well as our customers on performance and optimization," said Thomas.

Problem #1

One challenge INNOVATION faces as a solution developer is testing new product releases in every possible system scenario that customers employ. "I had noticed an increase in CPU time when I was doing development on FDRVIEWS, a product that uses FDREPORT extensively as the primary data gatherer," said Thomas. "In our shop, the increase had little proportional impact on the wall time. It was more of just a slight increase, so I ignored it. However, when a beta client called in with a significant increase in CPU time that had a much larger proportional impact on the wall time...as much as 150 percent, if I remember correctly...it caught my attention."

Thomas compared the source releases, examining all changes, line for line. "I found a few things, but nothing that added up to even a few percentage points in CPU time, so I asked Mike Geake, who works with me on the product, if he knew anything about this old software profiling product we had in-house, to see if it could locate the problem areas," said Thomas. "Mike ran the old profiler against three release levels to see if we could find some hot spots where the CPU time had markedly increased, but nothing major popped up. All in all, we were looking at output generated from instructions in six separate modules. We went over all of the entries that were listed, and again nothing more was found. This went on for more than a few weeks, looking, checking, chasing other problems, but always coming back to the listing with no significant results and with a second beta site reporting, now having to now tell two clients that I had found nothing that could account for their problem."

Coincidentally, Lou D'Agnolo, the manager of the INNOVATION Systems Group who was already familiar with Cole's z/XDC interactive debugging tool that INNOVATION had been successfully using for years, was reviewing information on ColeSoft's new z/XPF Profiler and passed it around for comments.

Frustrated and perplexed, Thomas decided to put ColeSoft Marketing Inc.'s new z/XPF software profiler to the task. "The first thing I tried with z/XPF gave me a great place to start," said Thomas. "I ran the same jobs with z/XPF that the other profiler had monitored. Using the ISPF panels provided with z/XPF, I requested z/XPF to schedule a Profile Data Capture session. It was easy. I took the future start option, entered my job name, and since I'm a heavy user of excp, set the excp sampling interval. When the job was complete, I again went to the panel and requested reports from the data that was collected. The Top CPU Consumer's Report was where I started."

Thomas said that he and his team had already spent two or three weeks off and on going over their modules with the other profiler. "We were pleasantly surprised," said Thomas. "After only a few days of using the CPU report and detailed listings from z/XPF, we were able to uncover the causes of our performance issues and implement the necessary changes."

Problem #2

Thomas's colleague and co-developer Rich Morse also found benefits from using z/XPF in problem resolution.

"I encountered a complex Locking problem that was causing delays in processing in our IAM/RLS file server address space, due to contention for a z/OS Local Lock. It was resulting from a high level of I/O activity from concurrently running jobs and CICS regions," said Morse. "One of our customers had also reported elongated run times for some critical batch work that was resulting in their not meeting schedule, and which caused delays in other critical work."

Initially, Morse tried to use the older profiling tool to determine why the local lock contention was occurring in z/OS, but he was unable to quantify how much of the delay was being caused by the contention.

"I decided to try the new z/XPF instead," said Morse. "z/XPF provided the details to quantify the amount of the delay that was being caused by the lock contention. It also helped to identify the functions that were causing the contention.....The z/XPF Work Unit Summary Report has a section on delays caused by contention of various resources." z/XPF is also the only tool of its kind that can report on Lock Suspends. Morse shared this example:

"We had already spent a few weeks speculating about what was causing the delay, and trying to use the older tool to determine what portion of the delay was due to lock contention," said Morse. "When we tried z/XPF, we discovered that z/XPF had this capability, and we were able to project from our internal testing and the z/XPF reports what portion of the delay was caused by lock contention. We accomplished this in a couple of days."

Because z/XPF is an event-driven profiler, it is able to obtain all of its information on what is occurring within an address space from outside of that address space, looking at the address space as a processor would. This contrasts with the "state" measurement techniques traditionally employed by other profiling products that actually stop the target address space for measurement-taking, which in turn extends elapsed time and can mask underlying causes which then adversely impact trouble-shooting and time to problem resolution.

"This is a great product, and it's fast becoming an important tool in our portfolio," said Morse. "It also comes with the enthusiasm, ease of contact, and the high level of support that ColeSoft provides that makes z/XPF easy to use."

Mary Shacklett

Transworld Data

Twd_transworld@msn.com