Print | Rate this content

HP StorageWorks Tape Storage - Performance Troubleshooting

Information

Information in this document is taken from the HP Support document titled HP Surestore and StorageWorks - Performance Troubleshooting and Using Performance Assessment Tools.

Tape drives are getting faster and it is not always easy to make the most of their performance in complex systems. Debugging performance problems can be tricky at the best of times. This document aims to provide the knowledge to help identify where performance related problems are in a tape backup system and gives guidance on how they can be fixed.

Note that this document applies equally to backup systems whatever type of tape drive is used. However, the examples used refer to HP's Ultrium and SDLT tape drives, as these are the fastest and most challenging to configure correctly.

Details

Performance will always be limited by one or more bottlenecks in the system, of which the tape drive is just one part. The goal is to make the tape drive the bottleneck. That way the system will achieve the performance figures as advertised on the drive's spec sheet. This puts the emphasis on the rest of the system. The flow of data throughout the system must be fast enough to provide the tape drive with data at its desired rates.

High-speed tape drives such as the HP Ultrium 460, 920, 960, and 1840 are so fast this can be a difficult goal to achieve. If backup performance is not matching the data sheet of the tape drive, then there is a bottleneck somewhere else in the system.

How fast should a tape drive go?

The answer to this is determined by multiplying the native data rate of the tape drive by the compression ratio of the data being backed up. For example, 2:1 compressible data on an HP Ultrium 460 (30 MB/sec native) should achieve a data rate of 60 MB/sec.

Note that the internal pipeline of the tape drive might limit the ultimate data rate. In the case of the HP Ultrium 460, this is around 90 MB/sec. This is unlikely to be a limiter except in the fastest of systems.

Compression ratio can vary a great deal.

More information regarding compression is available in the HP Support document titled HP Surestore/StorageWorks- Data Compression Information. Click here to view "HP Surestore/StorageWorks- Data Compression Information" (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=lpg50244) .

For simplicity, data is assumed to compress at 2:1, which can be used as a working default. A better alternative is to use the actual compression ratio of data, which can be found by using Library and Tape Tools (LTT) to calculate it directly from the compression logs of the tape drive. With the expected data rate in mind, the next step is to measure the other parts of the system to ensure they can transfer data at least as fast as that.

Tape drive performance

Troubleshooting

This is a good place to start, as the tape drive is often incorrectly considered to be the bottleneck. This can be established with (or otherwise) certainty by checking how fast the server can access the tape drive independent of the disk subsystem and application using Devperf in newer versions of L&TT. It works by transferring pre-loaded data directly from the memory of the server to the tape drive. It can also use different compression ratios.

If the expected data rates cannot be achieved, then any of the following factors may be apparent:

  • Cabling from server to tape drive in case of SCSI:

    • Should be good quality Ultra 160/320 SCSI cabling.

    • Should be within SCSI specs - watch for length.

    • Should be terminated properly.

    • Check for any bent pins or badly seated connectors.

  • Other peripherals on the bus - high-speed tape drives must have exclusive access to the HBA/bus to achieve their potential bandwidth. Also, SCSI devices negotiate to match the lowest speed device on the bus - a legacy device can bring everything else on the bus to its knees. Therefore the faster tape drives should be connected to a dedicated bus.

  • HBA/OS/driver compatibility - check that the configuration is supported as per EBS and that the drivers are up-to-date. For information on EBS, see the HP StorageWorks Enterprise Backup Solutions (EBS)- Overview & features web page. Click here to go the "HP StorageWorks Enterprise Backup Solutions (EBS)- Overview & features" web page (http://h18000.www1.hp.com/products/storageworks/ebs/) .

  • Damaged or worn media can cause excessive use of tape or physical retries; hence, a drop in performance is probable. Either L&TT can be used to check for poor media performance or try a new tape.

  • Dirty heads - the "clean light" should be on if this is the case. HP tape drives have self-cleaning mechanisms to virtually eliminate dirty heads but a cleaning cartridge with good life can still be used to make sure.

Disk and file system performance

There is a significant difference between the raw data rate of a disk and the rate at which files can be read from a file system. This is because traversing a file system requires multiple, random disk accesses; whereas, a continuous read is limited only by the optimized data rates of the disk. The difference between these two modes becomes more significant as file size reduces. For file systems where the files are typically smaller than 64 KB, sequential (or disk image) backups should be considered to achieve the data rates supported by high speed tape drives.

To measure the data rate from the disk subsystem, use L&TT Sysperf, and select the method used by the backup - file-by-file or sequential. This tool gives the actual read transfer rate that the disk system can provide, which can be compared with the expected data rate. If the disk subsystem does not transfer data fast enough, then it is a bottleneck. This could be down to any of the following factors:

  • File size - less than 64 KB can reduce the average data rate significantly.

    • User data is often made up of files less than 64 KB.

    • Also, deep file structures can increase the number of directory accesses.

  • The number of disk drives in a RAID subsystem - recommend at least one drive for each 4 MB/s of native tape data rate to support file-by-file backup.

  • RAID configuration - recommend RAID 5 with read cache enabled (best combination of data integrity and performance).

  • Access to the disk system is slow.

    • Internal disks - Need to be using a PCI-X backplane.

    • Direct attach - Need at least Ultra 160/320 SCSI or FC (2 GB).

    • Network - Need Gigabit Ethernet with large transfer sizes.

    • SAN - Check the bandwidth of the fabric - allowing for other traffic.

      • Watch out for fibre to SCSI bridges. They can be a bottleneck.

  • File system fragmentation - will cause additional seeks. Defragment before backing up.

  • Software compression - do not use - it is not as fast as the compression hardware in the tape drive. It also requires significantly more processing power.

  • Are any other applications accessing the disk subsystem?

    • Avoid any concurrent access - even to a different LUN (on the same subsystem).

    • Watch out for virus scans, often scheduled at similar times to backups.

As changes are made to the disk subsystem, re-run Sysperf to measure its performance.

For more detailed information about how the file system and disk configuration affect backup performance of the SDLT or Ultrium, see the HP Support document titled HP Surestore and StorageWorks - How File System and Disk Configuration Affect Backup Performance with a DLT or Ultrium Drive. Click here to view "HP Surestore and StorageWorks - How File System and Disk Configuration Affect Backup Performance with a DLT or Ultrium Drive" (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=lpg50032) .

Backup application configuration

At this point it has been shown that neither the disk subsystem or tape drive are the performance limiters in the system. The next, most likely contender is the backup application itself.

These are HP's recommendations for achieving high performance with backup applications:

  • For general backup, use an HP recommended backup application. Native apps (for example, tar, cpio on UNIX, NTBackup on Microsoft Windows) are not high performance.

  • For database applications (for example, Microsoft SQL, Microsoft Exchange, Oracle), use the backup functionality provided by those applications as they are tuned to make best use of their data structures.

  • Use large (SCSI) transfer sizes (greater than 64 K). The bigger the better.

  • Increase the system memory allocated to backup if possible.

  • Data compression - HP's tape drives do this in hardware at very high speed whereas it is much slower in software on the server. Also, compressing data twice tends to give non-optimal compression ratios.

    • Make sure any software data compression is turned off.

  • Use multi-threading (concurrency) if possible. This allows multiple backups to be interleaved to the tape, thus reducing the effect of disk seeks for each one.

    • Note this can have an impact on restore time as a particular file set is interleaved amongst other data.

  • If the system consists of small files (less than 64 KB) consider image/sequential backups.

  • Certain backup applications recommend the use of their own device drivers (like Symantec Backup exec, Netbackup), some of them does not require device drivers installed at the operating system level, check that compatibility before installing the device drivers.

For information about HP's recommendations regarding compatibility, see the HP Backup Compatibility web page. Click here to go to the "HP Backup Compatibility" web page (http://www.hp.com/products1/storage/compatibility/index.html) .

Server capacity and capability

The server is central to the backup process as it runs the backup software and the data is passed into and out of the server's main memory as it is read from disk subsystem and written to tape.

To see if the server is a possible bottleneck, use one of the readily available system monitoring tools, such as PerfMon, while performing a backup. Ensure there are still resources free. Watch for the following:

  • CPU bandwidth

  • Memory capacity

  • IO bandwidth

HP recommends:

  • Multi-processor or single 1+GHz processor with at least 512 MB of system memory.

  • 64 bit/66 MHz PCI and HBAs. PCI-X (133 MHz) is even better.

  • Use separate PCI buses for the disk and tape HBAs if possible. Total PCI bandwidth needs to be more than double the backup rate.

  • Ensuring synchronous negotiation is turned on in the SCSI HBA.

  • Not running any other applications during the backup. Watch for the following:

    • Virus scans

    • CPU intensive screen savers

Also, check for the following:

Server set-up

One thing to watch for is the allocation of IRQs for the HBAs for both the tape drive and the disk subsystem. If either of them share an IRQ (interrupt) with something else, then there will be additional overhead in the management of interrupts. This can reduce performance significantly. The best way to find out how the IRQs are allocated is to do so in the BIOS. This is also the best place to change them if it is desired to do it manually. Do not use the Windows device manager as this does not always give accurate IRQ allocation.

There are a number of possible workarounds:

  • Re-allocate the IRQs in the BIOS.

    • Force slow hardware to share IRQs (for example, serial ports).

    • Allow fast hardware to have its own IRQ.

  • Re-arrange the hardware to use different slots. With plug and play this can be a bit of a lottery. Use the manual method (BIOS allocation) if possible.

  • Remove the hardware of the shared interrupt (assuming that the hardware is not required).

  • Disable the hardware of the shared interrupt (assuming that the hardware is not required).

    • Can be done in the BIOS or device manager.

    • Has the advantage of not need to take the server apart.

    • Unused serial ports are a good candidate.

One other thing to watch for on Windows operating systems, is the Removable Storage Manager (RSM). This system service polls the drive regularly, which interrupts the flow of data and therefore reduces the overall throughput. The workaround is to disable the service, but beware - re-connecting any storage devices will automatically turn RSM back on again, including devices via USB.

Make sure to disable the RSM through the Registry hack in the device drivers (for example LTO.SYS, in the Robot drivers for MSL G3 libraries).

HBA settings are set as per the recommendations in the EBS Design Guide. For more information about EBS, see the HP StorageWorks Enterprise Backup Solutions (EBS)- Overview & features web page. Click here to go to the "HP StorageWorks Enterprise Backup Solutions (EBS)- Overview & features" web page (http://h18000.www1.hp.com/products/storageworks/ebs/) .

How important is performance?

The answer to this question depends on what is needed. If after following the advice above, the transfer rate of the tape drive still cannot be reached - and this really can be a challenge with the high transfer rates supported by HP's Ultrium and SDLT tape drives - then it is not necessarily a problem unless that performance is needed. The key requirement may just be that the backup completes within the available time window. As upgrades are done on the server, disk system, network or other parts of the whole system, better use of the performance of the tape drive will be possible. If the maximum performance is not being achieved at the moment, at least feel good about investing in a backup solution that will match the system well into the future.

The following table shows the native data transfer rates of a range of HP's currently supported tape drives. HP's Ultrium drives also have a feature called Adaptive Tape Speed (ATS), which matches the speed of the tape to the data rate of the server. It is fully variable between its lowest and highest rates and eliminates tape repositions within this range - thus reducing wear even if the server cannot keep up.

NOTE: The data compression engines of the VS80, VS160 and DLT80 products do not have a linear relationship between compression ratio and transfer rate. It is not recommended to use HPTapeperf with these products for performance measurements.

If a data rate can be achieve above the minimum ATS speed, then the drive will stream. For example, it will be operating at minimum wear rates. As a last resort, turn data compression off. This would be more likely to stream, although it would use more tape for the same amount of data.

Provide feedback

Please rate the information on this page to help us improve our content. Thank you!