I just sent an email to a customer about debugging slowness that might be caused by a slow SAN. Here's the salient excerpt (for my own reference later):
I'd suggest that you run this command on the database server: iostat -x 5 (for Debian, this command can be installed by doing "apt-get install sysstat") This prints reports spaced over 5 second intervals. Each report looks like this: Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util hda01 0.20 6.00 35.80 49.00 5422.40 3352.40 103.48 39.11 458.56 11.78 99.90 The example comes from a poorly performing SAN device from another customer. "svctime" is the average time it took to service a request after it was popped off the disk's request queue. "await" is the average response time in ms of an IO request to the device, including time spent waiting in the queue. As I understand it, the closer "svctime" is to "await", the better the device is performing; the further away these two numbers are, the worse. If you experience worse performance after the site begins running than it had been previously, and iostat tells you lots of requests for disk accesses are spent waiting in the queue, you'll want to tune your SAN or figure out a different strategy for serving the database.