Tracing volume groups with an HBA in linux

by Jesse 9. March 2009 15:32

I've been off in Linux land for a while and I've ran into HBAs, fiber channels, SANs and volume groups.  Honestly, they're a billion times more complex than they EVER need to be but if you are unlucky enough to run across such things, here's how I did it, do it and will continue to find it was the professor in the library with the candlestick.

First, log into your linux box and head over to /proc/scsi.  In here you should see a folder with a semi-recognizable driver, in this case a qlogic 2000.

[root@mybox scsi]# ls
device_info  qla2xxx  scsi  sg

If you want to know EXACTLY what card is installed, a lspci -v will display this for you ... in this case, ISP2312.

10:01.0 Fibre Channel: QLogic Corp. ISP2312-based 2Gb Fibre Channel to PCI-X HBA (rev 02)
        Subsystem: QLogic Corp. Unknown device 0100
        Flags: bus master, 66MHz, medium devsel, latency 128, IRQ 130
        I/O ports at 5000 [size=256]
        Memory at fdff0000 (64-bit, non-prefetchable) [size=4K]
        [virtual] Expansion ROM at d1000000 [disabled] [size=128K]
        Capabilities: [44] Power Management version 2
        Capabilities: [4c] PCI-X non-bridge device
        Capabilities: [54] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
        Capabilities: [64] #06 [0080]

So anyway, jump into your HBA's driver directory and you'll find a file or two, probably just numbers, 0, 1, 2, etc.  In my case, I have 1 and 2.  Run a more 1 on this and you'll get a dump of information similar to this.  I've snipped the top because it's irrelevant.

... 

SCSI Device Information:
scsi-qla0-adapter-node=200000145e249594;
scsi-qla0-adapter-port=210000145e249594;
scsi-qla0-target-0=5000402101fc137a;
scsi-qla0-target-1=5000402201fc137a;
scsi-qla0-target-2=5000402001fc137a;
scsi-qla0-target-3=5000402301fc137a;

FC Port Information:
scsi-qla0-port-0=2001000402fc137a:5000402101fc137a:010000:81;
scsi-qla0-port-1=2101000402fc137a:5000402201fc137a:010400:82;
scsi-qla0-port-2=2001000402fc137a:5000402001fc137a:020400:83;
scsi-qla0-port-3=2101000402fc137a:5000402301fc137a:020000:84;
scsi-qla0-port-4=200000145e249595:210000145e249595:020600:1;

SCSI LUN Information:
(Id:Lun)  * - indicates lun is not registered with the OS.
( 0: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:81 00
( 0: 8): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:15): Total reqs 80236, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:17): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:21): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:81 00
( 0:25): Total reqs 80253, Pending reqs 0, flags 0x0, 0:0:81 00
( 1: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:82 00
( 1: 1): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:11): Total reqs 80236, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:19): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:21): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 1:25): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:82 00
( 2: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:83 00
( 2: 8): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:15): Total reqs 80257, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:17): Total reqs 80253, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:21): Total reqs 80256, Pending reqs 0, flags 0x0, 0:0:83 00
( 2:25): Total reqs 80262, Pending reqs 0, flags 0x0, 0:0:83 00
( 3: 0): Total reqs 3, Pending reqs 0, flags 0x0*, 0:0:84 00
( 3: 1): Total reqs 80312, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:11): Total reqs 80236, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:19): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:21): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:84 00
( 3:25): Total reqs 80235, Pending reqs 0, flags 0x0, 0:0:84 00

This tells us a bunch of things, useful things we'll need later.  Most importantly the adapter-node/adapter-port IDs that I have bolded.  This is the ID we need to find out WHAT exactly this server has access to.  Out on the SAN, find this information and which volumes the ID has access to.  Those volumes should have a serial number of some kind which is the next step.  This part I can't help you with because every SAN is different, but I can tell you the SAN I am working with gave me an ID of 603BD80A that is 1000.0GB -- according to my linux server, I don't have one of those volumes mounted of that size, a df -h tells me this.  So what the hell?  There's a volume group that's combined multiple volumes into 1, hints the name v that happens to be over 1000GB, as vgdisplay tells me so...

  [root@mybox ~]# vgdisplay
  --- Volume group ---
  VG Name               datagroup
  System ID
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               2.73 TB
  PE Size               4.00 MB
  Total PE              715292
  Alloc PE / Size       715292 / 2.73 TB
  Free  PE / Size       0 / 0
  VG UUID               pY6jLp-rZcZ-MVQS-BKH2-XmHx-xJhb-5BpxSa


Ok, so what?  I haven't proved anything, and worse, what if there's multiple groups that have bigger than 1000GB?  Next step ... use that serial number we got earlier off our SAN to determine WHAT "drives" are what.  Time to hit the multipath -ll ...and look for our serial number

mpath13 (36000402001fc137a603bd80a00000000)
[size=931 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=1][active]
 \_ 2:0:2:25 sdaj 66:48  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 1:0:0:25 sdf  8:80   [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 1:0:2:25 sdp  8:240  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 2:0:0:25 sdz  65:144 [active][ready]

Sweet, we got a hit and its the right size.  So multipath13 is really sdaj, sdf, sdp and sdz.  Ok, so what?  Call on the powers of pvdisplay to tell you where it's really living...

[root@mybox ~]# pvdisplay /dev/sdaj
  --- Physical volume ---
  PV Name               /dev/dm-15
  VG Name               datagroup
  PV Size               931.39 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              238437
  Free PE               0
  Allocated PE          238437
  PV UUID               EUt6gs-8AIF-sAXe-q2MH-3L27-v57p-P6aESM

There you have it, it IS in fact living inside your datagroup volume group -- but what about the other drives sdf, sdp and sdz?  Do another pvdisplay on THOSE and notice the PV UUIDs are the same...

[root@mybox ~]# pvdisplay /dev/sdf
  --- Physical volume ---
  PV Name               /dev/dm-15
  VG Name               datagroup
  PV Size               931.39 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              238437
  Free PE               0
  Allocated PE          238437
  PV UUID               EUt6gs-8AIF-sAXe-q2MH-3L27-v57p-P6aESM

This come in very handy when doing an IO check and noticing a high state of wait on a particular drive and pvdisplay works on those as well.  In order to get the whole volume group, we would have to trace back the rest of them to complete the picture to get the full 3000GB assigned on the SAN.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags:

Linux

Comments

Powered by BlogEngine.NET 1.4.5.0
Theme by Mads Kristensen

About the author

Like the description says, at my core, I'm a scientist and engineer.  I came from humble beginnings on a 486DX2 Packard Hell playing doom2 on IPX to in a small time retail shop and got into hardware (ISO layers FTW!) and it was all downhill from there.  I'm infinitely curious about almost everything and always wanting to know.

According to personality tests (real ones) I classify under "Rational" more specifically, a Fieldmarshal.  I think there's something to that.

Some of the stuff I'm currently into/researching...

Sitefinity

Ninject

Subsonic 

Currently working on ...
i did the hundred


and some extra stuff

Disclaimer

The opinions expressed herein are my own personal opinions and do not represent my employer's, their brother, their dog, cat, ferret nor gold fish's view in anyway.  At all.  Ever.

© Copyright 2007-2009