Archive-Team
/
zfs
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 103 Wiki Activity

vdev_id: Support daisy-chained JBODs in multipath mode 

Within function sas_handler() userspace commands like
'/usr/sbin/multipath' have been replaced with sourcing
device details from within sysfs which reduced a
significant amount of overhead and processing time.
Multiple JBOD enclosures and their order are sourced
from the bsg driver (/sys/class/enclosure) to isolate
chassis top-level expanders, which are then dynamically
indexed based on host channel of the multipath subordinate
disk member device being processed. Additionally added a
"mixed" mode for slot identification for environments where
a ZFS server system may contain SAS disk slots where there
is no expander (direct connect to HBA) while an attached
external JBOD with an expander have different slot identifier
methods.

How Has This Been Tested?
~~~~~~~~~~~~~~~~~~~~~~~~~

Testing was performed on a AMD EPYC based dual-server
high-availability multipath environment with multiple
HBAs per ZFS server and four SAS JBODs. The two primary
JBODs were multipath/cross-connected between the two
ZFS-HA servers. The secondary JBODs were daisy-chained
off of the primary JBODs using aligned SAS expander
channels (JBOD-0 expanderA--->JBOD-1 expanderA,
          JBOD-0 expanderB--->JBOD-1 expanderB, etc).
Pools were created, exported and re-imported, imported
globally with 'zpool import -a -d /dev/disk/by-vdev'.
Low level udev debug outputs were traced to isolate
and resolve errors.

Result:
~~~~~~~

Initial testing of a previous version of this change
showed how reliance on userspace utilities like
'/usr/sbin/multipath' and '/usr/bin/lsscsi' were
exacerbated by increasing numbers of disks and JBODs.
With four 60-disk SAS JBODs and 240 disks the time to
process a udevadm trigger was 3 minutes 30 seconds
during which nearly all CPU cores were above 80%
utilization. By switching reliance on userspace
utilities to sysfs in this version, the udevadm
trigger processing time was reduced to 12.2 seconds
and negligible CPU load.

This patch also fixes few shellcheck complains.

Reviewed-by: Gabriel A. Devenyi <gdevenyi@gmail.com>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Co-authored-by: Jeff Johnson <jeff.johnson@aeoncomputing.com>
Signed-off-by: Jeff Johnson <jeff.johnson@aeoncomputing.com>
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Closes #11526
This commit is contained in:
Arshad Hussain 2021-02-10 02:34:09 +05:30 committed by Tony Hutter
parent 47ada9e5ed
commit bdd43a2396
1 changed files with 283 additions and 115 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

398
cmd/vdev_id/vdev_id
View File

@ -79,6 +79,34 @@
# channel 86:00.0 1 A # channel 86:00.0 1 A
# channel 86:00.0 0 B # channel 86:00.0 0 B
# #
# # Example vdev_id.conf - multipath / multijbod-daisychaining
# #
#
# multipath yes
# multijbod yes
#
# # PCI_ID HBA PORT CHANNEL NAME
# channel 85:00.0 1 A
# channel 85:00.0 0 B
# channel 86:00.0 1 A
# channel 86:00.0 0 B
# #
# # Example vdev_id.conf - multipath / mixed
# #
#
# multipath yes
# slot mix
#
# # PCI_ID HBA PORT CHANNEL NAME
# channel 85:00.0 3 A
# channel 85:00.0 2 B
# channel 86:00.0 3 A
# channel 86:00.0 2 B
# channel af:00.0 0 C
# channel af:00.0 1 C
# # # #
# # Example vdev_id.conf - alias # # Example vdev_id.conf - alias
# # # #
@ -92,9 +120,10 @@ PATH=/bin:/sbin:/usr/bin:/usr/sbin
CONFIG=/etc/zfs/vdev_id.conf CONFIG=/etc/zfs/vdev_id.conf
PHYS_PER_PORT= PHYS_PER_PORT=
DEV= DEV=
MULTIPATH=
TOPOLOGY= TOPOLOGY=
BAY= BAY=
ENCL_ID=""
UNIQ_ENCL_ID=""
usage() { usage() {
cat << EOF cat << EOF
@ -107,6 +136,7 @@ Usage: vdev_id [-h]
-e Create enclose device symlinks only (/dev/by-enclosure) -e Create enclose device symlinks only (/dev/by-enclosure)
-g Storage network topology [default="$TOPOLOGY"] -g Storage network topology [default="$TOPOLOGY"]
-m Run in multipath mode -m Run in multipath mode
-j Run in multijbod mode
-p number of phy's per switch port [default=$PHYS_PER_PORT] -p number of phy's per switch port [default=$PHYS_PER_PORT]
-h show this summary -h show this summary
EOF EOF
@ -117,12 +147,12 @@ map_slot() {
LINUX_SLOT=$1 LINUX_SLOT=$1
CHANNEL=$2 CHANNEL=$2
MAPPED_SLOT=`awk "\\$1 == \"slot\" && \\$2 == ${LINUX_SLOT} && \ MAPPED_SLOT=$(awk '$1 == "slot" && $2 == "${LINUX_SLOT}" && \
\\$4 ~ /^${CHANNEL}$|^$/ { print \\$3; exit }" $CONFIG` $4 ~ /^${CHANNEL}$|^$/ { print $3; exit}' $CONFIG)
if [ -z "$MAPPED_SLOT" ] ; then if [ -z "$MAPPED_SLOT" ] ; then
MAPPED_SLOT=$LINUX_SLOT MAPPED_SLOT=$LINUX_SLOT
fi fi
printf "%d" ${MAPPED_SLOT} printf "%d" "${MAPPED_SLOT}"
} }
map_channel() { map_channel() {
@ -132,40 +162,120 @@ map_channel() {
case $TOPOLOGY in case $TOPOLOGY in
"sas_switch") "sas_switch")
MAPPED_CHAN=`awk "\\$1 == \"channel\" && \\$2 == ${PORT} \ MAPPED_CHAN=$(awk -v port="$PORT" \
{ print \\$3; exit }" $CONFIG` '$1 == "channel" && $2 == ${PORT} \
{ print $3; exit }' $CONFIG)
;; ;;
"sas_direct"|"scsi") "sas_direct"|"scsi")
MAPPED_CHAN=`awk "\\$1 == \"channel\" && \ MAPPED_CHAN=$(awk -v pciID="$PCI_ID" -v port="$PORT" \
\\$2 == \"${PCI_ID}\" && \\$3 == ${PORT} \ '$1 == "channel" && $2 == pciID && $3 == port \
{ print \\$4; exit }" $CONFIG` {print $4}' $CONFIG)
;; ;;
esac esac
printf "%s" ${MAPPED_CHAN} printf "%s" "${MAPPED_CHAN}"
}
get_encl_id() {
set -- $(echo $1)
count=$#
i=1
while [ $i -le $count ] ; do
d=$(eval echo '$'{$i})
id=$(cat "/sys/class/enclosure/${d}/id")
ENCL_ID="${ENCL_ID} $id"
i=$((i + 1))
done
}
get_uniq_encl_id() {
for uuid in ${ENCL_ID}; do
found=0
for count in ${UNIQ_ENCL_ID}; do
if [ $count = $uuid ]; then
found=1
break
fi
done
if [ $found -eq 0 ]; then
UNIQ_ENCL_ID="${UNIQ_ENCL_ID} $uuid"
fi
done
}
# map_jbod explainer: The bsg driver knows the difference between a SAS
# expander and fanout expander. Use hostX instance along with top-level
# (whole enclosure) expander instances in /sys/class/enclosure and
# matching a field in an array of expanders, using the index of the
# matched array field as the enclosure instance, thereby making jbod IDs
# dynamic. Avoids reliance on high overhead userspace commands like
# multipath and lsscsi and instead uses existing sysfs data. $HOSTCHAN
# variable derived from devpath gymnastics in sas_handler() function.
map_jbod() {
DEVEXP=$(ls -l "/sys/block/$DEV/device/" | grep enclos | awk -F/ '{print $(NF-1) }')
DEV=$1
# Use "set --" to create index values (Arrays)
set -- $(ls -l /sys/class/enclosure | grep -v "^total" | awk '{print $9}')
# Get count of total elements
JBOD_COUNT=$#
JBOD_ITEM=$*
# Build JBODs (enclosure) id from sys/class/enclosure/<dev>/id
get_encl_id "$JBOD_ITEM"
# Different expander instances for each paths.
# Filter out and keep only unique id.
get_uniq_encl_id
# Identify final 'mapped jbod'
j=0
for count in ${UNIQ_ENCL_ID}; do
i=1
j=$((j + 1))
while [ $i -le $JBOD_COUNT ] ; do
d=$(eval echo '$'{$i})
id=$(cat "/sys/class/enclosure/${d}/id")
if [ "$d" = "$DEVEXP" ] && [ $id = $count ] ; then
MAPPED_JBOD=$j
break
fi
i=$((i + 1))
done
done
printf "%d" "${MAPPED_JBOD}"
} }
sas_handler() { sas_handler() {
if [ -z "$PHYS_PER_PORT" ] ; then if [ -z "$PHYS_PER_PORT" ] ; then
PHYS_PER_PORT=`awk "\\$1 == \"phys_per_port\" \ PHYS_PER_PORT=$(awk '$1 == "phys_per_port" \
{print \\$2; exit}" $CONFIG` {print $2; exit}' $CONFIG)
fi fi
PHYS_PER_PORT=${PHYS_PER_PORT:-4} PHYS_PER_PORT=${PHYS_PER_PORT:-4}
if ! echo $PHYS_PER_PORT | grep -q -E '^[0-9]+$' ; then
if ! echo "$PHYS_PER_PORT" | grep -q -E '^[0-9]+$' ; then
echo "Error: phys_per_port value $PHYS_PER_PORT is non-numeric" echo "Error: phys_per_port value $PHYS_PER_PORT is non-numeric"
exit 1 exit 1
fi fi
if [ -z "$MULTIPATH_MODE" ] ; then if [ -z "$MULTIPATH_MODE" ] ; then
MULTIPATH_MODE=`awk "\\$1 == \"multipath\" \ MULTIPATH_MODE=$(awk '$1 == "multipath" \
{print \\$2; exit}" $CONFIG` {print $2; exit}' $CONFIG)
fi
if [ -z "$MULTIJBOD_MODE" ] ; then
MULTIJBOD_MODE=$(awk '$1 == "multijbod" \
{print $2; exit}' $CONFIG)
fi fi
# Use first running component device if we're handling a dm-mpath device # Use first running component device if we're handling a dm-mpath device
if [ "$MULTIPATH_MODE" = "yes" ] ; then if [ "$MULTIPATH_MODE" = "yes" ] ; then
# If udev didn't tell us the UUID via DM_NAME, check /dev/mapper # If udev didn't tell us the UUID via DM_NAME, check /dev/mapper
if [ -z "$DM_NAME" ] ; then if [ -z "$DM_NAME" ] ; then
DM_NAME=`ls -l --full-time /dev/mapper | DM_NAME=$(ls -l --full-time /dev/mapper |
awk "/\/$DEV$/{print \\$9}"` grep "$DEV"$ | awk '{print $9}')
fi fi
# For raw disks udev exports DEVTYPE=partition when # For raw disks udev exports DEVTYPE=partition when
@ -175,28 +285,41 @@ sas_handler() {
# we have to append the -part suffix directly in the # we have to append the -part suffix directly in the
# helper. # helper.
if [ "$DEVTYPE" != "partition" ] ; then if [ "$DEVTYPE" != "partition" ] ; then
PART=`echo $DM_NAME | awk -Fp '/p/{print "-part"$2}'` PART=$(echo "$DM_NAME" | awk -Fp '/p/{print "-part"$2}')
fi fi
# Strip off partition information. # Strip off partition information.
DM_NAME=`echo $DM_NAME | sed 's/p[0-9][0-9]*$//'` DM_NAME=$(echo "$DM_NAME" | sed 's/p[0-9][0-9]*$//')
if [ -z "$DM_NAME" ] ; then if [ -z "$DM_NAME" ] ; then
return return
fi fi
# Get the raw scsi device name from multipath -ll. Strip off # Utilize DM device name to gather subordinate block devices
# leading pipe symbols to make field numbering consistent. # using sysfs to avoid userspace utilities
DEV=`multipath -ll $DM_NAME | DMDEV=$(ls -l --full-time /dev/mapper | grep $DM_NAME |
awk '/running/{gsub("^[|]"," "); print $3 ; exit}'` awk '{gsub("../", " "); print $NF}')
# Use sysfs pointers in /sys/block/dm-X/slaves because using
# userspace tools creates lots of overhead and should be avoided
# whenever possible. Use awk to isolate lowest instance of
# sd device member in dm device group regardless of string
# length.
DEV=$(ls "/sys/block/$DMDEV/slaves" | awk '
{ len=sprintf ("%20s",length($0)); gsub(/ /,0,str); a[NR]=len "_" $0; }
END {
asort(a)
print substr(a[1],22)
}')
if [ -z "$DEV" ] ; then if [ -z "$DEV" ] ; then
return return
fi fi
fi fi
if echo $DEV | grep -q ^/devices/ ; then if echo "$DEV" | grep -q ^/devices/ ; then
sys_path=$DEV sys_path=$DEV
else else
sys_path=`udevadm info -q path -p /sys/block/$DEV 2>/dev/null` sys_path=$(udevadm info -q path -p "/sys/block/$DEV" 2>/dev/null)
fi fi
# Use positional parameters as an ad-hoc array # Use positional parameters as an ad-hoc array
@ -206,84 +329,104 @@ sas_handler() {
# Get path up to /sys/.../hostX # Get path up to /sys/.../hostX
i=1 i=1
while [ $i -le $num_dirs ] ; do
d=$(eval echo \${$i}) while [ $i -le "$num_dirs" ] ; do
d=$(eval echo '$'{$i})
scsi_host_dir="$scsi_host_dir/$d" scsi_host_dir="$scsi_host_dir/$d"
echo $d | grep -q -E '^host[0-9]+$' && break echo "$d" | grep -q -E '^host[0-9]+$' && break
i=$(($i + 1)) i=$((i + 1))
done done
if [ $i = $num_dirs ] ; then # Lets grab the SAS host channel number and save it for JBOD sorting later
HOSTCHAN=$(echo "$d" | awk -F/ '{ gsub("host","",$NF); print $NF}')
if [ $i = "$num_dirs" ] ; then
return return
fi fi
PCI_ID=$(eval echo \${$(($i -1))} | awk -F: '{print $2":"$3}') PCI_ID=$(eval echo '$'{$((i -1))} | awk -F: '{print $2":"$3}')
# In sas_switch mode, the directory four levels beneath # In sas_switch mode, the directory four levels beneath
# /sys/.../hostX contains symlinks to phy devices that reveal # /sys/.../hostX contains symlinks to phy devices that reveal
# the switch port number. In sas_direct mode, the phy links one # the switch port number. In sas_direct mode, the phy links one
# directory down reveal the HBA port. # directory down reveal the HBA port.
port_dir=$scsi_host_dir port_dir=$scsi_host_dir
case $TOPOLOGY in case $TOPOLOGY in
"sas_switch") j=$(($i + 4)) ;; "sas_switch") j=$((i + 4)) ;;
"sas_direct") j=$(($i + 1)) ;; "sas_direct") j=$((i + 1)) ;;
esac esac
i=$(($i + 1)) i=$((i + 1))
while [ $i -le $j ] ; do while [ $i -le $j ] ; do
port_dir="$port_dir/$(eval echo \${$i})" port_dir="$port_dir/$(eval echo '$'{$i})"
i=$(($i + 1)) i=$((i + 1))
done done
PHY=`ls -d $port_dir/phy* 2>/dev/null | head -1 | awk -F: '{print $NF}'` PHY=$(ls -d "$port_dir"/phy* 2>/dev/null | head -1 | awk -F: '{print $NF}')
if [ -z "$PHY" ] ; then if [ -z "$PHY" ] ; then
PHY=0 PHY=0
fi fi
PORT=$(( $PHY / $PHYS_PER_PORT )) PORT=$((PHY / PHYS_PER_PORT))
# Look in /sys/.../sas_device/end_device-X for the bay_identifier # Look in /sys/.../sas_device/end_device-X for the bay_identifier
# attribute. # attribute.
end_device_dir=$port_dir end_device_dir=$port_dir
while [ $i -lt $num_dirs ] ; do
d=$(eval echo \${$i}) while [ $i -lt "$num_dirs" ] ; do
d=$(eval echo '$'{$i})
end_device_dir="$end_device_dir/$d" end_device_dir="$end_device_dir/$d"
if echo $d | grep -q '^end_device' ; then if echo "$d" | grep -q '^end_device' ; then
end_device_dir="$end_device_dir/sas_device/$d" end_device_dir="$end_device_dir/sas_device/$d"
break break
fi fi
i=$(($i + 1)) i=$((i + 1))
done done
# Add 'mix' slot type for environments where dm-multipath devices
# include end-devices connected via SAS expanders or direct connection
# to SAS HBA. A mixed connectivity environment such as pool devices
# contained in a SAS JBOD and spare drives or log devices directly
# connected in a server backplane without expanders in the I/O path.
SLOT= SLOT=
case $BAY in case $BAY in
"bay") "bay")
SLOT=`cat $end_device_dir/bay_identifier 2>/dev/null` SLOT=$(cat "$end_device_dir/bay_identifier" 2>/dev/null)
;;
"mix")
if [ $(cat "$end_device_dir/bay_identifier" 2>/dev/null) ] ; then
SLOT=$(cat "$end_device_dir/bay_identifier" 2>/dev/null)
else
SLOT=$(cat "$end_device_dir/phy_identifier" 2>/dev/null)
fi
;; ;;
"phy") "phy")
SLOT=`cat $end_device_dir/phy_identifier 2>/dev/null` SLOT=$(cat "$end_device_dir/phy_identifier" 2>/dev/null)
;; ;;
"port") "port")
d=$(eval echo \${$i}) d=$(eval echo '$'{$i})
SLOT=`echo $d | sed -e 's/^.*://'` SLOT=$(echo "$d" | sed -e 's/^.*://')
;; ;;
"id") "id")
i=$(($i + 1)) i=$((i + 1))
d=$(eval echo \${$i}) d=$(eval echo '$'{$i})
SLOT=`echo $d | sed -e 's/^.*://'` SLOT=$(echo "$d" | sed -e 's/^.*://')
;; ;;
"lun") "lun")
i=$(($i + 2)) i=$((i + 2))
d=$(eval echo \${$i}) d=$(eval echo '$'{$i})
SLOT=`echo $d | sed -e 's/^.*://'` SLOT=$(echo "$d" | sed -e 's/^.*://')
;; ;;
"ses") "ses")
# look for this SAS path in all SCSI Enclosure Services # look for this SAS path in all SCSI Enclosure Services
# (SES) enclosures # (SES) enclosures
sas_address=`cat $end_device_dir/sas_address 2>/dev/null` sas_address=$(cat "$end_device_dir/sas_address" 2>/dev/null)
enclosures=`lsscsi -g | \ enclosures=$(lsscsi -g | \
sed -n -e '/enclosu/s/^.* \([^ ][^ ]*\) *$/\1/p'` sed -n -e '/enclosu/s/^.* \([^ ][^ ]*\) *$/\1/p')
for enclosure in $enclosures; do for enclosure in $enclosures; do
set -- $(sg_ses -p aes $enclosure | \ set -- $(sg_ses -p aes "$enclosure" | \
awk "/device slot number:/{slot=\$12} \ awk "/device slot number:/{slot=\$12} \
/SAS address: $sas_address/\ /SAS address: $sas_address/\
{print slot}") {print slot}")
@ -298,42 +441,55 @@ sas_handler() {
return return
fi fi
CHAN=`map_channel $PCI_ID $PORT` if [ "$MULTIJBOD_MODE" = "yes" ] ; then
SLOT=`map_slot $SLOT $CHAN` CHAN=$(map_channel "$PCI_ID" "$PORT")
if [ -z "$CHAN" ] ; then SLOT=$(map_slot "$SLOT" "$CHAN")
return JBOD=$(map_jbod "$DEV")
if [ -z "$CHAN" ] ; then
return
fi
echo "${CHAN}"-"${JBOD}"-"${SLOT}${PART}"
else
CHAN=$(map_channel "$PCI_ID" "$PORT")
SLOT=$(map_slot "$SLOT" "$CHAN")
if [ -z "$CHAN" ] ; then
return
fi
echo "${CHAN}${SLOT}${PART}"
fi fi
echo ${CHAN}${SLOT}${PART}
} }
scsi_handler() { scsi_handler() {
if [ -z "$FIRST_BAY_NUMBER" ] ; then if [ -z "$FIRST_BAY_NUMBER" ] ; then
FIRST_BAY_NUMBER=`awk "\\$1 == \"first_bay_number\" \ FIRST_BAY_NUMBER=$(awk '$1 == "first_bay_number" \
{print \\$2; exit}" $CONFIG` {print $2; exit}' $CONFIG)
fi fi
FIRST_BAY_NUMBER=${FIRST_BAY_NUMBER:-0} FIRST_BAY_NUMBER=${FIRST_BAY_NUMBER:-0}
if [ -z "$PHYS_PER_PORT" ] ; then if [ -z "$PHYS_PER_PORT" ] ; then
PHYS_PER_PORT=`awk "\\$1 == \"phys_per_port\" \ PHYS_PER_PORT=$(awk '$1 == "phys_per_port" \
{print \\$2; exit}" $CONFIG` {print $2; exit}' $CONFIG)
fi fi
PHYS_PER_PORT=${PHYS_PER_PORT:-4} PHYS_PER_PORT=${PHYS_PER_PORT:-4}
if ! echo $PHYS_PER_PORT | grep -q -E '^[0-9]+$' ; then
if ! echo "$PHYS_PER_PORT" | grep -q -E '^[0-9]+$' ; then
echo "Error: phys_per_port value $PHYS_PER_PORT is non-numeric" echo "Error: phys_per_port value $PHYS_PER_PORT is non-numeric"
exit 1 exit 1
fi fi
if [ -z "$MULTIPATH_MODE" ] ; then if [ -z "$MULTIPATH_MODE" ] ; then
MULTIPATH_MODE=`awk "\\$1 == \"multipath\" \ MULTIPATH_MODE=$(awk '$1 == "multipath" \
{print \\$2; exit}" $CONFIG` {print $2; exit}' $CONFIG)
fi fi
# Use first running component device if we're handling a dm-mpath device # Use first running component device if we're handling a dm-mpath device
if [ "$MULTIPATH_MODE" = "yes" ] ; then if [ "$MULTIPATH_MODE" = "yes" ] ; then
# If udev didn't tell us the UUID via DM_NAME, check /dev/mapper # If udev didn't tell us the UUID via DM_NAME, check /dev/mapper
if [ -z "$DM_NAME" ] ; then if [ -z "$DM_NAME" ] ; then
DM_NAME=`ls -l --full-time /dev/mapper | DM_NAME=$(ls -l --full-time /dev/mapper |
awk "/\/$DEV$/{print \\$9}"` grep "$DEV"$ | awk '{print $9}')
fi fi
# For raw disks udev exports DEVTYPE=partition when # For raw disks udev exports DEVTYPE=partition when
@ -343,28 +499,28 @@ scsi_handler() {
# we have to append the -part suffix directly in the # we have to append the -part suffix directly in the
# helper. # helper.
if [ "$DEVTYPE" != "partition" ] ; then if [ "$DEVTYPE" != "partition" ] ; then
PART=`echo $DM_NAME | awk -Fp '/p/{print "-part"$2}'` PART=$(echo "$DM_NAME" | awk -Fp '/p/{print "-part"$2}')
fi fi
# Strip off partition information. # Strip off partition information.
DM_NAME=`echo $DM_NAME | sed 's/p[0-9][0-9]*$//'` DM_NAME=$(echo "$DM_NAME" | sed 's/p[0-9][0-9]*$//')
if [ -z "$DM_NAME" ] ; then if [ -z "$DM_NAME" ] ; then
return return
fi fi
# Get the raw scsi device name from multipath -ll. Strip off # Get the raw scsi device name from multipath -ll. Strip off
# leading pipe symbols to make field numbering consistent. # leading pipe symbols to make field numbering consistent.
DEV=`multipath -ll $DM_NAME | DEV=$(multipath -ll "$DM_NAME" |
awk '/running/{gsub("^[|]"," "); print $3 ; exit}'` awk '/running/{gsub("^[|]"," "); print $3 ; exit}')
if [ -z "$DEV" ] ; then if [ -z "$DEV" ] ; then
return return
fi fi
fi fi
if echo $DEV | grep -q ^/devices/ ; then if echo "$DEV" | grep -q ^/devices/ ; then
sys_path=$DEV sys_path=$DEV
else else
sys_path=`udevadm info -q path -p /sys/block/$DEV 2>/dev/null` sys_path=$(udevadm info -q path -p "/sys/block/$DEV" 2>/dev/null)
fi fi
# expect sys_path like this, for example: # expect sys_path like this, for example:
@ -377,44 +533,47 @@ scsi_handler() {
# Get path up to /sys/.../hostX # Get path up to /sys/.../hostX
i=1 i=1
while [ $i -le $num_dirs ] ; do
d=$(eval echo \${$i}) while [ $i -le "$num_dirs" ] ; do
d=$(eval echo '$'{$i})
scsi_host_dir="$scsi_host_dir/$d" scsi_host_dir="$scsi_host_dir/$d"
echo $d | grep -q -E '^host[0-9]+$' && break
i=$(($i + 1)) echo "$d" | grep -q -E '^host[0-9]+$' && break
i=$((i + 1))
done done
if [ $i = $num_dirs ] ; then if [ $i = "$num_dirs" ] ; then
return return
fi fi
PCI_ID=$(eval echo \${$(($i -1))} | awk -F: '{print $2":"$3}') PCI_ID=$(eval echo '$'{$((i -1))} | awk -F: '{print $2":"$3}')
# In scsi mode, the directory two levels beneath # In scsi mode, the directory two levels beneath
# /sys/.../hostX reveals the port and slot. # /sys/.../hostX reveals the port and slot.
port_dir=$scsi_host_dir port_dir=$scsi_host_dir
j=$(($i + 2)) j=$((i + 2))
i=$(($i + 1)) i=$((i + 1))
while [ $i -le $j ] ; do while [ $i -le $j ] ; do
port_dir="$port_dir/$(eval echo \${$i})" port_dir="$port_dir/$(eval echo '$'{$i})"
i=$(($i + 1)) i=$((i + 1))
done done
set -- $(echo $port_dir | sed -e 's/^.*:\([^:]*\):\([^:]*\)$/\1 \2/') set -- $(echo "$port_dir" | sed -e 's/^.*:\([^:]*\):\([^:]*\)$/\1 \2/')
PORT=$1 PORT=$1
SLOT=$(($2 + $FIRST_BAY_NUMBER)) SLOT=$(($2 + FIRST_BAY_NUMBER))
if [ -z "$SLOT" ] ; then if [ -z "$SLOT" ] ; then
return return
fi fi
CHAN=`map_channel $PCI_ID $PORT` CHAN=$(map_channel "$PCI_ID" "$PORT")
SLOT=`map_slot $SLOT $CHAN` SLOT=$(map_slot "$SLOT" "$CHAN")
if [ -z "$CHAN" ] ; then if [ -z "$CHAN" ] ; then
return return
fi fi
echo ${CHAN}${SLOT}${PART} echo "${CHAN}${SLOT}${PART}"
} }
# Figure out the name for the enclosure symlink # Figure out the name for the enclosure symlink
@ -425,7 +584,7 @@ enclosure_handler () {
# Get the enclosure ID ("0:0:0:0") # Get the enclosure ID ("0:0:0:0")
ENC=$(basename $(readlink -m "/sys/$DEVPATH/../..")) ENC=$(basename $(readlink -m "/sys/$DEVPATH/../.."))
if [ ! -d /sys/class/enclosure/$ENC ] ; then if [ ! -d "/sys/class/enclosure/$ENC" ] ; then
# Not an enclosure, bail out # Not an enclosure, bail out
return return
fi fi
@ -433,14 +592,14 @@ enclosure_handler () {
# Get the long sysfs device path to our enclosure. Looks like: # Get the long sysfs device path to our enclosure. Looks like:
# /devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/ ... /enclosure/0:0:0:0 # /devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0/ ... /enclosure/0:0:0:0
ENC_DEVICE=$(readlink /sys/class/enclosure/$ENC) ENC_DEVICE=$(readlink "/sys/class/enclosure/$ENC")
# Grab the full path to the hosts port dir: # Grab the full path to the hosts port dir:
# /devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0 # /devices/pci0000:00/0000:00:03.0/0000:05:00.0/host0/port-0:0
PORT_DIR=$(echo $ENC_DEVICE | grep -Eo '.+host[0-9]+/port-[0-9]+:[0-9]+') PORT_DIR=$(echo "$ENC_DEVICE" | grep -Eo '.+host[0-9]+/port-[0-9]+:[0-9]+')
# Get the port number # Get the port number
PORT_ID=$(echo $PORT_DIR | grep -Eo "[0-9]+$") PORT_ID=$(echo "$PORT_DIR" | grep -Eo "[0-9]+$")
# The PCI directory is two directories up from the port directory # The PCI directory is two directories up from the port directory
# /sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0 # /sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0
@ -450,8 +609,8 @@ enclosure_handler () {
PCI_ID=$(echo "$PCI_ID_LONG" | sed -r 's/^[0-9]+://g') PCI_ID=$(echo "$PCI_ID_LONG" | sed -r 's/^[0-9]+://g')
# Name our device according to vdev_id.conf (like "L0" or "U1"). # Name our device according to vdev_id.conf (like "L0" or "U1").
NAME=$(awk "/channel/{if (\$1 == \"channel\" && \$2 == \"$PCI_ID\" && \ NAME=$(awk '/channel/{if ($1 == "channel" && $2 == "$PCI_ID" && \
\$3 == \"$PORT_ID\") {print \$4int(count[\$4])}; count[\$4]++}" $CONFIG) $3 == "$PORT_ID") {print ${4}int(count[$4])}; count[$4]++}' $CONFIG)
echo "${NAME}" echo "${NAME}"
} }
@ -487,9 +646,9 @@ alias_handler () {
# ambiguity seems unavoidable, so devices using this facility # ambiguity seems unavoidable, so devices using this facility
# must not use such names. # must not use such names.
DM_PART= DM_PART=
if echo $DM_NAME | grep -q -E 'p[0-9][0-9]*$' ; then if echo "$DM_NAME" | grep -q -E 'p[0-9][0-9]*$' ; then
if [ "$DEVTYPE" != "partition" ] ; then if [ "$DEVTYPE" != "partition" ] ; then
DM_PART=`echo $DM_NAME | awk -Fp '/p/{print "-part"$2}'` DM_PART=$(echo "$DM_NAME" | awk -Fp '/p/{print "-part"$2}')
fi fi
fi fi
@ -497,21 +656,25 @@ alias_handler () {
for link in $DEVLINKS ; do for link in $DEVLINKS ; do
# Remove partition information to match key of top-level device. # Remove partition information to match key of top-level device.
if [ -n "$DM_PART" ] ; then if [ -n "$DM_PART" ] ; then
link=`echo $link | sed 's/p[0-9][0-9]*$//'` link=$(echo "$link" | sed 's/p[0-9][0-9]*$//')
fi fi
# Check both the fully qualified and the base name of link. # Check both the fully qualified and the base name of link.
for l in $link `basename $link` ; do for l in $link $(basename "$link") ; do
alias=`awk "\\$1 == \"alias\" && \\$3 == \"${l}\" \ if [ ! -z "$l" ]; then
{ print \\$2; exit }" $CONFIG` alias=$(awk -v var="$l" '($1 == "alias") && \
if [ -n "$alias" ] ; then ($3 == var) \
echo ${alias}${DM_PART} { print $2; exit }' $CONFIG)
return if [ -n "$alias" ] ; then
echo "${alias}${DM_PART}"
return
fi
fi fi
done done
done done
} }
while getopts 'c:d:eg:mp:h' OPTION; do # main
while getopts 'c:d:eg:jmp:h' OPTION; do
case ${OPTION} in case ${OPTION} in
c) c)
CONFIG=${OPTARG} CONFIG=${OPTARG}
@ -524,7 +687,9 @@ while getopts 'c:d:eg:mp:h' OPTION; do
# create the enclosure device symlinks only. We also need # create the enclosure device symlinks only. We also need
# "enclosure_symlinks yes" set in vdev_id.config to actually create the # "enclosure_symlinks yes" set in vdev_id.config to actually create the
# symlink. # symlink.
ENCLOSURE_MODE=$(awk '{if ($1 == "enclosure_symlinks") print $2}' $CONFIG) ENCLOSURE_MODE=$(awk '{if ($1 == "enclosure_symlinks") \
print $2}' "$CONFIG")
if [ "$ENCLOSURE_MODE" != "yes" ] ; then if [ "$ENCLOSURE_MODE" != "yes" ] ; then
exit 0 exit 0
fi fi
@ -535,6 +700,9 @@ while getopts 'c:d:eg:mp:h' OPTION; do
p) p)
PHYS_PER_PORT=${OPTARG} PHYS_PER_PORT=${OPTARG}
;; ;;
j)
MULTIJBOD_MODE=yes
;;
m) m)
MULTIPATH_MODE=yes MULTIPATH_MODE=yes
;; ;;
@ -544,7 +712,7 @@ while getopts 'c:d:eg:mp:h' OPTION; do
esac esac
done done
if [ ! -r $CONFIG ] ; then if [ ! -r "$CONFIG" ] ; then
echo "Error: Config file \"$CONFIG\" not found" echo "Error: Config file \"$CONFIG\" not found"
exit 0 exit 0
fi fi
@ -555,11 +723,11 @@ if [ -z "$DEV" ] && [ -z "$ENCLOSURE_MODE" ] ; then
fi fi
if [ -z "$TOPOLOGY" ] ; then if [ -z "$TOPOLOGY" ] ; then
TOPOLOGY=`awk "\\$1 == \"topology\" {print \\$2; exit}" $CONFIG` TOPOLOGY=$(awk '($1 == "topology") {print $2; exit}' "$CONFIG")
fi fi
if [ -z "$BAY" ] ; then if [ -z "$BAY" ] ; then
BAY=`awk "\\$1 == \"slot\" {print \\$2; exit}" $CONFIG` BAY=$(awk '($1 == "slot") {print $2; exit}' "$CONFIG")
fi fi
TOPOLOGY=${TOPOLOGY:-sas_direct} TOPOLOGY=${TOPOLOGY:-sas_direct}
@ -572,7 +740,7 @@ if [ "$ENCLOSURE_MODE" = "yes" ] && [ "$TOPOLOGY" = "sas_direct" ] ; then
fi fi
# Just create the symlinks to the enclosure devices and then exit. # Just create the symlinks to the enclosure devices and then exit.
ENCLOSURE_PREFIX=$(awk '/enclosure_symlinks_prefix/{print $2}' $CONFIG) ENCLOSURE_PREFIX=$(awk '/enclosure_symlinks_prefix/{print $2}' "$CONFIG")
if [ -z "$ENCLOSURE_PREFIX" ] ; then if [ -z "$ENCLOSURE_PREFIX" ] ; then
ENCLOSURE_PREFIX="enc" ENCLOSURE_PREFIX="enc"
fi fi
@ -582,16 +750,16 @@ if [ "$ENCLOSURE_MODE" = "yes" ] && [ "$TOPOLOGY" = "sas_direct" ] ; then
fi fi
# First check if an alias was defined for this device. # First check if an alias was defined for this device.
ID_VDEV=`alias_handler` ID_VDEV=$(alias_handler)
if [ -z "$ID_VDEV" ] ; then if [ -z "$ID_VDEV" ] ; then
BAY=${BAY:-bay} BAY=${BAY:-bay}
case $TOPOLOGY in case $TOPOLOGY in
sas_direct|sas_switch) sas_direct|sas_switch)
ID_VDEV=`sas_handler` ID_VDEV=$(sas_handler)
;; ;;
scsi) scsi)
ID_VDEV=`scsi_handler` ID_VDEV=$(scsi_handler)
;; ;;
*) *)
echo "Error: unknown topology $TOPOLOGY" echo "Error: unknown topology $TOPOLOGY"