Since I started working on Virtual I/O Servers and PowerVM I’ve created many Shared Ethernet Adapters in all modes (standard, failover, or sharing). I’ve learned one important lesson “be careful when creating a Shared Ethernet Adapter“. A single mistake can cause a network outage and I’m sure that you’ve already seen someone in your team creating an ARP storm by mismatching control channel adapter or by adding a vlan that is already added on a Virtual Ethernet Adapter. Because of this kind of errors I know some customers who are trying to avoid the configuration of Shared Ethernet Adapter in failover or sharing mode to avoid any network outage. With the new version of Virtual I/O Server (starting from 2.2.2.2) network loop and ARP storms are -in most cases- detected and stopped at the Virtual I/O Server level or at the firwmare level. I always check two or three times my configuration before creating a Shared Ethernet Adapter. All these errors come -most of the time- from a lack of rigor and are in -almost- all cases due to the system administrator. With the new version of PowerVM you can now create all Shared Ethernet Adapters without specifying any control channel adapter (The Hardware Management Console and the Virtual I/O Server will do it for you). A new discovery protocol implemented on Virtual I/O Server is matching Shared Ethernet Adapters between them and will take care of creating the Control Channel vlan for you (this one will not be visible on the Virtual I/O Server). Much simpler = less errors. Here is a practical how-to :
How does it work ?
A new discovery protocol called SEA HA match partners between them by using a dedicated vlan (not configurable by the user). Here are a few things to know :
- Multiple Shared Ethernet Adapters can share the vlan 4095 for their Control Channel link.
- The vlan 4095 is created per Virtual Switch for this Control Channel link.
- As always only two Shared Ethernet Adapters can be partners, the Hardware Management Console is ensuring that priority 1 and 2 are used (I’ve seen some customers using priority 3 and 4, do don’t this.)
- Both failover and sharing mode can be used.
- Shared Ethernet Adapters with a dedicated Control Channel Adapter, can be migrated to this configuration with a network outage, put the SEA in defined state before :
Here is any example of this configuration on a Shared Ethernet Adapter in Sharing Mode :
On the image below you can follow the steps of this new discovery protocol :
- 1/No dedicated Control Channel Adapter in Shared Ethernet Adapter Creation. The discovery protocol will be used if you are creating a SEA in failover or sharing mode without specifying the ctl_chan attribute.
- 2/Partners are identified by their PVID, both partners must have the same PVID.
- 3/This PVID has to be uniq per SEA pairs.
- 4/Additional vlans ID are compared : partners with not matching additional vlans IDs are still considered as partners if their PVID match.
- 5/Shared Ethernet Adapter with matching additional vlan IDs and not matching PVID are not considered as partners.
- 6/If partners are not matching their additional vlan IDs they are still considered partners but an error is logged in the errlog.
Prerequisites
Shared Ethernet Adapter without the need of a Control Channel Adapter can’t be created on all systems. At the time of writing this post only a few models of POWER7 machines (maybe POWER8) have the firmware implementing the feature. You have to check that the firmware of your machine is at least a XX780_XXX release. Be careful to check the release note of the firmware, some of the 780’s firmwares does not permit the creation of SEA without Control Channel Adapter (especially 9117-MMB) (here is an example on this page : link here, the release note says : “Support was added to the Management Console command line to allow configuring a shared control channel for multiple pairs of Shared Ethernet Adapters (SEAs). This simplifies the control channel configuration to reduce network errors when the SEAs are in fail-over mode. This feature is not supported on IBM Power 770 (9117-MMB) and IBM Power 780 (9179-MHB) systems.”). Because the Hardware Management Console is using the vlan 4095 to create the Control Channel link between Shared Ethernet Adapters it has to be aware of this feature and must ensure that the vlan 4095 is not usable or configurable by the administrator. The HMC v7R7.8.0 is aware of this that’s why the HMC must be updated at least to this level.
- Check your machine firmware, in my case I’m working on a 9117-MMD (P7+770) with the lastest firmware available (at the time of writing this post) :
# lsattr -El sys0 -a modelname modelname IBM,9117-MMD Machine name False # lsmcode -A sys0!system:AM780_056 (t) AM780_056 (p) AM780_056 (t)
- These prerequisites can be check directly from the Hardware Management Console :
hscroot@myhmc:~> lslic -t sys -m 9117-MMD-65XXXX lic_type=Managed System,management_status=Enabled,disabled_reason=,activated_level=56,activated_spname=FW780.10,installed_level=56,installed_spname=FW780.10,accepted_level=56,accepted_spname=FW780.10,ecnumber=01AM780,mtms=9117-MMD*658B2AD,deferred_level=None,deferred_spname=FW780.10,platform_ipl_level=56,platform_ipl_spname=FW780.10,curr_level_primary=56,curr_spname_primary=FW780.10,curr_ecnumber_primary=01AM780,curr_power_on_side_primary=temp,pend_power_on_side_primary=temp,temp_level_primary=56,temp_spname_primary=FW780.10,temp_ecnumber_primary=01AM780,perm_level_primary=56,perm_spname_primary=FW780.10,perm_ecnumber_primary=01AM780,update_control_primary=HMC,curr_level_secondary=56,curr_spname_secondary=FW780.10,curr_ecnumber_secondary=01AM780,curr_power_on_side_secondary=temp,pend_power_on_side_secondary=temp,temp_level_secondary=56,temp_spname_secondary=FW780.10,temp_ecnumber_secondary=01AM780,perm_level_secondary=56,perm_spname_secondary=FW780.10,perm_ecnumber_secondary=01AM780,update_control_secondary=HMC
- Check your Hardware Management Console release is at least V7R7.8.0 (in my case my HMC is at the latest level available at the time of writing this post) :
hscroot@myhmc:~> lshmc -V "version= Version: 7 Release: 7.9.0 Service Pack: 0 HMC Build level 20140409.1 MH01406: Required fix for HMC V7R7.9.0 (04-16-2014) ","base_version=V7R7.9.0 "
Shared Ethernet Adapter creation in sharing mode without control channel
The creation is simple, just identify your Real Adapter and your Virtual Adapter(s). Check on both Virtual I/O Server that PVID used on Virtual Adapters are the same and check priority are ok (use priority 1 on PRIMARY Virtual I/O Server and priority 2 on BACKUP Virtual I/O Server). I’m creating in this post a Shared Ethernet Adapter in Sharing Mode, steps are the same if you are creating a Shared Ethernet Adapter in auto mode.
- Identify the Real Adapter (in my case an LACP 802.3ad adapter) :
padmin@vios1$ lsdev -dev ent17 name status description ent17 Available EtherChannel / IEEE 802.3ad Link Aggregation padmin@vios2$ lsdev -dev ent17 name status description ent17 Available EtherChannel / IEEE 802.3ad Link Aggregation
padmin@vios1$ entstat -all ent13 | grep -iE "Priority|Port VLAN ID" Priority: 1 Active: False Port VLAN ID: 15 padmin@vios1$ entstat -all ent14 | grep -iE "Priority|Port VLAN ID" Priority: 1 Active: False Port VLAN ID: 16 padmin@vios2$ entstat -all ent13 | grep -iE "Priority|Port VLAN ID" Priority: 2 Active: True Port VLAN ID: 15 padmin@vios2$ entstat -all ent14 | grep -iE "Priority|Port VLAN ID" Priority: 2 Active: True Port VLAN ID: 16
padmin@vios1$ mkvdev -sea ent17 -vadapter ent13 ent14 -default ent13 -defaultid 15 -attr ha_mode=sharing largesend=1 large_receive=yes ent18 Available padmin@vios2$ mkvdev -sea ent17 -vadapter ent13 ent14 -default ent13 -defaultid 15 -attr ha_mode=sharing largesend=1 large_receive=yes ent18 Available
padmin@svios1$ lsdev -dev ent18 -attr attribute value description user_settable accounting disabled Enable per-client accounting of network statistics True adapter_reset yes Reset real adapter on HA takeover True ctl_chan Control Channel adapter for SEA failover True gvrp no Enable GARP VLAN Registration Protocol (GVRP) True ha_mode sharing High Availability Mode True [..] pvid 15 PVID to use for the SEA device True pvid_adapter ent13 Default virtual adapter to use for non-VLAN-tagged packets True qos_mode disabled N/A True queue_size 8192 Queue size for a SEA thread True real_adapter ent17 Physical adapter associated with the SEA True send_RARP yes Transmit Reverse ARP after HA takeover True thread 1 Thread mode enabled (1) or disabled (0) True virt_adapters ent13,ent14 List of virtual adapters associated with the SEA (comma separated) True
padmin@vios1$ entstat -all ent18 | grep -i "Control Channel PVID"
Control Channel PVID: 4095
- Looking at the entstat output SEA are partners (one PRIMARY_SH and one BACKUP_SH :
padmin@vios1$ entstat -all ent18 | grep -i state
State: PRIMARY_SH
padmin@vios2$ entstat -all ent18 | grep -i state
State: BACKUP_SH
Verbose and intelligent errlog
While configuring Shared Ethernet Adapter in this mode the errlog can give you a lot of informations about your configuration. For instance if additional vlan IDs does not match betweens Virtual Adapters of a Shared Ethernet Adapter you’ll be warned by an error in the errlog. Here are a few examples :
- Additional vlan IDs does not match between Virtual Adapters :
padmin@vios1$ errlog | more IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION A759776F 0506205214 I H ent18 SEA HA PARTNERS VLANS MISMATCH
- Looking on a detailed output you can get the missing vlan id :
padmin@vios1$
---------------------------------------------------------------------------
LABEL: VIOS_SEAHA_DSCV_VLA
IDENTIFIER: A759776F
Date/Time: Tue May 6 20:52:59 2014
Sequence Number: 704
Machine Id: 00XXXXXXXX00
Node Id: vios1
Class: H
Type: INFO
WPAR: Global
Resource Name: ent18
Resource Class: adapter
Resource Type: sea
Location:
Description
SEA HA PARTNERS VLANS MISMATCH
Probable Causes
VLAN MISCONFIGURATION
Failure Causes
VLAN MISCONFIGURATION
Recommended Actions
NONE
Detail Data
ERNUM
0000 001A
ABSTRACT
Discovered HA partner with unmatched VLANs
AREA
VLAN misconfiguration
BUILD INFO
BLD: 1309 30-10:08:58 y2013_40A0
LOCATION
Filename:sea_ha.c Function:seaha_process_dscv_init Line:6156
DATA
VLAN = 0x03E9
- The last line is the value of the missing vlan in hexadecimal (0x03E9, 1001 converted in decimal). We can manually check that this vlan is missing on vios1 :
# echo "ibase=16; 03E9" | bc 1001 padmin@vios1$ entstat -all ent18 | grep -i "VLAN Tag IDs:" VLAN Tag IDs: 1659 VLAN Tag IDs: 1682 VLAN Tag IDs: 1682 padmin@vios2$ entstat -all ent18 | grep -i "VLAN Tag IDs:" VLAN Tag IDs: 1659 VLAN Tag IDs: 1001 1682 VLAN Tag IDs: 1001 1682
- A loss of communication between SEA will also be logged in the errlog :
padmin@vios1$ errlog | more
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
B8C78C08 0502231214 I H ent18 SEA HA PARTNER LOST
padmin@vios1$ errlog -ls | more
Location:
Description
SEA HA PARTNER LOST
Probable Causes
SEA HA PARTNER DOWN
Failure Causes
SEA HA PARTNER DOWN
Recommended Actions
INITIATE PARTNER DISCOVERY
Detail Data
ERNUM
0000 0019
ABSTRACT
Initiating partner discovery due to lost partner
AREA
SEA HA discovery partner lost
BUILD INFO
BLD: 1309 30-10:08:58 y2013_40A0
LOCATION
Filename:sea_ha.c Function:seaha_dscv_ka_rcv_timeout Line:2977
DATA
Partner MAC: 0x1A:0xC4:0xFD:0x72:0x9B:0x0F
- Be careful looking at the errlog, a SEA in sharing mode will “become primary” even if it is the “backup” SEA (you have to look with errlog -ls command for the details) :
padmin@vios$ errlog | grep BECOME E48A73A4 0506205214 I H ent18 BECOME PRIMARY padmin@vios2$ errlog | grep BECOME 1FE2DD91 0506205314 I H ent18 BECOME PRIMARY
padmin@vios1$ errlog -ls | more LABEL: VIOS_SEAHA_PRIMARY IDENTIFIER: E48A73A4 [..] Description BECOME PRIMARY [..] padmin@vios2$ errlog -ls | more LABEL: VIOS_SEAHA_BACKUP IDENTIFIER: 1FE2DD91 [..] Description BECOME PRIMARY [..] ABSTRACT Transition from INIT to BACKUP [..] seahap->state= 0x00000003 Become the Backup SEA
Removing the control channel adapter from an existing Shared Ethernet Adapter
A “classic” Shared Ethernet Adapter can be modified to be usable without the need of a dedicated Control Channel Adapter. This modification require a network outage and the
- On both Virtual I/O Servers put the Shared Ethernet Adapter in defined state :
padmin@vios1$ oem_setup_env root@vios1# rmdev -l ent18 ent18 Defined padmin@vios2$ oem_setup_env root@vios2# rmdev -l ent18 ent18 Defined
- On both Virtual I/O Servers remove the dedicated Control Channel Adapter for both Shared Ethernet Adapters :
root@vios1# lsattr -El ent18 -a ctl_chan ctl_chan ent12 Control Channel adapter for SEA failover True root@vios1# chdev -l ent18 -a ctl_chan="" ent18 changed root@vios1# lsattr -El ent18 -a ctl_chan ctl_chan Control Channel adapter for SEA failover True root@vios2# lsattr -El ent18 -a ctl_chan ctl_chan ent12 Control Channel adapter for SEA failover True root@vios2# chdev -l ent18 -a ctl_chan="" ent18 changed root@vios2# lsattr -El ent18 -a ctl_chan ctl_chan Control Channel adapter for SEA failover True
- Put each Shared Ethernet Adapter in available state by using the mkdev command :
root@vios1# mkdev -l ent18 ent18 Available root@vios2# mkdev -l ent18 ent18 Available
- Verify that the Shared Ethernet Adapter is now using vlan 4095 as Control Channel PVID :
padmin@vios1$ entstat -all ent18 | grep -i "Control Channel PVID"
Control Channel PVID: 4095
padmin@vios2$ entstat -all ent18 | grep -i "Control Channel PVID"
Control Channel PVID: 4095
The first step to a global PowerVM simplification
Be aware that this simplification is one of the first step of a much larger project. With the latest version of the HMC v8R80.1 a lot of new features will be available (June 2014). I can’t wait to test the “single point of management” for Virtual I/O Servers. Anyway, creating a Shared Ethernet Adapter is easier than before. Use this method to avoid human errors and misconfiguration of your Shared Ethernet Adapters. As always I hope this post will help you to understand this simplification. 