IPMP Configuration On Solaris servers
IPMP
(Internet Protocol Multi Pathing) is a standard feature of Solaris.
Internet Protocol Multi Pathing is used to provide failover and outbound
load
spreading for network interfaces. You can use the normal Solaris, ifconfig, command to set it up.
IPMP has the following features:
It eliminates a single network adapter as a single point of failure in these cases, ( a. Network adapter failure b. Network link failure)
It enables interfaces to fail over within approximately 10 seconds when using the default configuration.
It can be configured by adjusting the parameters in the ”/etc/default/mpathd” file.
It can be configured for use with both IPv4 and IPv6.
It enables interfaces to be configured as standby interfaces.
It can failback automatically . In /etc/default/mpathd There is a parameter called FAILBACK=yes, this could be changed to FAILBACK=no if you DON'T want the interface to fail back automatically.
IPMP failure detection:
Probe-based failure detection
-detects network error by sending ICMP ECHO_REQUEST messages
Link based failure detection
-detects network error by checking the IFF_RUNING
Before configuring IP multi pathing, you must set the interfaces to use their own unique MAC (Media Access Control) addresses. You can do this by setting
the local-mac-address? OBP parameter to true. You can either do this with the eeprom command, or setting it on the OBP.Below is an example using the eeprom.
This method works for both SPARC and x86 systems. You could also use this command while Solaris is running.
bash-3.00# eeprom "local-mac-address?=true"
bash-3.00# eeprom local-mac-address?
local-mac-address?=true
bash-3.00#
LINK-BASED IPMP:
Link-Based IPMP Requirements:
Solaris 9 12/02 OS, at a minimum, must be installed.
Network interfaces must use any of the following drivers: hme, eri, ce, ge, bge, qfe, dmfe, e1000g, ixgb, nge, nxge, rge, xge
Unique MAC addresses must be configured on each of the interfaces.
An IPMP group name must be assigned to interfaces
Configuring Link-Based IPMP:
To make persistent Changes:
root@<Hostname> # cat /etc/hostname.ce0
<Hostname> netmask + broadcast + group ipmp0 up
root@<Hostname> # cat /etc/hostname.ce1
group ipmp0 standby up
root@<Hostname> #
ifconfig ce0 unplumb
ifconfig ce0 plumb
ifconfig ce1 plumb
ifconfig ce0 `cat /etc/hostname.ce0` up
ifconfig ce1 `cat /etc/hostname.ce1` up
Test Plan:
1. Once plumbed and up, we'll failover the IP to STANDBY NIC and try pinging that IP outside of the server.
ifconfig -a
if_mpadm -d ce0
itsev1# ping <IP of the Host>
2. if the IP is pinging outside of the server, then implementation is successful. Check the messages file as well for failover and failback information.
if_mpadm -r ce0
No test address configured on interface ce1; disabling probe-based failure detection on it
No test address configured on interface ce0; disabling probe-based failure detection on it
Successfully failed over from NIC ce0 to NIC ce1
Successfully failed back to NIC ce0
PROBE-BASED IPMP
Probe-Based IPMP Requirements:
The Solaris 8 10/00 OS, as a minimum, must be installed.
Unique MAC addresses must be configured on each network interface.
Multiple network adapter interfaces must be connected on each subnet.
An IPMP group name must be assigned to the group of interfaces.
A test address is assigned to an interface.
Additional hosts or devices must exist on the same subnet.
Configuring Probe-Based IPMP:
For probe based multipathing, we need at least 3 IP addresses. 2 for test addresses and 1 for the active address.
We will use the following IP addresses.
10.194.36.23 on ce0 and this will be the active IP.
10.194.36.24 on ce0:1 will be the test IP for ce0
10.194.36.25 on ce1 will be the test IP for the ce1 interface.
Cool. Let's configure probe based IP multi pathing. I will do this step by step cause it's very easy to get this wrong and then you wonder why it did not work.
Again, I'll start from scratch plumbing both interfaces. I will first do ce0 and then I will move on to ce1.
bash-3.00# ifconfig ce0 plumb
bash-3.00# ifconfig ce1 plumb
bash-3.00# ifconfig ce0 10.194.36.23 netmask + broadcast + up
Setting netmask of ce0 to 255.255.255.128
bash-3.00# ifconfig ce0 group ipmp0
Now ce0 is setup with an IP address and put in the ipmp0 group.
Next, we will configure the test IP address on ce0.
bash-3.00# ifconfig ce0 addif 10.194.36.24 netmask + broadcast + -failover deprecated up
Created new logical interface ce0:1
Setting netmask of ce0:1 to 255.255.255.128
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 5
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=9040843DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=1000842 mtu 1500 index 4
inet 0.0.0.0 netmask 0
ether 8:0:27:1e:1d:e6
bash-3.00#
Notice that I did not use the word test anywhere in the ifconfig command. So, how did I configure it then? There are two things I want to point out here.
1) I used the addif command to create a logical interface. This command means, add interface. This will check if there are any logical interfaces created
and then just add the next available number to the logical interface. Some people use the logical interface as the failover IP. It doesn't really matter.
You can make the logical or the physical the test. As long as there is a test interface.
2) To create a test interface you specify two flags, -failover and deprecated. These two flags tell IPMP that this interface is a test interface.
-failover means don't failover this IP address when the interface fails. This makes sense cause you don't want the test IP to failover in the event the physical
interface fails. You want it to keep on pinging to check when the interface is fixed. deprecated means do not use this interface when initiating packets from
this system. This is useful and needed when you use client software that initiated traffic from the host, or client system to the server where the server
expects packets from a known interface.
Now for the ce1 interface. It's already plumbed so I just need to configure an IP address, put it in the same group as ce0 and specify it as a test IP.
bash-3.00# ifconfig ce1 10.194.36.25 netmask + broadcast + up
Setting netmask of ce1 to 255.255.255.128
bash-3.00# ifconfig ce1 group ipmp0 -failover deprecated
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 5
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=9040843 mtu 1500 index 5
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=9040843 mtu 1500 index 4
inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:1e:1d:e6
I now it looks complicated but it really isn't. All we did was give ce1 an IP address, placed it in the ipmp0 group and made its a test interface.
Go and test it.
Again, I will disable the ce0 physical interface. Then I will ping with Ubuntu and I will check the ifconfig command if the IP did fail over.
bash-3.00# if_mpadm -d ce0
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=89000842
mtu 0 index 2
inet 0.0.0.0 netmask 0
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=89040842
mtu 1500 index 2
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=9040843
mtu 1500 index 3
inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:1e:1d:e6
ce1:1: flags=1000843 mtu 1500 index 3
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
bash-3.00#
Look at the output of ifconfig -a. You will notice that ce1 now has a logical interface called ce1:1 and that the IP address 10.194.36.23, the active IP is
plumbed and working. The test IP on ce0 is still pinging in the background to see if the interface has been fixed.
Let me ping from Ubuntu to see if it's still working and let's have a look at the messages file.
user@hostname:~$ ping 10.194.36.23
PING 10.194.36.23 (10.194.36.23) 56(84) bytes of data.
64 bytes from 10.194.36.23: icmp_req=1 ttl=255 time=0.537 ms
64 bytes from 10.194.36.23: icmp_req=2 ttl=255 time=0.482 ms
Jan 12 09:22:09 qserver in.mpathd[1248]: [ID 832587 daemon.error]
Successfully failed over from NIC ce0 to NIC ce1
Cool, it's working as planned. Now, let's fix the ce0 and see if the IP fails back.
bash-3.00# if_mpadm -r ce0
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=9040843 mtu 1500 index 2
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=9040843 mtu 1500 index 3
inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:1e:1d:e6
bash-3.00# tail -f /var/adm/messages
Jan 12 09:29:49 qserver in.mpathd[1248]: [ID 620804 daemon.error]
Successfully failed back to NIC ce0
user@hostname:~$ ping 10.194.36.23
PING 10.194.36.23 (10.194.36.23) 56(84) bytes of data.
64 bytes from 10.194.36.23: icmp_req=1 ttl=255 time=0.433 ms
64 bytes from 10.194.36.23: icmp_req=2 ttl=255 time=0.482 ms
64 bytes from 10.194.36.23: icmp_req=3 ttl=255 time=0.490 ms
Yep, still works. The interface has failed over and back and everything is still working.
Again we need to place this setup in the hostname files, otherwise the config is lost across reboots. Let's set that up.
For /etc/hostname.ce0
10.194.36.23 netmask + broadcast + group ipmp0 up
addif 10.194.36.24 netmask + broadcast + deprecated -failover up
For /etc/hostname.ce1
10.194.36.25 netmask + broadcast + group ipmp0 deprecated -failover up
spreading for network interfaces. You can use the normal Solaris, ifconfig, command to set it up.
IPMP has the following features:
It eliminates a single network adapter as a single point of failure in these cases, ( a. Network adapter failure b. Network link failure)
It enables interfaces to fail over within approximately 10 seconds when using the default configuration.
It can be configured by adjusting the parameters in the ”/etc/default/mpathd” file.
It can be configured for use with both IPv4 and IPv6.
It enables interfaces to be configured as standby interfaces.
It can failback automatically . In /etc/default/mpathd There is a parameter called FAILBACK=yes, this could be changed to FAILBACK=no if you DON'T want the interface to fail back automatically.
IPMP failure detection:
Probe-based failure detection
-detects network error by sending ICMP ECHO_REQUEST messages
Link based failure detection
-detects network error by checking the IFF_RUNING
Before configuring IP multi pathing, you must set the interfaces to use their own unique MAC (Media Access Control) addresses. You can do this by setting
the local-mac-address? OBP parameter to true. You can either do this with the eeprom command, or setting it on the OBP.Below is an example using the eeprom.
This method works for both SPARC and x86 systems. You could also use this command while Solaris is running.
bash-3.00# eeprom "local-mac-address?=true"
bash-3.00# eeprom local-mac-address?
local-mac-address?=true
bash-3.00#
LINK-BASED IPMP:
Link-Based IPMP Requirements:
Solaris 9 12/02 OS, at a minimum, must be installed.
Network interfaces must use any of the following drivers: hme, eri, ce, ge, bge, qfe, dmfe, e1000g, ixgb, nge, nxge, rge, xge
Unique MAC addresses must be configured on each of the interfaces.
An IPMP group name must be assigned to interfaces
Configuring Link-Based IPMP:
To make persistent Changes:
root@<Hostname> # cat /etc/hostname.ce0
<Hostname> netmask + broadcast + group ipmp0 up
root@<Hostname> # cat /etc/hostname.ce1
group ipmp0 standby up
root@<Hostname> #
ifconfig ce0 unplumb
ifconfig ce0 plumb
ifconfig ce1 plumb
ifconfig ce0 `cat /etc/hostname.ce0` up
ifconfig ce1 `cat /etc/hostname.ce1` up
Test Plan:
1. Once plumbed and up, we'll failover the IP to STANDBY NIC and try pinging that IP outside of the server.
ifconfig -a
if_mpadm -d ce0
itsev1# ping <IP of the Host>
2. if the IP is pinging outside of the server, then implementation is successful. Check the messages file as well for failover and failback information.
if_mpadm -r ce0
No test address configured on interface ce1; disabling probe-based failure detection on it
No test address configured on interface ce0; disabling probe-based failure detection on it
Successfully failed over from NIC ce0 to NIC ce1
Successfully failed back to NIC ce0
PROBE-BASED IPMP
Probe-Based IPMP Requirements:
The Solaris 8 10/00 OS, as a minimum, must be installed.
Unique MAC addresses must be configured on each network interface.
Multiple network adapter interfaces must be connected on each subnet.
An IPMP group name must be assigned to the group of interfaces.
A test address is assigned to an interface.
Additional hosts or devices must exist on the same subnet.
Configuring Probe-Based IPMP:
For probe based multipathing, we need at least 3 IP addresses. 2 for test addresses and 1 for the active address.
We will use the following IP addresses.
10.194.36.23 on ce0 and this will be the active IP.
10.194.36.24 on ce0:1 will be the test IP for ce0
10.194.36.25 on ce1 will be the test IP for the ce1 interface.
Cool. Let's configure probe based IP multi pathing. I will do this step by step cause it's very easy to get this wrong and then you wonder why it did not work.
Again, I'll start from scratch plumbing both interfaces. I will first do ce0 and then I will move on to ce1.
bash-3.00# ifconfig ce0 plumb
bash-3.00# ifconfig ce1 plumb
bash-3.00# ifconfig ce0 10.194.36.23 netmask + broadcast + up
Setting netmask of ce0 to 255.255.255.128
bash-3.00# ifconfig ce0 group ipmp0
Now ce0 is setup with an IP address and put in the ipmp0 group.
Next, we will configure the test IP address on ce0.
bash-3.00# ifconfig ce0 addif 10.194.36.24 netmask + broadcast + -failover deprecated up
Created new logical interface ce0:1
Setting netmask of ce0:1 to 255.255.255.128
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 5
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=9040843DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 5
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=1000842 mtu 1500 index 4
inet 0.0.0.0 netmask 0
ether 8:0:27:1e:1d:e6
bash-3.00#
Notice that I did not use the word test anywhere in the ifconfig command. So, how did I configure it then? There are two things I want to point out here.
1) I used the addif command to create a logical interface. This command means, add interface. This will check if there are any logical interfaces created
and then just add the next available number to the logical interface. Some people use the logical interface as the failover IP. It doesn't really matter.
You can make the logical or the physical the test. As long as there is a test interface.
2) To create a test interface you specify two flags, -failover and deprecated. These two flags tell IPMP that this interface is a test interface.
-failover means don't failover this IP address when the interface fails. This makes sense cause you don't want the test IP to failover in the event the physical
interface fails. You want it to keep on pinging to check when the interface is fixed. deprecated means do not use this interface when initiating packets from
this system. This is useful and needed when you use client software that initiated traffic from the host, or client system to the server where the server
expects packets from a known interface.
Now for the ce1 interface. It's already plumbed so I just need to configure an IP address, put it in the same group as ce0 and specify it as a test IP.
bash-3.00# ifconfig ce1 10.194.36.25 netmask + broadcast + up
Setting netmask of ce1 to 255.255.255.128
bash-3.00# ifconfig ce1 group ipmp0 -failover deprecated
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 5
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=9040843 mtu 1500 index 5
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=9040843 mtu 1500 index 4
inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:1e:1d:e6
I now it looks complicated but it really isn't. All we did was give ce1 an IP address, placed it in the ipmp0 group and made its a test interface.
Go and test it.
Again, I will disable the ce0 physical interface. Then I will ping with Ubuntu and I will check the ifconfig command if the IP did fail over.
bash-3.00# if_mpadm -d ce0
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=89000842
mtu 0 index 2
inet 0.0.0.0 netmask 0
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=89040842
mtu 1500 index 2
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=9040843
mtu 1500 index 3
inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:1e:1d:e6
ce1:1: flags=1000843 mtu 1500 index 3
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
bash-3.00#
Look at the output of ifconfig -a. You will notice that ce1 now has a logical interface called ce1:1 and that the IP address 10.194.36.23, the active IP is
plumbed and working. The test IP on ce0 is still pinging in the background to see if the interface has been fixed.
Let me ping from Ubuntu to see if it's still working and let's have a look at the messages file.
user@hostname:~$ ping 10.194.36.23
PING 10.194.36.23 (10.194.36.23) 56(84) bytes of data.
64 bytes from 10.194.36.23: icmp_req=1 ttl=255 time=0.537 ms
64 bytes from 10.194.36.23: icmp_req=2 ttl=255 time=0.482 ms
Jan 12 09:22:09 qserver in.mpathd[1248]: [ID 832587 daemon.error]
Successfully failed over from NIC ce0 to NIC ce1
Cool, it's working as planned. Now, let's fix the ce0 and see if the IP fails back.
bash-3.00# if_mpadm -r ce0
bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 10.194.36.23 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:6f:c5:6e
ce0:1: flags=9040843 mtu 1500 index 2
inet 10.194.36.24 netmask ffffff80 broadcast 10.194.36.127
ce1: flags=9040843 mtu 1500 index 3
inet 10.194.36.25 netmask ffffff80 broadcast 10.194.36.127
groupname ipmp0
ether 8:0:27:1e:1d:e6
bash-3.00# tail -f /var/adm/messages
Jan 12 09:29:49 qserver in.mpathd[1248]: [ID 620804 daemon.error]
Successfully failed back to NIC ce0
user@hostname:~$ ping 10.194.36.23
PING 10.194.36.23 (10.194.36.23) 56(84) bytes of data.
64 bytes from 10.194.36.23: icmp_req=1 ttl=255 time=0.433 ms
64 bytes from 10.194.36.23: icmp_req=2 ttl=255 time=0.482 ms
64 bytes from 10.194.36.23: icmp_req=3 ttl=255 time=0.490 ms
Yep, still works. The interface has failed over and back and everything is still working.
Again we need to place this setup in the hostname files, otherwise the config is lost across reboots. Let's set that up.
For /etc/hostname.ce0
10.194.36.23 netmask + broadcast + group ipmp0 up
addif 10.194.36.24 netmask + broadcast + deprecated -failover up
For /etc/hostname.ce1
10.194.36.25 netmask + broadcast + group ipmp0 deprecated -failover up
About Manas Ranjan Tripathy
No comments