[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[SAGE] technical help with routing on Solaris 10
I sent the following to sun-managers, but haven't gotten much help there.
I'm thinking this may be a more generic networking problem, so I'm hoping
one of you might be able to help.
I have what seems to be a weird problem with routing that I hope y'all can help
with.
I have a Sun Fire V210 running Solaris 10 with a recommended patchcluster a
couple of weeks old.
It's got 8 interfaces -- the four on board plus a quad gigaswift (ce) card.
It's on a pretty complicated network. We've got three VLANs on it, each
running IPMP for fault tolerance (active/standby):
bge0/ce0 are on a management (mgt) VLAN
bge1/ce1 are on a network-attached storage (nas) VLAN
bge3/ce3 are on an "application" (app) VLAN
mgt is 10.66.0.0/16
nas is 10.67.0.0/16
app is 10.65.0.0/16
There are no default routes, because none of the networks are routable anyway,
and it's not supposed to talk to the net.
The network interfaces are configured as follows:
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232
index 1
inet 127.0.0.1 netmask ff000000
bge0: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 2
inet 10.66.3.2 netmask ffff0000 broadcast 10.66.255.255
groupname mgt
ether 0:3:ba:ee:64:a9
bge0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 10.66.3.1 netmask ffff0000 broadcast 10.66.255.255
bge1: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 3
inet 10.67.3.2 netmask ffff0000 broadcast 10.67.255.255
groupname nas
ether 0:3:ba:ee:64:aa
bge1:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet 10.67.3.1 netmask ffff0000 broadcast 10.67.255.255
bge3: flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER> mtu 1500 index 4
inet 10.65.3.2 netmask ffff0000 broadcast 10.65.255.255
groupname app
ether 0:3:ba:ee:64:ac
bge3:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
inet 10.65.3.1 netmask ffff0000 broadcast 10.65.255.255
ce0: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 5
inet 10.66.3.3 netmask ffff0000 broadcast 10.66.255.255
groupname mgt
ether 0:3:ba:da:69:77
ce1: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 6
inet 10.67.3.3 netmask ffff0000 broadcast 10.67.255.255
groupname nas
ether 0:3:ba:da:69:78
ce3: flags=69040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,STANDBY,INACTIVE> mtu 1500 index 7
inet 10.65.3.3 netmask ffff0000 broadcast 10.65.255.255
groupname app
ether 0:3:ba:da:69:7a
Now we have a problem: it needs to talk to the net. Our network guy set up our
firewall (cisco FWSM in a 6509-e chassis) so that 10.66.0.3 (a context
within the firewall module) is a gateway.
I then add a static route:
route add -host 66.94.234.13 10.66.0.3 -static
That 66 address is yahoo.com, just for testing.
When I try to telnet to that address on port 80, I get no errors for several
minutes. I get the "Trying ..." message, and then several minutes later, it
times out and fails.
I've used tcpdump to watch the interfaces (all of them!) and I don't see the
packets at all. If I try to telnet to the 10.66.0.3 gateway, I do see the
packets, though the connection is refused by the firewall. I do see other
packets on my interfaces, as well as ICMP traffic to the 10.66.0.3 gateway
(probes from the IPMP module). I just don't see any traffic when I
actually try to use the gateway to get out. Naturally, I don't see any
packets any farther down the line, either. Something seems to be wrong on
the Sun that the packets are never actually leaving.
The firewall rule allows ip any any, but I don't think the problem is the
firewall since I'm not seeing packets on the interface.
The routing table looks like this:
Routing Table: IPv4
Destination Gateway Flags Ref Use Interface
-------------------- -------------------- ----- ----- ------ ---------
10.65.0.10 10.65.0.10 UGH 1 0
10.66.0.10 10.66.0.10 UGH 1 0
10.67.0.10 10.67.0.10 UGH 1 0
66.94.234.13 10.66.0.3 UGH 1 0
10.66.0.0 10.66.3.1 U 1 943 bge0:1
10.66.0.0 10.66.3.1 U 1 0 bge0
10.66.0.0 10.66.3.1 U 1 742 ce0
10.67.0.0 10.67.3.1 U 1 911 bge1:1
10.67.0.0 10.67.3.1 U 1 0 bge1
10.67.0.0 10.67.3.1 U 1 775 ce1
10.65.0.0 10.65.3.1 U 1 911 bge3:1
10.65.0.0 10.65.3.1 U 1 0 bge3
10.65.0.0 10.65.3.1 U 1 511 ce3
224.0.0.0 10.66.3.1 U 1 0 bge0:1
127.0.0.1 127.0.0.1 UH 28 73742 lo0
The .10 interfaces at the top are static routes to a router to answer ICMP
echoes for the IPMP probes.
Any ideas?
Thanks much,
-Adam