One machine, Two Nics, one subnet

Me and Schick drove ourselves nuts trying to get the 2 tg3 gig ethernet controllers on our HP proliant server to behave properly when they each had an IP on the same subnet. Long story short, Linux by default, sets arp_filter to 0 (The kernel can respond to arp requests with addresses from other interfaces.) By setting this value to 1, you can have multiple nics on the same subnet. Now to make sure that packets get routed properly, we have to add a custom routing table that adds a default gatway for the second nic. We then add a rule that will lookup all traffic on the second nic through the custom table. Here is the code goodness.

add this to /etc/sysctl.conf
net.ipv4.conf.default.arp_filter = 1

now add these two lines to your local.start
ip route add default via ${GATEWAY} dev $(NIC2_DEV} table XEN
ip rule add from ${NIC2_IP} table XEN

replace ${GATEWAY}, ${NIC2_DEV}, and ${NIC2_IP} with the appropriate values

Edit:
The reasoning for doing this was to provide a dedicated nic on the same subnet for domain0 on our XEN server. While the above works fine for 2 nics, if you need to add a bridge that XEN can utilize, you have to make sure that the second nic is brought up but not assigned an address. You can then attach it to your bridge and then treat the bridge like I described above.