Routing With 2 Public Ips

Home » General » Routing With 2 Public Ips
General 18 Comments

Hello,

i have a server with 2 public ips on 2 devices.

I want that the request of incoming traffic dont use the default gateway. Incoming traffic sould be answered using the gateway of the incoming device

Could i realize this with firewalld? Or directly iptables?

Greeting J

18 thoughts on - Routing With 2 Public Ips

  • No you can not do that via firewalld or iptables. The problem is you have to tell the packets to go out the proper interface which must be done via routing tables. For that purpose you need ip route. I suggest you take a look at

    https://kindlund.wordpress.com/2007/11/19/configuring-multiple-default-routes-in-linux/

    This link provides a very thorough description of what must be done.

    Just a warning is that you will want your routing tables to be maintained across system boots. I put my routes for my bridged interfaces into:

    /etc/sysconfig/network-scripts/route-br1
    /etc/sysconfig/network-scripts/route-br2

    You can put your routes into similar files… just replace the br1/br2
    with your appropriate interface names.

  • This is half true. Depends on the application or the way that the network traffic is flowing you could use some iptables rules to mark a connection for example by the source MAC address per new connections which would be a specific router and by that mark the connection, then in the routing level decide which default gateway to use for this specific connection. You can take a look at an example that I wrote and modify it to use a MAC address match instead of NFQUEUE at:
    http://wiki.squid-cache.org/EliezerCroitoru/Drafts/MwanLB#iptables_rules_example

    The idea is that you mark a new connection from a specific router with a unique mark and then restore the connection mark to force a specific routing table on this mark(IE connection)

    Hope it Helps, Eliezer

  • While that’s true, you still have to select the default route using “ip rule”. And since you can do that using the source address for outgoing packets, there’s no reason to mark them. It’s completely redundant.

  • Can you match the MAC address?? in ip rule?
    If so it’s much simpler then I was estimating.

    Eliezer

  • No, but you don’t have to. In the scenario presented, two links with two IP addresses in different broadcast domains, traffic that is sent in response to requests received on the second link/IP address will have the second IP address in the source address field. You can use that as the rule.

    Remember that Ethernet and IP are separate technologies. You can make routing policies entirely in the IP layer without mixing in Ethernet stuff like MAC addresses.

  • I still do not understand something. The thread started with:
    i have a server with 2 public ips on 2 devices.

    I want that the request of incoming traffic dont use the default gateway. Incoming traffic sould be answered using the gateway of the incoming device

    Could i realize this with firewalld? Or directly iptables?
    ##END OF QUOTE

    Which means he has 1 server with two gateway devices which each has it’s own broadcast space\network. It’s not clear to me if there are two gateways in the same broadcast\network or not. if it’s on the same network then he must have some routing rules and the issue is not about a specific src address but about a connection.. Now with both of these devices there he has an issue. He sure needs to use basic routing skills to make it work using some metrics if he wants a static routing setup… but when it becomes almost asymmetric it is possible to have a “reverse-path” routing situation which is because the server has two default gateways and not one. For this situation he cannot utilize the source address but only the source mac address unless these 2 devices are some sort reverse proxies which in this case do not require any routing settings at all and not even a default gateway or direct Internet access.

    So from what I understood he will need to do some connection marking by the MAC address if these two devices are two routers which does NAT.

    Eliezer

  • Can you explain what you mean? Not only am I not assuming that, I can hardly conceive of any situation in which a host will receive traffic for its own gateway.

  • I think it’s safe to assume that the two addresses and, necessarily, the gateways, are in separate broadcast domains. However, even if that weren’t the case, it is still sufficient to create two routing tables and use “ip rule” to select the appropriate table (and the gateway it specifies) based on the source address of the packet being routed.

    Just to walk you through it, assume his server has two addresses in separate broadcast domains. The first interface has 1.2.3.4/24 with gateway 1.2.3.1. The second interface has 2.3.4.5/24 with gateway 2.3.4.1.

    Now, a host at 192.0.2.2 initiates a connection. It sends a TCP SYN
    packet to 1.2.3.4. The server receives that packet and sends a TCP
    SYN/ACK to 192.0.2.2. The source address of that packet is 1.2.3.4. A
    rule exits that matches packets from 1.2.3.4 and selects the first routing table, where the default gateway is 1.2.3.1.

    Later, a host at 198.51.100.3 initiates a connection. It sends a TCP
    SYN packet to 2.3.4.5. The server receives that packet and sends a TCP
    SYN/ACK to 198.51.100.3. The source address of that packet is 2.3.4.5, since that is the address that the SYN was sent to. A rule exists on the server that matches packets from 2.3.4.5 and selects the second routing table, where the default gateway is 2.3.4.1.

    You wouldn’t normally have two addresses on two interfaces in the same broadcast domain. You’d probably bond the interfaces instead. But if you did, it wouldn’t change the process. Reply packets will still have their source address set to the same address that received the request, and you’d still be able to specify the routing table based on that address.

    So, again, you *can* mark connections and select a route that way, but it’s slower and more complex than using information that’s already available. There’s simply no reason to do that in a standard multi-homed setup.

  • … Basic 1:1 NAT … you have two gateways while you have two ip addresses or one on the interface. Just to illustrate the issue: AWS instance with two interfaces which have two ip addresses NATTED to them by AWS front tier using some kind of virtual gateway.

    Eliezer

    * Now I am sure that you didn’t understood the situation\network as I am!

  • I’m struggling to understand what you meant when you said that the destination is the gateway. If you just mean that the traffic is NATed, then again, I was not assuming that in any of my explanations.

    A host with two addresses and two NAT gateways would apply routing policy just like one that isn’t behind NAT gateways. In that configuration, NAT isn’t relevant.

    Now, if you had a host with just one address that was behind two different NAT routers, then that would be a configuration that might require marking connections based on the MAC address of incoming packets, and applying rules based on those marks. However, such a configuration is broken in several different ways, and connection marking just digs that hole deeper. Don’t do this.

    At some point, I’d remind you of the advice of Dr Robert Anthony: “

  • I may not understood\interpreted the scenario pretty well. I will try again:
    “i have a server with 2 public ips on 2 devices.”
    He has two servers or two gateways or both??

    “I want that the request of incoming traffic dont use the default gateway. Incoming traffic sould be answered using the gateway of the incoming device ”
    How to interpret this sentence???
    The scenario I can think of is that these servers have more then one gateway and in this case it’s really unclear to me if the gateways are serving the same networks or not. From what I understood in this situation he wants to prevent a reverse path routing or in another words he wants the connection that comes from some host and gateway(which is unknown) to be returned\continued using this same gateway. So it’s really unclear(to me) if it’s routing includes two gateways for the same network and some routing protocol that allows that or another option.

    In any case since he spoke about “incoming” traffic which to my basic and simple understanding means the whole Internet he cannot use basic routing settings to do that *unless* he can predict that all incoming traffic is going to be from a specific gateway. Again I understood that he doesn’t know which gateway the traffic will come from but he wants to preserve the reverse path to them.

    If he will bother to clear it out I will continue to respond and if not… well it’s clear that there are couple possibilities to the scenario and I was referring to a specific one.

    So in any case I will add that in the past the linux kernel implemented a routing cache which was removed somewhere in the 3.X versions and while it existed it was so that if someone was contacting a server that had this kernel the routing cache was causing a weird scenario which the kernel would route traffic using the same gateway as long as the cache entry exists. However in the kernels which this cache was removed a Packet By Packet routing decision is being made and unless you can know who are all your clients you cannot predict their routing path using a simple static linux routing setup and you would be required to choose some other alternative.
    –

  • I said that, assuming the host with 2 public ips mentioned in the OP
    could be the gateway for a lan as I suspect routing based on source address that you suggested will not work for transit traffic. There’s a routeback option in shorewall which probably does what the OP wants but I have no idea how to achieve this with firewalld or iptables.

  • I’m sorry but I have been following this thread for a while and everything that Gordon (and I mentioned in my first post) is accurate. This is a routing problem not a NATing problem. Even if it can be handled via NATing (which I serious doubt) it would require ridiculous server resources which are not necessary. Please look at our links carefully, This problem is simply handled by proper routing rules. Whether those are implemented via my suggested generic linux rules or via the Redhat specific rules Gordon suggested that is the proper way to handle the problem. I too am puzzled by what you are trying to do here otherwise.

    And just for the record I have a server with two interfaces on the same broadcast network and did exactly what I showed in the link I sent and got exactly what I wanted. From this single sentence you seem to want to do what either Gordon or I suggested in the links we sent. Why do you insist on playing with iptables when that is not the proper tool for this job?

  • I do not insist on anything else then there are missing things in the picture… it is that simple nothing more then that. If the thread poster will bother to clear the picture then I will be able to understand. I do not want anything and I do not need anything for me, my servers and routing works just great but the thread started by someone which is not here anymore so if everything works for me and if everything works for you and Gordon I really do not think any more time or words should be invested in something that is unclear at-least to me.

    Eliezer

  • Hello,

    i follow your discussion. The first 2 posts using multiple default routes solve my problem perfect.

    Thank you all.

    J

    Am 2015-12-30 17:21, schrieb Eliezer Croitoru: