I am experimenting with routing tables to obtain a little understanding of how things work.

I have a kvm hypervisor host (KVM1) with two physical Ethernet nics configured as bridges (br0 and br1). KVM1 br0 is configured with a public ipv4 address [x.y.z.42/24] and br1 is configured with a private ipv4 address [192.168.216.42/24]. A second kvm hypervisor, KVM2, is similarly configured albeit with different IPv4 addresses of course. Packet forwarding is enabled on both hypervisor hosts and both have IPTABLES configured, albeit not for masquerading.

Guest systems on each of the two kvm hosts are also configured with two virtual nics, eth0 and eth1, mapped to br0 and br1 respectively. Guests on KVM2 usually have eth0 downed and eth1 configured with a private ipv4 address in this address space, [192.168.216.0/24]. Guests on KVM1 usually have eth0 configured with a public ipv4 and always have eth1 configured with a private address also belonging to
[192.168.216.0/24].

The KVM1 hypervisor host has its br1 connected via x-over cable to the br1 port of the KVM2 hypervisor host. The reason for the X-over is to allow matching guests on the different KVM systems to conduct large network transfers between themselves without hitting the LAN.

We have a gateway router whose eth1 faces the LAN with the public address [x.y.z.1/24] and the WAN with an address in the form
[x.y.A.0/30]. The ASCII diagram below represents the situation. You will need to view this part with a monospaced font.

                                                              ~
   ISP ethx[x.y.A.2/30] >-----< eth0[x.y.A.3/30]              |
              ISP                      GWAY                   |
                                eth1[x.y.z.1/24] >----------< |
                                       GWAY                   |
                                                              |
         br0[x.y.z.42/24] >---------------------------------< |
               KVM2                                           |
                                     br0[x.y.z.42/24] >-----< |
                                             KVM1             ~
      br1[192.168.216.43/24] >--X--< br1[192.168.216.42/24]
               KVM2                          KVM1        |
          |                                              |
          | >-----< eth1[192.168.216.21/24]              |
          | >-----< eth1[192.168.216.25/24]              |
                                                         |
                        eth1[192.168.216.221/24] >-----< |
                        eth1[192.168.216.223/24] >-----< |

This setup works as intended. However, it is sufficiently oddball that I consider it a good candidate to discover for myself the practical issues surrounding network routing, a field of knowledge of which I readily admit profound ignorance.

My trial is to allow a ping from eth1[192.168.216.221/24] to reach any other public addressed [x.y.z.0/24]i/f on the LAN and successfully be returned.

The questions are:

1. What additional (virtual) network i/f(s) needsbe configured to get this to happen?

2. Do any systems require static routes to be preconfigured? If so, which systems and what is that configuration?

3. Is masquerading required or not?

My thoughts so far:

The default gateway for all of the public hosts on our LAN is eth1[x.y.z.1/24] so it seems necessary that eth1 on GWAY must have some interface that recognizes [192.168.216.0/24] and a route that will send the ping back to the public i/f from which it originates, br0[x.y.z.42/24].

The KVM1 host must have a route to allow [192.168.216.221/24] out onto the LAN via br0[x.y.z.42/24] for all destinations other than
[192.168.216.0/24] and route that netblock via br1[192.168.216.42/24].

If this is anywhere near the actual requirements then I think something like this is required on GWAY eth1[x.y.z.1/24]:

ifcfg-eth1:192168
BOOTPROTO=none BROADCAST2.168.255.255
DEVICE=eth1:192168
IPADDR2.168.0.1
IPV6INIT=no NETMASK%5.255.0.0
NETWORK2.168.0.0
#ONBOOT=yes ONPARENT=yes

route-eth1
192.168.216.0/24 via x.y.z.42 dev eth1

And on KVM1 I would need:

ifcfg-br0
BOOTPROTO="none"
DEFROUTE="yes"
DEVICE="br0"
GATEWAY="x.y.z.1"
IPADDR="x.y.z.42"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
NAME="System br0"
NETMASK="255.255.255.0"
NETWORK="x.y.z.0"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Bridge"
USERCTL="no

ifcfg-br1
BOOTPROTO="none"
DEVICE="br1"
GATEWAY="x.w.z.42"
IPADDR="192.168.216.42"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
NAME="System br1"
NM_CONTROLLED="no"
ONBOOT="yes"
PREFIX="24"
TYPE="Bridge"
USERCTL="no"

How far off am I in these musings? I would like to have some clue of the trouble I am in for before I go and break something. Your comments and suggestions are welcome.