I’m createing a distributed cloud infrastructure, and I need to connect VMs on multiple Host Machines. So, that's what we're going to do today.

A solid knowledge of routing (IPTables) and networking is required for the information below. Technically, this is what containers like Docker or software routers do internally when they need to connect two different network.

Let's assume:

• HostMachine1 has VM’s on network 10.0.0.1/24

• HostMachine2 has VM’s on network 192.168.0.1/24

Our Gateway1 and Gateway2 have two Network Interfaces/NIC cards. Gateway1 and Gateway2 are connected by a switch, hence they're on same network, as well as their respective VM networks — as they have two NIC cards connected.

Let assume Gateway1's IP address is 10.0.0.2, and Gateway2's is IP 192.168.0.2.

Both Gateway1 and Gateway2 can connect to each other, as they are directly connected.

Here's my current configuration on Gateway1, which is our target router. I've got these Network Interfaces on it.

enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:9d:08:3f brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.169/24 brd 10.0.0.255 scope global enp0s9
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe9d:83f/64 scope link
       valid_lft forever preferred_lft forever
 enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:7b:12:89 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.120/16 brd 192.168.255.255 scope global enp0s10
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe7b:1289/64 scope link

Now we need to add routing table configurations on either Gateway1 or Gateway2 to forward packets from one network to another.

We can do this two ways.

Either by creating a network bridge:

 # Install brige utils that gives us easy commands to create bridge network 
 apt-get install bridge-utils 
 or 
 yum install bridge-utils 

 # Create new Bridge 
 brctl addbr br0 

 # Enable Spanning Tree Support if you need 
 brctl stp br0 on 

 # Make sure you get your Devices down before creating bridge 
 #and explicity assign 0.0.0.0 to make sure they loose ip 
 ifconfig enp0s9 0.0.0.0 down 
 ifconfig enp0s10 0.0.0.0 down 

 # Add them to our newly created bridge network 
 brctl addif br0 enp0s9 
 brctl addif br0 enp0s10 

 # Finally get all interfaces up. 
 ifconfig enp0s9 up 
 ifconfig enp0s10 up 
 ifconfig br0 up 

Or by modifying routing table. I’ll explain that second concept here.

Enable forwarding in the kernel:

 echo 1 >> /proc/sys/net/ipv4/ip_forward 

To set this value on boot, uncomment this line in/etc/sysctl.conf

#net.ipv4.ip_forward=1

Now, I need to route traffic from one Interface to another using routing tables. The following statements can do that.

# Always accept loopback traffic
iptables -A INPUT -i lo -j ACCEPT

# We allow traffic from the HostMachine1 side
iptables -A INPUT -i enp0s9  -j ACCEPT

# We allow traffic from the HostMachine2 side
iptables -A INPUT -i enp0s10  -j ACCEPT

######################################################################
#
#                         ROUTING
#
######################################################################

# enp0s9 is HostMachine1 Network
# enp0s10 is HostMachine2 Network

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Masquerade.
iptables -t nat -A POSTROUTING -o enp0s10  -j MASQUERADE

# fowarding
iptables -A FORWARD -i enp0s9 -o enp0s10  -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow outgoing connections from the HostMachine1 side.
iptables -A FORWARD -i enp0s10 -o enp0s9  -j ACCEPT

Finally, on HostMachine1, route all traffic on subnet 192.168.0.1/24 to Gateway1. I use my Mac as my host machine, hence the command: sudo route -n add 192.168.1.1/24 10.0.0.169.

If you are using any *nix systems, the command you need to use would be: ip route add 192.168.1.1/24 via 10.0.0.169 dev.