SCI-CD

h1. Network setup

{{toc}}

Back to [[SETUP]] page.

Network configuration may be highly various.

h2. Basic schema - one ethernet to all.

One ethernet, one subnet, internet connection provided by external (not in claster) router.

By default installer create bridge named xen-br0. You can customize parameters by editing /etc/network/interfaces.

In this case you must have nodes connected to gigabit ethernet switch.

By default it looks like:

<pre>

auto xen-br0

iface xen-br0 inet static

        address 192.168.5.88

        netmask 255.255.255.0

        network 192.168.5.0

        broadcast 192.168.5.255

        gateway 192.168.5.1

        bridge_ports eth0

        bridge_stp off

        bridge_fd 0

#       up ifconfig eth0 mtu 9000

#       up ifconfig xen-br0 mtu 9000

</pre>

Important parameters besides ipv4 settings is:

<pre>

bridge_ports eth0

</pre>

- means that physical interface eth0 enslaved to this bridge.

<pre>

up ifconfig eth0 mtu 9000

up ifconfig xen-br0 mtu 9000

</pre>

- setting jumbo frame on bridge for more network speed and less cpu utilization.

It will be actual on interface where drbd link will be.

However, setting mtu higher than 1500 will cause problems with any network equipment that

doesn't support jumbo frames. That's the reason because it option commented out by default.

h2. Default schema - two ethernets, one for interlink(ganeti interoperation+drbd link) and one for lan.

This schema suits most cases. It doesn't required a gigabit switch, provide good performance and reliability.

Two gigabit network interfaces on the nodes are connected directly or via a gigabit

switch (if you want more than two nodes in the cluster).

Other interfaces connected to lan.

Lan failure doesn't affect cluster in this setup.

This is /etc/network/interfaces file for this setup:

<pre>auto xen-br0

iface xen-br0 inet static

	address 192.168.236.1

	netmask 255.255.255.0

	network 192.168.236.0

	broadcast 192.168.236.255

        bridge_ports eth0

        bridge_stp off

        bridge_fd 0

#	up ifconfig eth0 mtu 9000

#	up ifconfig xen-br0 mtu 9000

auto xen-lan

iface xen-lan inet static

	address 192.168.5.55

	netmask 255.255.255.0

	network 192.168.5.0

	broadcast 192.168.5.255

	gateway 192.168.5.1

	bridge_ports eth1

	bridge_stp off

	bridge_fd 0

</pre>

xen-br0 used by ganeti interoperation and drbd link, it was configured by the installer.

Also the dns server and the gateway was configured by the installer - it will be our service instance(sci) address.

xen-lan used by lan connection, its configuration must be added by hands.

In this network configuration you must fill these variables in sci.conf:

NODE1_IP - already configured by installer.

NODE1_NAME - already configured by installer.

NODE2_IP - set interlink ip address of second node. e.g. 192.168.236.2

NODE2_NAME - set second node name. e.g. gnt2

NODE1_LAN_IP - lan ip for first node. It will be available by dns name $NODE1_NAME-lan. 192.168.5.55

NODE2_LAN_IP - lan ip for second node. It will be available by dns name $NODE2_NAME-lan. e.g. 192.168.5.58

CLUSTER_IP - cluster address in lan. Must not match any exist host address in lan. 192.168.5.35

CLUSTER_NAME - cluster name in lan. In will be available by dns name $CLUSTER_NAME.

SCI_LAN_IP - if you want presence sci intance in your lan, assign ip. e.g. 192.168.5.59

h2. Mupltiple bridges with routing, firewalling and wan access.

Here is a bit more complicated network setup.

In this setup we have, for example, two private netwokrs and wan by ethernet. All routing and firewalling

performed by separate firewall instance in our cluster. This setup fit when you don't have expensive hardware routers and firewalls.

This is /etc/network/interfaces file in this setup:

<pre>

auto lan

iface lan inet static

	address 192.168.21.10

	netmask 255.255.255.0

        bridge_ports eth0

        bridge_stp off

        bridge_fd 0

auto server

iface server inet static

	address 192.168.20.10

	netmask 255.255.255.0

	gateway 192.168.20.1

        bridge_ports eth1

        bridge_stp off

        bridge_fd 0

        up ifconfig eth1 mtu 9000

        up ifconfig dmz mtu 9000

auto wan1

iface wan1 inet manual

        bridge_ports eth2

        bridge_stp off

        bridge_fd 0

</pre>

In this example we have separate lan interfaces, server interface(in this case servers separated from lan and

clients go to servers thru router) and wan interface. server interface - ganeti interoperation dev and drbd link

interfase, so there is mtu 9000.

There is no address in wan for hypervisor, although we recommend you to get subnet from

your ISP in order to assign IP addresses to nodes to management it even if router instance

is down.

In this network configuration you must fill these variables in sci.conf:

NODE1_IP - already configured by installer.

NODE1_NAME - already configured by installer.

NODE2_IP - set interlink ip address of second node. e.g. 192.168.20.2

NODE2_NAME - set second node name. e.g. gnt2

NODE1_LAN_IP - lan ip for first node. It will be available by dns name $NODE1_NAME-lan. 192.168.21.55

NODE2_LAN_IP - lan ip for second node. It will be available by dns name $NODE2_NAME-lan. e.g. 192.168.21.58

CLUSTER_IP - cluster address in lan. Must not match any exist host address in lan. 192.168.21.35

CLUSTER_NAME - cluster name in lan. In will be available by dns name $CLUSTER_NAME.

SCI_LAN_IP - if you want presence sci intance in your lan, assign ip. e.g. 192.168.21.59

Here is an example /etc/network/interfaces in router instance:

<pre>

auto eth0

iface eth0 inet static

   address 192.168.20.1

   netmask 255.255.255.0

auto eth1

iface eth1 inet static

   address 192.168.21.1

   netmask 255.255.255.0

auto eth2

iface eth2 inet static

   address 1.1.1.2

   netmask 255.255.255.0

   address 1.1.1.1

</pre>

Where eth0 linked to bridge server, eth1 linked to lan, eth2 linked to wan.

h2. Datacenter schema - separate interfaces for lan, ganeti interoperation, drbd link.

If you have powerful networking infrastructure

Here we have separate interfaces for ganeti interoperation(in this case it may be named management interface)

<pre>auto mgmt

iface mgmt inet static

    address 192.168.236.1

    netmask 255.255.255.0

    network 192.168.236.0

    gateway 192.168.236.1

    broadcast 192.168.236.255

        bridge_ports eth0

        bridge_stp off

        bridge_fd 0

auto xen-san

iface xen-san inet static

    address 192.168.237.1

    netmask 255.255.255.0

    network 192.168.237.0

    broadcast 192.168.237.255

    bridge_ports eth1

    bridge_stp off

    bridge_fd 0

    up ifconfig eth1 mtu 9000

    up ifconfig xen-san mtu 9000

auto xen-lan

iface xen-lan inet manual

    bridge_ports eth2

    bridge_stp off

    bridge_fd 0

</pre>

In this example nodes don't have addresses in lan.

You must fill these vars in sci.conf to create cluster fits this network config:

NODE1_IP - already configured by installer.

NODE1_NAME - already configured by installer.

NODE2_IP - set interlink ip address of second node. e.g. 192.168.236.2

NODE2_NAME - set second node name. e.g. gnt2

NODE1_SAN_IP - lan ip for first node. It will be available by dns name $NODE1_NAME-san. 192.168.237.1

NODE2_SAN_IP - lan ip for second node. It will be available by dns name $NODE2_NAME-san. e.g. 192.168.237.2

CLUSTER_IP - cluster address in lan. Must not match any exist host address in lan. 192.168.236.35

CLUSTER_NAME - cluster name in lan. In will be available by dns name $CLUSTER_NAME.

SCI_LAN_IP - if you want presence sci intance in your lan, assign ip. e.g. 192.168.35.5

SCI_LAN_NETMASK - your nodes don't have addresses in lan, so you must enter netmask for this segment by hand. e.g. 255.255.255.0

SCI_LAN_GATEWAY - your nodes don't have addresses in lan, so you must enter gateway for this segment by hand. e.g. 192.168.35.1

Of course, it is easy to use VLANS in datacenter conditions. Next example will explain how. However, remember it is recommended

that drbd link must be on separate ethernet.

h2. VLAN schema

If you have managed switches, you can set networking with VLANs.

You should add something like this for each VLAN:

<pre>

auto eth0.55

iface eth0.55 inet manual

        up ifconfig eth0.55 up

auto bridge-example-vlan

iface bridge-example-vlan inet manual

        up brctl addbr bridge-example-vlan

        up brctl addif bridge-example-vlan eth0.55

        up brctl stp bridge-example-vlan off

        up ifconfig bridge-example-vlan up

        down ifconfig bridge-example-vlan down

        down brctl delbr bridge-example-vlan

</pre>

Where 55 - VLAN number.

In this example node don't have an ip address in this VLAN, although you could

assign an ip to bridge just like standard bridge.

Alternative schema is:

<pre>

auto vlan55

iface vlan55 inet manual

   vlan_raw_device eth0

auto bridge-example-vlan

iface bridge-example-vlan inet manual

           bridge_ports vlan55

        bridge_stp off

        bridge_fd 0

</pre>

It do the same, but in another way.