Проект

Общее

Профиль

SETUP » История » Версия 10

Владимир Ипатов, 08.11.2012 01:26

1 1 Dmitry Chernyak
h1. SETUP
2 1 Dmitry Chernyak
3 1 Dmitry Chernyak
{{toc}}
4 1 Dmitry Chernyak
5 1 Dmitry Chernyak
Ensure both nodes are up.
6 1 Dmitry Chernyak
7 1 Dmitry Chernyak
If you planning to use the secondary network for SAN and DRBD synchronization, you
8 1 Dmitry Chernyak
should configure secondary IP interfaces manually on both nodes at this time.
9 1 Dmitry Chernyak
10 1 Dmitry Chernyak
Log in to the first node via ssh. Due to lack of DNS there may be
11 1 Dmitry Chernyak
a minute timeout before password prompt.
12 1 Dmitry Chernyak
13 2 Владимир Ипатов
h2. NETWORK CONFIGURATION
14 2 Владимир Ипатов
15 2 Владимир Ипатов
Network configuration may be highly various.
16 2 Владимир Ипатов
Here we describe several schemas.
17 2 Владимир Ипатов
18 7 Владимир Ипатов
h3. Basic schema - one ethernet to all.
19 2 Владимир Ипатов
20 3 Владимир Ипатов
one ethernet, one subnet, internet connection provided by external (not in claster) router.
21 1 Dmitry Chernyak
By default installer create bridge named xen-br0. You can customize parameters by editing /etc/network/interfaces.
22 7 Владимир Ипатов
In this case you must have nodes connected to gigabit ethernet switch.
23 3 Владимир Ипатов
By default it looks like:
24 3 Владимир Ипатов
<pre>
25 3 Владимир Ипатов
auto xen-br0
26 3 Владимир Ипатов
iface xen-br0 inet static
27 3 Владимир Ипатов
        address 192.168.5.88
28 3 Владимир Ипатов
        netmask 255.255.255.0
29 3 Владимир Ипатов
        network 192.168.5.0
30 3 Владимир Ипатов
        broadcast 192.168.5.255
31 3 Владимир Ипатов
        gateway 192.168.5.1
32 3 Владимир Ипатов
        bridge_ports eth0
33 3 Владимир Ипатов
        bridge_stp off
34 3 Владимир Ипатов
        bridge_fd 0
35 3 Владимир Ипатов
#       up ifconfig eth0 mtu 9000
36 3 Владимир Ипатов
#       up ifconfig xen-br0 mtu 9000
37 3 Владимир Ипатов
</pre>
38 3 Владимир Ипатов
Important parameters besides ipv4 settings is:
39 3 Владимир Ипатов
bridge_ports eth0 - means that physical interface eth0 enslaved to this bridge.
40 3 Владимир Ипатов
41 3 Владимир Ипатов
up ifconfig eth0 mtu 9000
42 3 Владимир Ипатов
up ifconfig xen-br0 mtu 9000 - setting jumbo frame on bridge for more net speed and less cpu utilization.
43 3 Владимир Ипатов
It will be actual on interface where drbd link will be.
44 3 Владимир Ипатов
However, setting mtu higher than 1500 will cause problems with any network equipment that
45 3 Владимир Ипатов
doesn't support jumbo frames. That's the reason because it option commented out by default.
46 3 Владимир Ипатов
47 3 Владимир Ипатов
Also it is important to specify broadcast and network adresses - it will help automatically
48 3 Владимир Ипатов
fullfill /etc/ganeti/networks file(a file that specify networks for instances).
49 1 Dmitry Chernyak
However, it ins't required.
50 3 Владимир Ипатов
51 7 Владимир Ипатов
h3. Default schema - two ethernets, one for interlink(ganeti interoperation+drbd link) and one for lan.
52 1 Dmitry Chernyak
53 7 Владимир Ипатов
This schema suits most cases. It doesn't required gigabit switch, provide good performance and reliability.
54 7 Владимир Ипатов
Two gigabit network interfaces on nodes connected directly or via gigabit switch(if you want more than two nodes in cluster).
55 7 Владимир Ипатов
Other interfaces connected to lan. Routing, firewalling, dhcp, dns in lan performed by external router or server.
56 7 Владимир Ипатов
Lan failure doesn't affect cluster in this setup.
57 8 Владимир Ипатов
This is /etc/network/interfaces file for this setup:
58 8 Владимир Ипатов
<pre>auto xen-br0
59 8 Владимир Ипатов
iface xen-br0 inet static
60 8 Владимир Ипатов
	address 192.168.236.1
61 8 Владимир Ипатов
	netmask 255.255.255.0
62 8 Владимир Ипатов
	network 192.168.236.0
63 8 Владимир Ипатов
	broadcast 192.168.236.255
64 8 Владимир Ипатов
	gateway 192.168.236.15
65 8 Владимир Ипатов
        bridge_ports eth0
66 8 Владимир Ипатов
        bridge_stp off
67 8 Владимир Ипатов
        bridge_fd 0
68 8 Владимир Ипатов
#	up ifconfig eth0 mtu 9000
69 8 Владимир Ипатов
#	up ifconfig xen-br0 mtu 9000
70 7 Владимир Ипатов
71 8 Владимир Ипатов
auto xen-lan
72 8 Владимир Ипатов
iface xen-lan inet static
73 8 Владимир Ипатов
	address 192.168.5.55
74 8 Владимир Ипатов
	netmask 255.255.255.0
75 8 Владимир Ипатов
	network 192.168.5.0
76 8 Владимир Ипатов
	broadcast 192.168.5.255
77 8 Владимир Ипатов
	bridge_ports eth1
78 8 Владимир Ипатов
	bridge_stp off
79 8 Владимир Ипатов
	bridge_fd 0
80 8 Владимир Ипатов
</pre>
81 8 Владимир Ипатов
xen-br0 used by ganeti interoperation and drbd link, it was configured in installer.
82 8 Владимир Ипатов
Also gateway and dns server was configured in installer - it will be our service instance(sci) address.
83 8 Владимир Ипатов
xen-lan used by lan connection, its configuration must be added by hand.
84 8 Владимир Ипатов
In this network configuration you must fill these variables in sci.conf:
85 9 Владимир Ипатов
NODE1_IP - already configured by installer.
86 9 Владимир Ипатов
NODE1_NAME - already configured by installer.
87 9 Владимир Ипатов
NODE2_IP - set interlink ip address of second node. e.g. 192.168.236.2
88 9 Владимир Ипатов
NODE2_NAME - set second node name. e.g. gnt2
89 9 Владимир Ипатов
NODE1_LAN_IP - lan ip for first node. It will be available by dns name $NODE1_NAME-lan. 192.168.5.55
90 9 Владимир Ипатов
NODE2_LAN_IP - lan ip for first node. It will be available by dns name $NODE1_NAME-lan. e.g. 192.168.5.58
91 9 Владимир Ипатов
CLUSTER_IP - cluster address in lan. Must not match any exist host address in lan. 192.168.5.35
92 9 Владимир Ипатов
CLUSTER_NAME - cluster name in lan. In will be available by dns name $CLUSTER_NAME.
93 7 Владимир Ипатов
94 7 Владимир Ипатов
h3. Mupltiple bridges with routing, firewalling and wan access.
95 7 Владимир Ипатов
96 3 Владимир Ипатов
Here is a bit more complicated network setup.
97 3 Владимир Ипатов
In this setup we have, for example, two private netwokrs and wan by ethernet. All routing and firewalling
98 7 Владимир Ипатов
performed by separate firewall instance in our cluster. This setup fit when you don't have expensive hardware routers and firewalls.
99 3 Владимир Ипатов
This is /etc/network/interfaces file in this setup:
100 3 Владимир Ипатов
<pre>
101 5 Владимир Ипатов
auto lan
102 5 Владимир Ипатов
iface lan inet static
103 5 Владимир Ипатов
	address 192.168.21.10
104 5 Владимир Ипатов
	netmask 255.255.255.0
105 1 Dmitry Chernyak
        bridge_ports eth0
106 1 Dmitry Chernyak
        bridge_stp off
107 1 Dmitry Chernyak
        bridge_fd 0
108 5 Владимир Ипатов
109 5 Владимир Ипатов
auto dmz
110 5 Владимир Ипатов
iface dmz inet static
111 5 Владимир Ипатов
	address 192.168.20.10
112 5 Владимир Ипатов
	netmask 255.255.255.0
113 5 Владимир Ипатов
	gateway 192.168.20.1
114 5 Владимир Ипатов
        bridge_ports eth1
115 5 Владимир Ипатов
        bridge_stp off
116 5 Владимир Ипатов
        bridge_fd 0
117 5 Владимир Ипатов
        up ifconfig eth1 mtu 9000
118 5 Владимир Ипатов
        up ifconfig dmz mtu 9000
119 5 Владимир Ипатов
120 5 Владимир Ипатов
auto wan1
121 5 Владимир Ипатов
iface wan1 inet manual
122 5 Владимир Ипатов
        bridge_ports eth2
123 5 Владимир Ипатов
        bridge_stp off
124 5 Владимир Ипатов
        bridge_fd 0
125 1 Dmitry Chernyak
</pre>
126 5 Владимир Ипатов
In this example we have separate lan interfaces, dmz interface(it isn't actually dmz,
127 5 Владимир Ипатов
it just named this) and wan interface. dmz interface - ganeti master dev and drbd link
128 5 Владимир Ипатов
interfase, so there is mtu 9000.
129 6 Владимир Ипатов
Also in this example you must edit MASTER_NETDEV and LINK_NETDEV in /etc/sci/sci.conf from default xen-br0 to dmz.
130 5 Владимир Ипатов
There is no address in wan for hypervisor, although we recommend you to get subnet from
131 5 Владимир Ипатов
your ISP in order to assign IP addresses to nodes to management it even if router instance
132 5 Владимир Ипатов
is down.
133 5 Владимир Ипатов
134 5 Владимир Ипатов
Here is an example /etc/network/interfaces in router instance:
135 5 Владимир Ипатов
<pre>
136 5 Владимир Ипатов
auto eth0
137 5 Владимир Ипатов
iface eth0 inet static
138 5 Владимир Ипатов
   address 192.168.20.1
139 5 Владимир Ипатов
   netmask 255.255.255.0
140 5 Владимир Ипатов
141 5 Владимир Ипатов
auto eth1
142 5 Владимир Ипатов
iface eth1 inet static
143 5 Владимир Ипатов
   address 192.168.21.1
144 5 Владимир Ипатов
   netmask 255.255.255.0
145 5 Владимир Ипатов
146 5 Владимир Ипатов
auto eth2
147 5 Владимир Ипатов
iface eth2 inet static
148 5 Владимир Ипатов
   address 1.1.1.2
149 5 Владимир Ипатов
   netmask 255.255.255.0
150 1 Dmitry Chernyak
   address 1.1.1.1
151 1 Dmitry Chernyak
</pre>
152 1 Dmitry Chernyak
Where eth0 linked to bridge dmz, eth1 linked to lan, eth2 linked to wan.
153 7 Владимир Ипатов
154 7 Владимир Ипатов
h3. Datacenter schema - separate interfaces for lan, ganeti interoperation, drbd link.
155 10 Владимир Ипатов
156 7 Владимир Ипатов
If you have powerful networking infrastructure
157 5 Владимир Ипатов
158 5 Владимир Ипатов
h3. VLAN schema
159 5 Владимир Ипатов
160 5 Владимир Ипатов
If you have managed switches, you can set networking with VLANs.
161 5 Владимир Ипатов
You should add something like this for each VLAN:
162 5 Владимир Ипатов
<pre>
163 5 Владимир Ипатов
auto eth0.55
164 5 Владимир Ипатов
iface eth0.55 inet manual
165 5 Владимир Ипатов
        up ifconfig eth0.55 up
166 5 Владимир Ипатов
167 5 Владимир Ипатов
auto bridge-example-vlan
168 5 Владимир Ипатов
iface bridge-example-vlan inet manual
169 5 Владимир Ипатов
        up brctl addbr bridge-example-vlan
170 5 Владимир Ипатов
        up brctl addif bridge-example-vlan eth0.55
171 5 Владимир Ипатов
        up brctl stp bridge-example-vlan off
172 5 Владимир Ипатов
        up ifconfig bridge-example-vlan up
173 5 Владимир Ипатов
        down ifconfig bridge-example-vlan down
174 5 Владимир Ипатов
        down brctl delbr bridge-example-vlan
175 5 Владимир Ипатов
</pre>
176 5 Владимир Ипатов
Where 55 - VLAN number.
177 5 Владимир Ипатов
In this example node don't have an ip address in this VLAN, although you could
178 5 Владимир Ипатов
assign an ip to bridge just like standard bridge.
179 5 Владимир Ипатов
180 5 Владимир Ипатов
Alternative schema is:
181 5 Владимир Ипатов
<pre>
182 5 Владимир Ипатов
auto vlan55
183 5 Владимир Ипатов
iface vlan55 inet manual
184 5 Владимир Ипатов
   vlan_raw_device eth0
185 5 Владимир Ипатов
186 5 Владимир Ипатов
auto bridge-example-vlan
187 5 Владимир Ипатов
iface bridge-example-vlan inet manual
188 5 Владимир Ипатов
           bridge_ports vlan55
189 5 Владимир Ипатов
        bridge_stp off
190 5 Владимир Ипатов
        bridge_fd 0
191 2 Владимир Ипатов
</pre>
192 1 Dmitry Chernyak
It do the same, but in another way.
193 1 Dmitry Chernyak
194 1 Dmitry Chernyak
h2. DEFINING ENVIRONMENT
195 1 Dmitry Chernyak
196 1 Dmitry Chernyak
Edit @/etc/sci/sci.conf@
197 1 Dmitry Chernyak
198 8 Владимир Ипатов
Most of values rely of your network setup. In section network setup it was described for most cases.
199 8 Владимир Ипатов
200 8 Владимир Ипатов
Here is additional notes about sci.conf configuring:
201 8 Владимир Ипатов
202 1 Dmitry Chernyak
* You should specify node1 and node2 data as you have installed them.
203 1 Dmitry Chernyak
*NOTE*: You can setup the cluster even with one node. In this case just leave NODE2_
204 1 Dmitry Chernyak
lines as is. In fact this is a dangerous setup, so you will be warned about this duging
205 1 Dmitry Chernyak
the procedures.
206 1 Dmitry Chernyak
207 1 Dmitry Chernyak
* You should specify the cluster's name and IP.
208 1 Dmitry Chernyak
209 1 Dmitry Chernyak
* NODE#_SAN_IP should be specified on both nodes or none.
210 1 Dmitry Chernyak
211 8 Владимир Ипатов
* NODE#_LAN_IP should be specified on both nodes or none.
212 8 Владимир Ипатов
213 1 Dmitry Chernyak
* If you haven't Internet uplink or have a local package mirrors, you should correct
214 1 Dmitry Chernyak
APT_ - settings.
215 1 Dmitry Chernyak
216 6 Владимир Ипатов
* If you need to uplink to the DNS hierarchy other than root hint zones, specify DNS_FORWARDERS
217 6 Владимир Ипатов
(note trailing ';').
218 1 Dmitry Chernyak
219 8 Владимир Ипатов
* MASTER_NETDEV - master interface name for cluster address. Auto-detected by default.
220 6 Владимир Ипатов
221 8 Владимир Ипатов
* LAN_NETDEV - Network interface to bind to virtual machies by default. Auto-detected by default.
222 6 Владимир Ипатов
223 6 Владимир Ипатов
* RESERVED_VOLS - list of volumes ignored by ganeti. Comma separated. You must specify vg for all volumes in this list.
224 6 Владимир Ипатов
225 1 Dmitry Chernyak
226 1 Dmitry Chernyak
h2. SETUP CLUSTER
227 1 Dmitry Chernyak
228 1 Dmitry Chernyak
Issue:
229 1 Dmitry Chernyak
230 1 Dmitry Chernyak
<pre>
231 1 Dmitry Chernyak
# sci-setup cluster
232 1 Dmitry Chernyak
</pre>
233 1 Dmitry Chernyak
234 1 Dmitry Chernyak
Check and confirm settings printed.
235 1 Dmitry Chernyak
236 1 Dmitry Chernyak
The process will go on.
237 1 Dmitry Chernyak
238 1 Dmitry Chernyak
Next you will be prompted to accept ssh key from node2 and for the root's password to node2.
239 1 Dmitry Chernyak
240 1 Dmitry Chernyak
On finish you will look something like this:
241 1 Dmitry Chernyak
242 1 Dmitry Chernyak
<pre>
243 1 Dmitry Chernyak
Verify
244 1 Dmitry Chernyak
Wed Jan 12 15:36:10 2011 * Verifying global settings
245 1 Dmitry Chernyak
Wed Jan 12 15:36:10 2011 * Gathering data (1 nodes)
246 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying node status
247 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying instance status
248 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying orphan volumes
249 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying orphan instances
250 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying N+1 Memory redundancy
251 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Other Notes
252 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Hooks Results
253 1 Dmitry Chernyak
Node                    DTotal  DFree MTotal MNode MFree Pinst Sinst
254 1 Dmitry Chernyak
gnt1.ganeti.example.org 100.0G 100.0G  1020M  379M  625M     0     0
255 1 Dmitry Chernyak
gnt2.ganeti.example.org 100.0G 100.0G  1020M  379M  625M     0     0
256 1 Dmitry Chernyak
If all is ok, proceed with /usr/local/sbin/sci-setup service
257 1 Dmitry Chernyak
</pre>
258 1 Dmitry Chernyak
259 1 Dmitry Chernyak
h2. SETUP SERVICE INSTANCE
260 1 Dmitry Chernyak
261 1 Dmitry Chernyak
The service instance is named 'sci' and have a few aliases.
262 1 Dmitry Chernyak
On setup, it's IP address is determined from @/etc/resolv.conf@ of your first node.
263 1 Dmitry Chernyak
This instance will be hardcoded in @/etc/hosts@ file of all cluster nodes and instances.
264 1 Dmitry Chernyak
265 1 Dmitry Chernyak
Issue:
266 1 Dmitry Chernyak
267 1 Dmitry Chernyak
<pre>
268 1 Dmitry Chernyak
# sci-setup service
269 1 Dmitry Chernyak
</pre>
270 1 Dmitry Chernyak
271 1 Dmitry Chernyak
You'll see the progress of DRBD syncing disks, then the message
272 1 Dmitry Chernyak
<pre>
273 1 Dmitry Chernyak
* running the instance OS create scripts...
274 1 Dmitry Chernyak
</pre>
275 1 Dmitry Chernyak
appears. The further may take a while. The process finishes with
276 1 Dmitry Chernyak
<pre>
277 1 Dmitry Chernyak
* starting instance...
278 1 Dmitry Chernyak
</pre>
279 1 Dmitry Chernyak
message.
280 1 Dmitry Chernyak
281 1 Dmitry Chernyak
Now you can log on to the sci instance using:
282 1 Dmitry Chernyak
283 1 Dmitry Chernyak
<pre>
284 1 Dmitry Chernyak
# gnt-instance console sci
285 1 Dmitry Chernyak
</pre>
286 1 Dmitry Chernyak
287 1 Dmitry Chernyak
Log in as root, the password is empty.
288 1 Dmitry Chernyak
*NOTE*: Due to empty password all remote connections to new instance is prohibited.
289 1 Dmitry Chernyak
You should change password and install @openssh-server@ package manually after
290 1 Dmitry Chernyak
successful bootstrap procedure.
291 1 Dmitry Chernyak
292 1 Dmitry Chernyak
h2. SERVICE INSTANCE BOOTSTRAP
293 1 Dmitry Chernyak
294 1 Dmitry Chernyak
The system will setup itself via puppet. This is the iterative process. You can monitor
295 1 Dmitry Chernyak
it by looking into @/var/log/daemon.log@. At start there is no @less@ command yet, so
296 1 Dmitry Chernyak
you can use @more@, @cat@, @tail@ or @tail -f@ until @less@ will be auto-installed.
297 1 Dmitry Chernyak
298 1 Dmitry Chernyak
By default the iterations are repeated in 20 minutes. To shorten the wait time you can
299 1 Dmitry Chernyak
issue
300 1 Dmitry Chernyak
301 1 Dmitry Chernyak
<pre>
302 1 Dmitry Chernyak
# /etc/init.d/puppet restart
303 1 Dmitry Chernyak
</pre>
304 1 Dmitry Chernyak
305 1 Dmitry Chernyak
and then look into @daemon.log@ how it finishes.
306 1 Dmitry Chernyak
307 1 Dmitry Chernyak
Repeat this a few times until puppet will do nothing in turn.
308 1 Dmitry Chernyak
309 1 Dmitry Chernyak
h2. PREPARING FOR NEW INSTANCES
310 1 Dmitry Chernyak
311 1 Dmitry Chernyak
New instances are created just by regular Ganeti commands such as:
312 1 Dmitry Chernyak
313 1 Dmitry Chernyak
<pre>
314 1 Dmitry Chernyak
gnt-instance add -t drbd -o debootstrap+default -s 10g -B memory=256m -n NODE1_NAME:NODE2_NAME INSTANCE_NAME
315 1 Dmitry Chernyak
</pre>
316 1 Dmitry Chernyak
317 1 Dmitry Chernyak
Altought, some tuning hooks are provided by SCI-CD project:
318 1 Dmitry Chernyak
# Each instance has installed @puppet@ for autoconfiguration and @openssh-client@ for file transfers etc.
319 1 Dmitry Chernyak
# The instance uses pygrub to boot kernel from /vmlinuz & Co on the innstance's own disk.
320 1 Dmitry Chernyak
# The instance's network interfaces may be set up automatically as described below.
321 1 Dmitry Chernyak
322 1 Dmitry Chernyak
h3. INSTANCE INTERFACE AUTOCONFIGURATION
323 1 Dmitry Chernyak
324 1 Dmitry Chernyak
If your instances may sit on several networks and you need static addressing in them, you should fulfill
325 1 Dmitry Chernyak
the file @/etc/ganeti/networks@ with all known networks you want to attach your instances.
326 1 Dmitry Chernyak
Each line in the file has format
327 1 Dmitry Chernyak
328 1 Dmitry Chernyak
|NETWORK|NETMASK|BROADCAST|GATEWAY|
329 1 Dmitry Chernyak
330 1 Dmitry Chernyak
Ganeti instance debootstrap hook looks in this file for the network, mathing the address of bootstraped
331 1 Dmitry Chernyak
instance and fulfill it's @/etc/network/interfaces@ accordingly.
332 1 Dmitry Chernyak
333 1 Dmitry Chernyak
*NOTE*: If you have only one default network, you shouldn't care because it's data are preinstalled.
334 1 Dmitry Chernyak
*NOTE*: networks file must be copied to all cluster nodes (not automated yet).
335 1 Dmitry Chernyak
336 1 Dmitry Chernyak
h2. SCI OPERATIONS
337 1 Dmitry Chernyak
338 1 Dmitry Chernyak
Read [[OPERATIONS]] next.