Проект

Общее

Профиль

SETUP » История » Версия 18

Dmitry Chernyak, 22.02.2013 01:30

1 1 Dmitry Chernyak
h1. SETUP
2 1 Dmitry Chernyak
3 1 Dmitry Chernyak
{{toc}}
4 1 Dmitry Chernyak
5 18 Dmitry Chernyak
[[OVERVIEW]] | [[INSTALL]] | [[BUILD-ISO]] | [[SETUP]] | [[OPERATIONS]] | [[GITMAGIC]] | [[LICENSE]] | [[STATUS]]
6 16 Dmitry Chernyak
на Русском: [[ОБЗОР]] | [[СБОРКА-ISO]] | [[УСТАНОВКА]] | [[НАСТРОЙКА]] | [[ОПЕРАЦИИ]] | [[МАГИЯ GIT]] | [[ЛИЦЕНЗИЯ]] | [[СОСТОЯНИЕ]]
7 16 Dmitry Chernyak
8 1 Dmitry Chernyak
Ensure both nodes are up.
9 1 Dmitry Chernyak
10 1 Dmitry Chernyak
If you planning to use the secondary network for SAN and DRBD synchronization, you
11 1 Dmitry Chernyak
should configure secondary IP interfaces manually on both nodes at this time.
12 1 Dmitry Chernyak
13 11 Dmitry Chernyak
Log in to the first node via ssh. Due to lack of DNS there may be a minute timeout
14 11 Dmitry Chernyak
before the server anwers yoy with the password prompt.
15 1 Dmitry Chernyak
16 2 Владимир Ипатов
h2. NETWORK CONFIGURATION
17 11 Dmitry Chernyak
18 1 Dmitry Chernyak
Network configuration may be highly various.
19 16 Dmitry Chernyak
Here we describe the typical scheme with two interfaces , one for interlink(ganeti interoperation+drbd link) and one for LAN.
20 3 Владимир Ипатов
21 16 Dmitry Chernyak
This schema suits most cases. It doesn't require a gigabit switch, provide good performance and reliability.
22 1 Dmitry Chernyak
Two gigabit network interfaces on the nodes are connected directly or via a gigabit
23 1 Dmitry Chernyak
switch (if you want more than two nodes in the cluster).
24 16 Dmitry Chernyak
Other interfaces connected to LAN.
25 16 Dmitry Chernyak
Any LAN failure doesn't affect cluster in this setup.
26 15 Владимир Ипатов
This is /etc/network/interfaces file for this setup:
27 14 Владимир Ипатов
<pre>auto xen-br0
28 7 Владимир Ипатов
iface xen-br0 inet static
29 8 Владимир Ипатов
	address 192.168.236.1
30 8 Владимир Ипатов
	netmask 255.255.255.0
31 8 Владимир Ипатов
	network 192.168.236.0
32 8 Владимир Ипатов
	broadcast 192.168.236.255
33 8 Владимир Ипатов
        bridge_ports eth0
34 8 Владимир Ипатов
        bridge_stp off
35 8 Владимир Ипатов
        bridge_fd 0
36 8 Владимир Ипатов
#	up ifconfig eth0 mtu 9000
37 1 Dmitry Chernyak
#	up ifconfig xen-br0 mtu 9000
38 1 Dmitry Chernyak
39 1 Dmitry Chernyak
auto xen-lan
40 1 Dmitry Chernyak
iface xen-lan inet static
41 8 Владимир Ипатов
	address 192.168.5.55
42 7 Владимир Ипатов
	netmask 255.255.255.0
43 8 Владимир Ипатов
	network 192.168.5.0
44 8 Владимир Ипатов
	broadcast 192.168.5.255
45 8 Владимир Ипатов
	gateway 192.168.5.1
46 1 Dmitry Chernyak
	bridge_ports eth1
47 12 Владимир Ипатов
	bridge_stp off
48 8 Владимир Ипатов
	bridge_fd 0
49 8 Владимир Ипатов
</pre>
50 8 Владимир Ипатов
51 8 Владимир Ипатов
xen-br0 used by ganeti interoperation and drbd link, it was configured by the installer.
52 11 Dmitry Chernyak
Also the dns server and the gateway was configured by the installer - it will be our service instance(sci) address.
53 16 Dmitry Chernyak
xen-lan used by lan connection, its configuration must be added by hands (the 'interfaces' contains a template).
54 16 Dmitry Chernyak
55 11 Dmitry Chernyak
In this network configuration you must fill these variables in sci.conf:
56 11 Dmitry Chernyak
NODE1_IP - already configured by installer.
57 9 Владимир Ипатов
NODE1_NAME - already configured by installer.
58 3 Владимир Ипатов
NODE2_IP - set interlink ip address of second node. e.g. 192.168.236.2
59 5 Владимир Ипатов
NODE2_NAME - set second node name. e.g. gnt2
60 5 Владимир Ипатов
NODE1_LAN_IP - lan ip for first node. It will be available by dns name $NODE1_NAME-lan. 192.168.5.55
61 5 Владимир Ипатов
NODE2_LAN_IP - lan ip for second node. It will be available by dns name $NODE2_NAME-lan. e.g. 192.168.5.58
62 5 Владимир Ипатов
CLUSTER_IP - cluster address in lan. Must not match any exist host address in lan. 192.168.5.35
63 5 Владимир Ипатов
CLUSTER_NAME - cluster name in lan. In will be available by dns name $CLUSTER_NAME.
64 5 Владимир Ипатов
SCI_LAN_IP - if you want presence sci intance in your lan, assign ip. e.g. 192.168.5.59
65 5 Владимир Ипатов
66 16 Dmitry Chernyak
On the [[Network setup]] page you may look and pick the other schemes for different cases.
67 2 Владимир Ипатов
68 1 Dmitry Chernyak
69 1 Dmitry Chernyak
h2. DEFINING ENVIRONMENT
70 1 Dmitry Chernyak
71 1 Dmitry Chernyak
Edit @/etc/sci/sci.conf@
72 1 Dmitry Chernyak
73 1 Dmitry Chernyak
Most of values rely of your network setup. In section network setup it was described for most cases.
74 8 Владимир Ипатов
75 8 Владимир Ипатов
Here is additional notes about sci.conf configuring:
76 8 Владимир Ипатов
77 8 Владимир Ипатов
* You should specify node1 and node2 data as you have installed them.
78 1 Dmitry Chernyak
*NOTE*: You can setup the cluster even with one node. In this case just leave NODE2_
79 1 Dmitry Chernyak
lines as is. In fact this is a dangerous setup, so you will be warned about this duging
80 1 Dmitry Chernyak
the procedures.
81 1 Dmitry Chernyak
82 1 Dmitry Chernyak
* You should specify the cluster's name and IP.
83 1 Dmitry Chernyak
84 1 Dmitry Chernyak
* NODE#_SAN_IP should be specified on both nodes or none.
85 1 Dmitry Chernyak
86 1 Dmitry Chernyak
* NODE#_LAN_IP should be specified on both nodes or none.
87 8 Владимир Ипатов
88 1 Dmitry Chernyak
* If you haven't Internet uplink or have a local package mirrors, you should correct
89 1 Dmitry Chernyak
APT_ - settings.
90 1 Dmitry Chernyak
91 6 Владимир Ипатов
* If you need to uplink to the DNS hierarchy other than root hint zones, specify DNS_FORWARDERS
92 6 Владимир Ипатов
(note trailing ';').
93 1 Dmitry Chernyak
94 8 Владимир Ипатов
* MASTER_NETDEV - master interface name for cluster address. Auto-detected by default.
95 6 Владимир Ипатов
96 8 Владимир Ипатов
* LAN_NETDEV - Network interface to bind to virtual machies by default. Auto-detected by default.
97 6 Владимир Ипатов
98 16 Dmitry Chernyak
* RESERVED_VOLS - list of volumes ignored by ganeti. Comma separated. You should specify vg for all volumes in this list. It is preset with reasonable default, needed for SCI-CD.
99 6 Владимир Ипатов
100 1 Dmitry Chernyak
101 17 Dmitry Chernyak
h2. SETUP THE CLUSTER
102 1 Dmitry Chernyak
103 1 Dmitry Chernyak
Issue:
104 1 Dmitry Chernyak
105 1 Dmitry Chernyak
<pre>
106 1 Dmitry Chernyak
# sci-setup cluster
107 1 Dmitry Chernyak
</pre>
108 1 Dmitry Chernyak
109 1 Dmitry Chernyak
Check and confirm settings printed.
110 1 Dmitry Chernyak
111 1 Dmitry Chernyak
The process will go on.
112 1 Dmitry Chernyak
113 1 Dmitry Chernyak
Next you will be prompted to accept ssh key from node2 and for the root's password to node2.
114 1 Dmitry Chernyak
115 1 Dmitry Chernyak
On finish you will look something like this:
116 1 Dmitry Chernyak
117 1 Dmitry Chernyak
<pre>
118 1 Dmitry Chernyak
Verify
119 1 Dmitry Chernyak
Wed Jan 12 15:36:10 2011 * Verifying global settings
120 1 Dmitry Chernyak
Wed Jan 12 15:36:10 2011 * Gathering data (1 nodes)
121 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying node status
122 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying instance status
123 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying orphan volumes
124 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying orphan instances
125 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying N+1 Memory redundancy
126 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Other Notes
127 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Hooks Results
128 1 Dmitry Chernyak
Node                    DTotal  DFree MTotal MNode MFree Pinst Sinst
129 1 Dmitry Chernyak
gnt1.ganeti.example.org 100.0G 100.0G  1020M  379M  625M     0     0
130 1 Dmitry Chernyak
gnt2.ganeti.example.org 100.0G 100.0G  1020M  379M  625M     0     0
131 1 Dmitry Chernyak
If all is ok, proceed with /usr/local/sbin/sci-setup service
132 1 Dmitry Chernyak
</pre>
133 1 Dmitry Chernyak
134 17 Dmitry Chernyak
h2. SETUP THE SERVICE INSTANCE
135 1 Dmitry Chernyak
136 1 Dmitry Chernyak
The service instance is named 'sci' and have a few aliases.
137 1 Dmitry Chernyak
On setup, it's IP address is determined from @/etc/resolv.conf@ of your first node.
138 1 Dmitry Chernyak
This instance will be hardcoded in @/etc/hosts@ file of all cluster nodes and instances.
139 1 Dmitry Chernyak
140 1 Dmitry Chernyak
Issue:
141 1 Dmitry Chernyak
142 1 Dmitry Chernyak
<pre>
143 1 Dmitry Chernyak
# sci-setup service
144 1 Dmitry Chernyak
</pre>
145 1 Dmitry Chernyak
146 1 Dmitry Chernyak
You'll see the progress of DRBD syncing disks, then the message
147 1 Dmitry Chernyak
<pre>
148 1 Dmitry Chernyak
* running the instance OS create scripts...
149 1 Dmitry Chernyak
</pre>
150 1 Dmitry Chernyak
appears. The further may take a while. The process finishes with
151 1 Dmitry Chernyak
<pre>
152 1 Dmitry Chernyak
* starting instance...
153 1 Dmitry Chernyak
</pre>
154 1 Dmitry Chernyak
message.
155 1 Dmitry Chernyak
156 1 Dmitry Chernyak
Now you can log on to the sci instance using:
157 1 Dmitry Chernyak
158 1 Dmitry Chernyak
<pre>
159 1 Dmitry Chernyak
# gnt-instance console sci
160 1 Dmitry Chernyak
</pre>
161 1 Dmitry Chernyak
162 1 Dmitry Chernyak
Log in as root, the password is empty.
163 1 Dmitry Chernyak
*NOTE*: Due to empty password all remote connections to new instance is prohibited.
164 1 Dmitry Chernyak
You should change password and install @openssh-server@ package manually after
165 1 Dmitry Chernyak
successful bootstrap procedure.
166 1 Dmitry Chernyak
167 1 Dmitry Chernyak
h2. SERVICE INSTANCE BOOTSTRAP
168 1 Dmitry Chernyak
169 1 Dmitry Chernyak
The system will setup itself via puppet. This is the iterative process. You can monitor
170 1 Dmitry Chernyak
it by looking into @/var/log/daemon.log@. At start there is no @less@ command yet, so
171 1 Dmitry Chernyak
you can use @more@, @cat@, @tail@ or @tail -f@ until @less@ will be auto-installed.
172 1 Dmitry Chernyak
173 1 Dmitry Chernyak
By default the iterations are repeated in 20 minutes. To shorten the wait time you can
174 1 Dmitry Chernyak
issue
175 1 Dmitry Chernyak
176 1 Dmitry Chernyak
<pre>
177 1 Dmitry Chernyak
# /etc/init.d/puppet restart
178 1 Dmitry Chernyak
</pre>
179 1 Dmitry Chernyak
180 1 Dmitry Chernyak
and then look into @daemon.log@ how it finishes.
181 1 Dmitry Chernyak
182 1 Dmitry Chernyak
Repeat this a few times until puppet will do nothing in turn.
183 1 Dmitry Chernyak
184 1 Dmitry Chernyak
h2. PREPARING FOR NEW INSTANCES
185 1 Dmitry Chernyak
186 1 Dmitry Chernyak
New instances are created just by regular Ganeti commands such as:
187 1 Dmitry Chernyak
188 1 Dmitry Chernyak
<pre>
189 1 Dmitry Chernyak
gnt-instance add -t drbd -o debootstrap+default -s 10g -B memory=256m -n NODE1_NAME:NODE2_NAME INSTANCE_NAME
190 1 Dmitry Chernyak
</pre>
191 1 Dmitry Chernyak
192 1 Dmitry Chernyak
Altought, some tuning hooks are provided by SCI-CD project:
193 1 Dmitry Chernyak
# Each instance has installed @puppet@ for autoconfiguration and @openssh-client@ for file transfers etc.
194 1 Dmitry Chernyak
# The instance uses pygrub to boot kernel from /vmlinuz & Co on the innstance's own disk.
195 1 Dmitry Chernyak
# The instance's network interfaces may be set up automatically as described below.
196 1 Dmitry Chernyak
197 1 Dmitry Chernyak
h3. INSTANCE INTERFACE AUTOCONFIGURATION
198 1 Dmitry Chernyak
199 1 Dmitry Chernyak
If your instances may sit on several networks and you need static addressing in them, you should fulfill
200 1 Dmitry Chernyak
the file @/etc/ganeti/networks@ with all known networks you want to attach your instances.
201 1 Dmitry Chernyak
Each line in the file has format
202 1 Dmitry Chernyak
203 1 Dmitry Chernyak
|NETWORK|NETMASK|BROADCAST|GATEWAY|
204 1 Dmitry Chernyak
205 1 Dmitry Chernyak
Ganeti instance debootstrap hook looks in this file for the network, mathing the address of bootstraped
206 1 Dmitry Chernyak
instance and fulfill it's @/etc/network/interfaces@ accordingly.
207 1 Dmitry Chernyak
208 1 Dmitry Chernyak
*NOTE*: If you have only one default network, you shouldn't care because it's data are preinstalled.
209 1 Dmitry Chernyak
*NOTE*: networks file must be copied to all cluster nodes (not automated yet).
210 1 Dmitry Chernyak
211 1 Dmitry Chernyak
h2. SCI OPERATIONS
212 1 Dmitry Chernyak
213 1 Dmitry Chernyak
Read [[OPERATIONS]] next.