Проект

Общее

Профиль

SETUP » История » Версия 3

Владимир Ипатов, 23.10.2012 22:26

1 1 Dmitry Chernyak
h1. SETUP
2 1 Dmitry Chernyak
3 1 Dmitry Chernyak
{{toc}}
4 1 Dmitry Chernyak
5 1 Dmitry Chernyak
Ensure both nodes are up.
6 1 Dmitry Chernyak
7 1 Dmitry Chernyak
If you planning to use the secondary network for SAN and DRBD synchronization, you
8 1 Dmitry Chernyak
should configure secondary IP interfaces manually on both nodes at this time.
9 1 Dmitry Chernyak
10 1 Dmitry Chernyak
Log in to the first node via ssh. Due to lack of DNS there may be
11 1 Dmitry Chernyak
a minute timeout before password prompt.
12 1 Dmitry Chernyak
13 2 Владимир Ипатов
h2. NETWORK CONFIGURATION
14 2 Владимир Ипатов
15 2 Владимир Ипатов
Network configuration may be highly various.
16 2 Владимир Ипатов
Here we describe several schemas.
17 2 Владимир Ипатов
18 3 Владимир Ипатов
h.3 Default schema
19 2 Владимир Ипатов
20 3 Владимир Ипатов
one ethernet, one subnet, internet connection provided by external (not in claster) router.
21 3 Владимир Ипатов
By default installer create bridge named xen-br0. You can customize parameters by editing /etc/network/interfaces.
22 3 Владимир Ипатов
By default it looks like:
23 3 Владимир Ипатов
<pre>
24 3 Владимир Ипатов
auto xen-br0
25 3 Владимир Ипатов
iface xen-br0 inet static
26 3 Владимир Ипатов
        address 192.168.5.88
27 3 Владимир Ипатов
        netmask 255.255.255.0
28 3 Владимир Ипатов
        network 192.168.5.0
29 3 Владимир Ипатов
        broadcast 192.168.5.255
30 3 Владимир Ипатов
        gateway 192.168.5.1
31 3 Владимир Ипатов
        bridge_ports eth0
32 3 Владимир Ипатов
        bridge_stp off
33 3 Владимир Ипатов
        bridge_fd 0
34 3 Владимир Ипатов
#       up ifconfig eth0 mtu 9000
35 3 Владимир Ипатов
#       up ifconfig xen-br0 mtu 9000
36 3 Владимир Ипатов
</pre>
37 3 Владимир Ипатов
Important parameters besides ipv4 settings is:
38 3 Владимир Ипатов
bridge_ports eth0 - means that physical interface eth0 enslaved to this bridge.
39 3 Владимир Ипатов
40 3 Владимир Ипатов
up ifconfig eth0 mtu 9000
41 3 Владимир Ипатов
up ifconfig xen-br0 mtu 9000 - setting jumbo frame on bridge for more net speed and less cpu utilization.
42 3 Владимир Ипатов
It will be actual on interface where drbd link will be.
43 3 Владимир Ипатов
However, setting mtu higher than 1500 will cause problems with any network equipment that
44 3 Владимир Ипатов
doesn't support jumbo frames. That's the reason because it option commented out by default.
45 3 Владимир Ипатов
46 3 Владимир Ипатов
Also it is important to specify broadcast and network adresses - it will help automatically
47 3 Владимир Ипатов
fullfill /etc/ganeti/networks file(a file that specify networks for instances).
48 3 Владимир Ипатов
However, it ins't required.
49 3 Владимир Ипатов
50 3 Владимир Ипатов
h3. Mupltiple bridges with routing
51 3 Владимир Ипатов
52 3 Владимир Ипатов
Here is a bit more complicated network setup.
53 3 Владимир Ипатов
In this setup we have, for example, two private netwokrs and wan by ethernet. All routing and firewalling
54 3 Владимир Ипатов
performed by instance in our cluster. This setup fit when you don't have expensive hardware routers and firewalls.
55 3 Владимир Ипатов
This is /etc/network/interfaces file in this setup:
56 3 Владимир Ипатов
<pre>
57 3 Владимир Ипатов
58 3 Владимир Ипатов
auto xen-br0
59 3 Владимир Ипатов
iface xen-br0 inet static
60 3 Владимир Ипатов
        address 192.168.5.88
61 3 Владимир Ипатов
        netmask 255.255.255.0
62 3 Владимир Ипатов
        network 192.168.5.0
63 3 Владимир Ипатов
        broadcast 192.168.5.255
64 3 Владимир Ипатов
        gateway 192.168.5.1
65 3 Владимир Ипатов
        bridge_ports eth0
66 3 Владимир Ипатов
        bridge_stp off
67 3 Владимир Ипатов
        bridge_fd 0
68 3 Владимир Ипатов
#       up ifconfig eth0 mtu 9000
69 3 Владимир Ипатов
#       up ifconfig xen-br0 mtu 9000
70 3 Владимир Ипатов
</pre>
71 2 Владимир Ипатов
72 1 Dmitry Chernyak
h2. DEFINING ENVIRONMENT
73 1 Dmitry Chernyak
74 1 Dmitry Chernyak
Edit @/etc/sci/sci.conf@
75 1 Dmitry Chernyak
76 1 Dmitry Chernyak
* You should specify node1 and node2 data as you have installed them.
77 1 Dmitry Chernyak
*NOTE*: You can setup the cluster even with one node. In this case just leave NODE2_
78 1 Dmitry Chernyak
lines as is. In fact this is a dangerous setup, so you will be warned about this duging
79 1 Dmitry Chernyak
the procedures.
80 1 Dmitry Chernyak
81 1 Dmitry Chernyak
* You should specify the cluster's name and IP.
82 1 Dmitry Chernyak
83 1 Dmitry Chernyak
* NODE#_SAN_IP should be specified on both nodes or none.
84 1 Dmitry Chernyak
85 1 Dmitry Chernyak
* If you haven't Internet uplink or have a local package mirrors, you should correct
86 1 Dmitry Chernyak
APT_ - settings.
87 1 Dmitry Chernyak
88 1 Dmitry Chernyak
* If you need to uplink to the DNS hierarchy other than root hint zones, specify DNS_FORWARDERS
89 1 Dmitry Chernyak
(note trailing ';').
90 1 Dmitry Chernyak
91 1 Dmitry Chernyak
h2. SETUP CLUSTER
92 1 Dmitry Chernyak
93 1 Dmitry Chernyak
Issue:
94 1 Dmitry Chernyak
95 1 Dmitry Chernyak
<pre>
96 1 Dmitry Chernyak
# sci-setup cluster
97 1 Dmitry Chernyak
</pre>
98 1 Dmitry Chernyak
99 1 Dmitry Chernyak
Check and confirm settings printed.
100 1 Dmitry Chernyak
101 1 Dmitry Chernyak
The process will go on.
102 1 Dmitry Chernyak
103 1 Dmitry Chernyak
Next you will be prompted to accept ssh key from node2 and for the root's password to node2.
104 1 Dmitry Chernyak
105 1 Dmitry Chernyak
On finish you will look something like this:
106 1 Dmitry Chernyak
107 1 Dmitry Chernyak
<pre>
108 1 Dmitry Chernyak
Verify
109 1 Dmitry Chernyak
Wed Jan 12 15:36:10 2011 * Verifying global settings
110 1 Dmitry Chernyak
Wed Jan 12 15:36:10 2011 * Gathering data (1 nodes)
111 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying node status
112 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying instance status
113 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying orphan volumes
114 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying orphan instances
115 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Verifying N+1 Memory redundancy
116 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Other Notes
117 1 Dmitry Chernyak
Wed Jan 12 15:36:11 2011 * Hooks Results
118 1 Dmitry Chernyak
Node                    DTotal  DFree MTotal MNode MFree Pinst Sinst
119 1 Dmitry Chernyak
gnt1.ganeti.example.org 100.0G 100.0G  1020M  379M  625M     0     0
120 1 Dmitry Chernyak
gnt2.ganeti.example.org 100.0G 100.0G  1020M  379M  625M     0     0
121 1 Dmitry Chernyak
If all is ok, proceed with /usr/local/sbin/sci-setup service
122 1 Dmitry Chernyak
</pre>
123 1 Dmitry Chernyak
124 1 Dmitry Chernyak
h2. SETUP SERVICE INSTANCE
125 1 Dmitry Chernyak
126 1 Dmitry Chernyak
The service instance is named 'sci' and have a few aliases.
127 1 Dmitry Chernyak
On setup, it's IP address is determined from @/etc/resolv.conf@ of your first node.
128 1 Dmitry Chernyak
This instance will be hardcoded in @/etc/hosts@ file of all cluster nodes and instances.
129 1 Dmitry Chernyak
130 1 Dmitry Chernyak
Issue:
131 1 Dmitry Chernyak
132 1 Dmitry Chernyak
<pre>
133 1 Dmitry Chernyak
# sci-setup service
134 1 Dmitry Chernyak
</pre>
135 1 Dmitry Chernyak
136 1 Dmitry Chernyak
You'll see the progress of DRBD syncing disks, then the message
137 1 Dmitry Chernyak
<pre>
138 1 Dmitry Chernyak
* running the instance OS create scripts...
139 1 Dmitry Chernyak
</pre>
140 1 Dmitry Chernyak
appears. The further may take a while. The process finishes with
141 1 Dmitry Chernyak
<pre>
142 1 Dmitry Chernyak
* starting instance...
143 1 Dmitry Chernyak
</pre>
144 1 Dmitry Chernyak
message.
145 1 Dmitry Chernyak
146 1 Dmitry Chernyak
Now you can log on to the sci instance using:
147 1 Dmitry Chernyak
148 1 Dmitry Chernyak
<pre>
149 1 Dmitry Chernyak
# gnt-instance console sci
150 1 Dmitry Chernyak
</pre>
151 1 Dmitry Chernyak
152 1 Dmitry Chernyak
Log in as root, the password is empty.
153 1 Dmitry Chernyak
*NOTE*: Due to empty password all remote connections to new instance is prohibited.
154 1 Dmitry Chernyak
You should change password and install @openssh-server@ package manually after
155 1 Dmitry Chernyak
successful bootstrap procedure.
156 1 Dmitry Chernyak
157 1 Dmitry Chernyak
h2. SERVICE INSTANCE BOOTSTRAP
158 1 Dmitry Chernyak
159 1 Dmitry Chernyak
The system will setup itself via puppet. This is the iterative process. You can monitor
160 1 Dmitry Chernyak
it by looking into @/var/log/daemon.log@. At start there is no @less@ command yet, so
161 1 Dmitry Chernyak
you can use @more@, @cat@, @tail@ or @tail -f@ until @less@ will be auto-installed.
162 1 Dmitry Chernyak
163 1 Dmitry Chernyak
By default the iterations are repeated in 20 minutes. To shorten the wait time you can
164 1 Dmitry Chernyak
issue
165 1 Dmitry Chernyak
166 1 Dmitry Chernyak
<pre>
167 1 Dmitry Chernyak
# /etc/init.d/puppet restart
168 1 Dmitry Chernyak
</pre>
169 1 Dmitry Chernyak
170 1 Dmitry Chernyak
and then look into @daemon.log@ how it finishes.
171 1 Dmitry Chernyak
172 1 Dmitry Chernyak
Repeat this a few times until puppet will do nothing in turn.
173 1 Dmitry Chernyak
174 1 Dmitry Chernyak
h2. PREPARING FOR NEW INSTANCES
175 1 Dmitry Chernyak
176 1 Dmitry Chernyak
New instances are created just by regular Ganeti commands such as:
177 1 Dmitry Chernyak
178 1 Dmitry Chernyak
<pre>
179 1 Dmitry Chernyak
gnt-instance add -t drbd -o debootstrap+default -s 10g -B memory=256m -n NODE1_NAME:NODE2_NAME INSTANCE_NAME
180 1 Dmitry Chernyak
</pre>
181 1 Dmitry Chernyak
182 1 Dmitry Chernyak
Altought, some tuning hooks are provided by SCI-CD project:
183 1 Dmitry Chernyak
# Each instance has installed @puppet@ for autoconfiguration and @openssh-client@ for file transfers etc.
184 1 Dmitry Chernyak
# The instance uses pygrub to boot kernel from /vmlinuz & Co on the innstance's own disk.
185 1 Dmitry Chernyak
# The instance's network interfaces may be set up automatically as described below.
186 1 Dmitry Chernyak
187 1 Dmitry Chernyak
h3. INSTANCE INTERFACE AUTOCONFIGURATION
188 1 Dmitry Chernyak
189 1 Dmitry Chernyak
If your instances may sit on several networks and you need static addressing in them, you should fulfill
190 1 Dmitry Chernyak
the file @/etc/ganeti/networks@ with all known networks you want to attach your instances.
191 1 Dmitry Chernyak
Each line in the file has format
192 1 Dmitry Chernyak
193 1 Dmitry Chernyak
|NETWORK|NETMASK|BROADCAST|GATEWAY|
194 1 Dmitry Chernyak
195 1 Dmitry Chernyak
Ganeti instance debootstrap hook looks in this file for the network, mathing the address of bootstraped
196 1 Dmitry Chernyak
instance and fulfill it's @/etc/network/interfaces@ accordingly.
197 1 Dmitry Chernyak
198 1 Dmitry Chernyak
*NOTE*: If you have only one default network, you shouldn't care because it's data are preinstalled.
199 1 Dmitry Chernyak
*NOTE*: networks file must be copied to all cluster nodes (not automated yet).
200 1 Dmitry Chernyak
201 1 Dmitry Chernyak
h2. SCI OPERATIONS
202 1 Dmitry Chernyak
203 1 Dmitry Chernyak
Read [[OPERATIONS]] next.