Fast Deploy Nebula Graph Cluster With Docker Swarm
This post describes in detail how to deploy a Nebula Graph cluster with Docker Swarm.
by
·
Join the DZone community and get the full member experience.
Join For FreeThis post describes in detail how to deploy a Nebula Graph cluster with Docker Swarm.
Deploy a Nebula Graph Cluster
2.1 Environment Preparation
Prepare hosts as below.
| ip | Memory (GB) | CPU (# of Cores) |
|---|---|---|
| 192.168.1.166 | 16 | 4 |
| 192.168.1.167 | 16 | 4 |
| 192.168.1.168 | 16 | 4 |
Please make sure that Docker has been installed on all the machines.
2.2 Initialize the Swarm Cluster
Execute the commands below on the host 192.168.1.166:
YAML
x
1
$ docker swarm init --advertise-addr 192.168.1.166
2
Swarm initializedcurrent node (dxn1zf6l61qsb1josjja83ngz) is now a manager.
3
To add a worker to this swarm, run the following command
4
docker swarm join \
5
--token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
6
192.168.1.166:2377
7
8
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
2.3 Add a Worker Node
Add a Swarm worker node per the notification message of the init commands. Execute the following commands on 192.168.1.167 and 192.168.1.168 respectively.
YAML
xxxxxxxxxx
1
1
docker swarm join \
2
--token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \
3
192.168.1.166:2377
2.4 Authenticate the Cluster
YAML
xxxxxxxxxx
1
1
docker node ls
2
3
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
4
h0az2wzqetpwhl9ybu76yxaen * KF2-DATA-166 Ready Active Reachable 18.06.1-ce
5
q6jripaolxsl7xqv3cmv5pxji KF2-DATA-167 Ready Active Leader 18.06.1-ce
6
h1iql1uvm7123h3gon9so69dy KF2-DATA-168 Ready Active 18.06.1-ce
2.5 Configure Docker Stack
YAML
xxxxxxxxxx
1
1
vi docker-stack.yml
Configure Docker Stack as below:
YAML
xxxxxxxxxx
1
427
1
version'3.6'
2
services
3
metad0
4
imagevesoft/nebula-metadnightly
5
env_file
6
./nebula.env
7
command
8
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
9
--local_ip=192.168.1.166
10
--ws_ip=192.168.1.166
11
--port=45500
12
--data_path=/data/meta
13
--log_dir=/logs
14
--v=0
15
--minloglevel=2
16
deploy
17
replicas1
18
restart_policy
19
conditionon-failure
20
placement
21
constraints
22
node.hostname == KF2-DATA-166
23
healthcheck
24
test"CMD" "curl" "-f" "http://192.168.1.166:11000/status"
25
interval30s
26
timeout10s
27
retries3
28
start_period20s
29
ports
30
target11000
31
published11000
32
protocoltcp
33
modehost
34
target11002
35
published11002
36
protocoltcp
37
modehost
38
target45500
39
published45500
40
protocoltcp
41
modehost
42
volumes
43
data-metad0:/data/meta
44
logs-metad0:/logs
45
networks
46
nebula-net
47
48
metad1
49
imagevesoft/nebula-metadnightly
50
env_file
51
./nebula.env
52
command
53
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
54
--local_ip=192.168.1.167
55
--ws_ip=192.168.1.167
56
--port=45500
57
--data_path=/data/meta
58
--log_dir=/logs
59
--v=0
60
--minloglevel=2
61
deploy
62
replicas1
63
restart_policy
64
conditionon-failure
65
placement
66
constraints
67
node.hostname == KF2-DATA-167
68
healthcheck
69
test"CMD" "curl" "-f" "http://192.168.1.167:11000/status"
70
interval30s
71
timeout10s
72
retries3
73
start_period20s
74
ports
75
target11000
76
published11000
77
protocoltcp
78
modehost
79
target11002
80
published11002
81
protocoltcp
82
modehost
83
target45500
84
published45500
85
protocoltcp
86
modehost
87
volumes
88
data-metad1:/data/meta
89
logs-metad1:/logs
90
networks
91
nebula-net
92
93
metad2
94
imagevesoft/nebula-metadnightly
95
env_file
96
./nebula.env
97
command
98
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
99
--local_ip=192.168.1.168
100
--ws_ip=192.168.1.168
101
--port=45500
102
--data_path=/data/meta
103
--log_dir=/logs
104
--v=0
105
--minloglevel=2
106
deploy
107
replicas1
108
restart_policy
109
conditionon-failure
110
placement
111
constraints
112
node.hostname == KF2-DATA-168
113
healthcheck
114
test"CMD" "curl" "-f" "http://192.168.1.168:11000/status"
115
interval30s
116
timeout10s
117
retries3
118
start_period20s
119
ports
120
target11000
121
published11000
122
protocoltcp
123
modehost
124
target11002
125
published11002
126
protocoltcp
127
modehost
128
target45500
129
published45500
130
protocoltcp
131
modehost
132
volumes
133
data-metad2:/data/meta
134
logs-metad2:/logs
135
networks
136
nebula-net
137
138
storaged0
139
imagevesoft/nebula-storagednightly
140
env_file
141
./nebula.env
142
command
143
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
144
--local_ip=192.168.1.166
145
--ws_ip=192.168.1.166
146
--port=44500
147
--data_path=/data/storage
148
--log_dir=/logs
149
--v=0
150
--minloglevel=2
151
deploy
152
replicas1
153
restart_policy
154
conditionon-failure
155
placement
156
constraints
157
node.hostname == KF2-DATA-166
158
depends_on
159
metad0
160
metad1
161
metad2
162
healthcheck
163
test"CMD" "curl" "-f" "http://192.168.1.166:12000/status"
164
interval30s
165
timeout10s
166
retries3
167
start_period20s
168
ports
169
target12000
170
published12000
171
protocoltcp
172
modehost
173
target12002
174
published12002
175
protocoltcp
176
modehost
177
volumes
178
data-storaged0:/data/storage
179
logs-storaged0:/logs
180
networks
181
nebula-net
182
storaged1
183
imagevesoft/nebula-storagednightly
184
env_file
185
./nebula.env
186
command
187
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
188
--local_ip=192.168.1.167
189
--ws_ip=192.168.1.167
190
--port=44500
191
--data_path=/data/storage
192
--log_dir=/logs
193
--v=0
194
--minloglevel=2
195
deploy
196
replicas1
197
restart_policy
198
conditionon-failure
199
placement
200
constraints
201
node.hostname == KF2-DATA-167
202
depends_on
203
metad0
204
metad1
205
metad2
206
healthcheck
207
test"CMD" "curl" "-f" "http://192.168.1.167:12000/status"
208
interval30s
209
timeout10s
210
retries3
211
start_period20s
212
ports
213
target12000
214
published12000
215
protocoltcp
216
modehost
217
target12002
218
published12004
219
protocoltcp
220
modehost
221
volumes
222
data-storaged1:/data/storage
223
logs-storaged1:/logs
224
networks
225
nebula-net
226
227
storaged2
228
imagevesoft/nebula-storagednightly
229
env_file
230
./nebula.env
231
command
232
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
233
--local_ip=192.168.1.168
234
--ws_ip=192.168.1.168
235
--port=44500
236
--data_path=/data/storage
237
--log_dir=/logs
238
--v=0
239
--minloglevel=2
240
deploy
241
replicas1
242
restart_policy
243
conditionon-failure
244
placement
245
constraints
246
node.hostname == KF2-DATA-168
247
depends_on
248
metad0
249
metad1
250
metad2
251
healthcheck
252
test"CMD" "curl" "-f" "http://192.168.1.168:12000/status"
253
interval30s
254
timeout10s
255
retries3
256
start_period20s
257
ports
258
target12000
259
published12000
260
protocoltcp
261
modehost
262
target12002
263
published12006
264
protocoltcp
265
modehost
266
volumes
267
data-storaged2:/data/storage
268
logs-storaged2:/logs
269
networks
270
nebula-net
271
graphd1
272
imagevesoft/nebula-graphdnightly
273
env_file
274
./nebula.env
275
command
276
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
277
--port=3699
278
--ws_ip=192.168.1.166
279
--log_dir=/logs
280
--v=0
281
--minloglevel=2
282
deploy
283
replicas1
284
restart_policy
285
conditionon-failure
286
placement
287
constraints
288
node.hostname == KF2-DATA-166
289
depends_on
290
metad0
291
metad1
292
metad2
293
healthcheck
294
test"CMD" "curl" "-f" "http://192.168.1.166:13000/status"
295
interval30s
296
timeout10s
297
retries3
298
start_period20s
299
ports
300
target3699
301
published3699
302
protocoltcp
303
modehost
304
target13000
305
published13000
306
protocoltcp
307
# mode: host
308
target13002
309
published13002
310
protocoltcp
311
modehost
312
volumes
313
logs-graphd:/logs
314
networks
315
nebula-net
316
317
graphd2
318
imagevesoft/nebula-graphdnightly
319
env_file
320
./nebula.env
321
command
322
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
323
--port=3699
324
--ws_ip=192.168.1.167
325
--log_dir=/logs
326
--v=2
327
--minloglevel=2
328
deploy
329
replicas1
330
restart_policy
331
conditionon-failure
332
placement
333
constraints
334
node.hostname == KF2-DATA-167
335
depends_on
336
metad0
337
metad1
338
metad2
339
healthcheck
340
test"CMD" "curl" "-f" "http://192.168.1.167:13001/status"
341
interval30s
342
timeout10s
343
retries3
344
start_period20s
345
ports
346
target3699
347
published3640
348
protocoltcp
349
modehost
350
target13000
351
published13001
352
protocoltcp
353
modehost
354
target13002
355
published13003
356
protocoltcp
357
# mode: host
358
volumes
359
logs-graphd2:/logs
360
networks
361
nebula-net
362
graphd3
363
imagevesoft/nebula-graphdnightly
364
env_file
365
./nebula.env
366
command
367
--meta_server_addrs=192.168.1.166:45500,192.168.1.167:45500,192.168.1.168:45500
368
--port=3699
369
--ws_ip=192.168.1.168
370
--log_dir=/logs
371
--v=0
372
--minloglevel=2
373
deploy
374
replicas1
375
restart_policy
376
conditionon-failure
377
placement
378
constraints
379
node.hostname == KF2-DATA-168
380
depends_on
381
metad0
382
metad1
383
metad2
384
healthcheck
385
test"CMD" "curl" "-f" "http://192.168.1.168:13002/status"
386
interval30s
387
timeout10s
388
retries3
389
start_period20s
390
ports
391
target3699
392
published3641
393
protocoltcp
394
modehost
395
target13000
396
published13002
397
protocoltcp
398
# mode: host
399
target13002
400
published13004
401
protocoltcp
402
modehost
403
volumes
404
logs-graphd3:/logs
405
networks
406
nebula-net
407
networks
408
nebula-net
409
externaltrue
410
attachabletrue
411
namehost
412
volumes
413
data-metad0
414
logs-metad0
415
data-metad1
416
logs-metad1
417
data-metad2
418
logs-metad2
419
data-storaged0
420
logs-storaged0
421
data-storaged1
422
logs-storaged1
423
data-storaged2
424
logs-storaged2
425
logs-graphd
426
logs-graphd2
427
logs-graphd3
Edit the nebula.env file by adding the items below:
YAML
xxxxxxxxxx
1
1
TZ=UTC
2
USER=root
2.6 Start the Nebula Graph Cluster
YAML
xxxxxxxxxx
1
1
docker stack deploy nebula -c docker-stack.yml
Cluster Configuration for Load Balancing and High Availability
The Nebula Graph clients (1.X) don’t provide load balancing capability currently. They randomly select any graphd to connect the database. Therefore, you need to configure load balancing and high availability on your own if you want to use Nebula Graph in production.
Seen from the figure above, the entire deployment can be divided into three layers, i.e. data layer, load balance layer, and high availability layer.
The load balance layer is responsible for distributing the requests from the client to the data layer for load balancing purpose.
The HA layer is realized by HAProxy and it ensures that the load balancing service works properly so that the entire cluster works properly.
3.1 Load Balancing Configuration
HAProxy uses Docker compose for configuration. Edit the three files below respectively:
1. Dockerfile
YAML
xxxxxxxxxx
1
1
FROM haproxy:1.7
2
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
3
EXPOSE 3640
2. docker-compose.yml
YAML
xxxxxxxxxx
1
16
1
version"3.2"
2
services
3
haproxy
4
container_namehaproxy
5
build.
6
volumes
7
./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
8
ports
9
3640:3640
10
restartalways
11
networks
12
app_net
13
networks
14
app_net
15
externaltrue
16
3. haproxy.cfg
YAML
xxxxxxxxxx
1
32
1
global
2
daemon
3
maxconn 30000
4
log 127.0.0.1 local0 info
5
log 127.0.0.1 local1 warning
6
7
defaults
8
log-format %hr\ %ST\ %B\ %Ts
9
log global
10
mode http
11
option http-keep-alive
12
timeout connect 5000ms
13
timeout client 10000ms
14
timeout server 50000ms
15
timeout http-request 20000ms
16
17
# Customize your own frontends && backends && listen conf
18
#CUSTOM
19
20
listen graphd-cluster
21
bind *:3640
22
mode tcp
23
maxconn 300
24
balance roundrobin
25
server server1 192.168.1.166:3699 maxconn 300 check
26
server server2 192.168.1.167:3699 maxconn 300 check
27
server server3 192.168.1.168:3699 maxconn 300 check
28
29
listen stats
30
bind *:1080
31
stats refresh 30s
32
stats uri /stats
3.2 Start HAProxy
YAML
xxxxxxxxxx
1
1
docker-compose up -d
3.3 HA Configuration
Follow the configuration steps on 192.168.1.166, 192.168.1.167, and 192.168.1.168.
1. Install Keepalived
YAML
xxxxxxxxxx
1
1
apt-get update && apt-get upgrade && apt-get install keepalived -y
2. Modify Keepalived’s configuration file, i.e. /etc/keepalived/keepalived.conf. Note that the priority item should be set to different values for the three hosts so that there is a priority among them.
Note: Please be noted that vip (virtual IP) is required to configure Keepalive. In the configurations below, 192.168.1.99 is the virtual IP.
1) Configuration on 192.168.1.166
YAML
xxxxxxxxxx
1
28
1
global_defs
2
router_id lb01 #An identifier；
3
4
vrrp_script chk_haproxy
5
script "killall -0 haproxy" interval 2
6
7
vrrp_instance VI_1
8
state MASTER
9
interface ens160
10
virtual_router_id 52
11
priority 999
12
# Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
13
advert_int 1
14
# Configure the authentication type and password
15
authentication
16
# Configure the authentication type, mainly PASS and AH
17
auth_type PASS
18
# Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
19
auth_pass amber1
20
21
virtual_ipaddress
22
# The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
23
192.168.1.99/24 dev ens160 label ens160:1
24
25
track_script
26
chk_haproxy
27
28
2) Configuration on 192.168.1.167
YAML
xxxxxxxxxx
1
28
1
global_defs
2
router_id lb01 #An identifier；
3
4
vrrp_script chk_haproxy
5
script "killall -0 haproxy" interval 2
6
7
vrrp_instance VI_1
8
state BACKUP
9
interface ens160
10
virtual_router_id 52
11
priority 888
12
# Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
13
advert_int 1
14
# Configure the authentication type and password
15
authentication
16
# Configure the authentication type, mainly PASS and AH
17
auth_type PASS
18
# Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
19
auth_pass amber1
20
21
virtual_ipaddress
22
# The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
23
192.168.1.99/24 dev ens160 label ens160:1
24
25
track_script
26
chk_haproxy
27
28
3) Configuration on 192.168.1.168
YAML
xxxxxxxxxx
1
28
1
global_defs
2
router_id lb01 #An identifier；
3
4
vrrp_script chk_haproxy
5
script "killall -0 haproxy" interval 2
6
7
vrrp_instance VI_1
8
state BACKUP
9
interface ens160
10
virtual_router_id 52
11
priority 777
12
# Set the interval (in seconds) for sync check between MASTER and BACKUP load balancers.
13
advert_int 1
14
# Configure the authentication type and password
15
authentication
16
# Configure the authentication type, mainly PASS and AH
17
auth_type PASS
18
# Set the authentication password.The MASTER and BACKUP load balancers must use the same password within the same vrrp_instance to communicate properly.
19
auth_pass amber1
20
21
virtual_ipaddress
22
# The virtual IP is 192.168.1.99/24; Bounded interface is ens160; Alias for both MASTER and BACKUP load balancers is ens160:1
23
192.168.1.99/24 dev ens160 label ens160:1
24
25
track_script
26
chk_haproxy
27
28
Relevant commands used in Keepalived:
Shell
xxxxxxxxxx
1
1
# Start Keepalived
2
systemctl start keepalived
3
# Auto start Keepalived
4
systemctl enable keeplived
5
# Re-start Keepalived
6
systemctl restart keepalived
Deploy the Cluster Offline
How to deploy the Nebula Graph cluster with Docker Swarm offline? The answer is simple. Changing the Docker image to private image repo will do.
Please let me know if you have any questions deploying your own cluster by leaving comments below!
Docker (software)
Nebula (computing)
Graph (Unix)
Load balancing (computing)
YAML
Published at DZone with permission of henson wu. See the original article here.
Opinions expressed by DZone contributors are their own.
Trending
-
A Complete Guide to Agile Software Development
-
An Overview of Cloud Cryptography
-
Authorization: Get It Done Right, Get It Done Early
-
Unlocking Game Development: A Review of ‘Learning C# By Developing Games With Unity'
Comments