How to Secure a Previously Insecure Cluster in Production

Learn how to secure a previously insecure cluster in production and how to mitigate any pitfalls that come with running an insecure cluster in production.

Artem Ervits

CORE ·

Feb. 08, 22 · Tutorial

Likes (3)

Comment

Save

4.8K Views

Cockroach Labs does not recommend running an insecure cluster in production. There are only a few additional steps necessary to secure an instance, so why do it? Convenience, you say. It can hurt you down the line but fret not, this article will demonstrate how to fix this. We are going to follow the standard insecure cluster start-up procedure. Once complete, we're going to flip to the documentation for a secure cluster to turn each node on with security enabled. Here's a handy video of the procedure in action:

Step by step instructions are below:

Secure Cluster Take Over

Begin with an insecure cluster.

cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=l 
ocalhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
*

bash-3.2$ cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=l 
ocalhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.

bash-3.2$ cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=l 
ocalhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress.

bash-3.2$ cockroach init --insecure --host=localhost:26257
Cluster successfully initialized
bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11
CockroachDB node starting at 2020-04-27 13:39:56.808756 +0000 UTC (took 19.4s)
build:               CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9)
webui:               http://localhost:8080
sql:                 postgresql://root@localhost:26257?sslmode=disable
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --insecure
logs:                /Users/artem/Downloads/secure/node1/logs
temp dir:            /Users/artem/Downloads/secure/node1/cockroach-temp775384998
external I/O path:   /Users/artem/Downloads/secure/node1/extern
store[0]:            path=/Users/artem/Downloads/secure/node1
storage engine:      rocksdb
status:              initialized new cluster
clusterID:           42062b66-c4c5-4167-bae6-9b0e2ebc47f5

bash-3.2$ cockroach sql --insecure --host=localhost:26257
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26257/defaultdb> 
CREATE DATABASE bank;
CREATE DATABASE

Time: 4.644ms

root@localhost:26257/defaultdb> 
CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMA 
L);
CREATE TABLE

Time: 5.005ms

root@localhost:26257/defaultdb> 
INSERT INTO bank.accounts VALUES (1, 1000.50);
INSERT 1

Time: 5.354ms

root@localhost:26257/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 1.465ms

root@localhost:26257/defaultdb> \q
bash-3.2$ # connect from another node
bash-3.2$ cockroach sql --insecure --host=localhost:26258
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26258/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 7.537ms

root@localhost:26258/defaultdb> \q

Secure cluster take over:

bash-3.2$ mkdir certs my-safe-directory
bash-3.2$ cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
bash-3.2$ cockroach cert create-node localhost $(hostname) --certs-dir=certs --ca-key=my-safe- 
directory/ca.key
bash-3.2$ cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.ke 
y

Stop each node one by one and restart in secure mode, because each node has the same hostname, we will reuse the cert:

bash-3.2$ ps aux | grep cokroach
artem            73363   1.9  0.7  4843760 220372 s003  S     9:39AM   0:05.50 cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73368   1.0  0.5  4787908 175784 s003  S     9:39AM   0:04.55 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73371   0.9  0.5  4787176 170988 s003  S     9:39AM   0:04.20 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73461   0.0  0.0  4417788    840 s003  S+    9:42AM   0:00.00 grep cockroach
bash-3.2$ kill 73363 # stopping node1
bash-3.2$ cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http 
-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ clear
bash-3.2$ ps aux | grep cockroach
artem            73371   4.3  0.5  4795368 174144 s003  S     9:39AM   0:04.86 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73368   3.6  0.5  4796100 177476 s003  S     9:39AM   0:05.18 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73470   0.9  0.2  4622236  60644 s003  S     9:42AM   0:00.39 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73476   0.0  0.0  4399356    800 s003  S+    9:42AM   0:00.00 grep cockroach
kill 73368 # node 2
bash-3.2$ cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http 
-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ clear
bash-3.2$ ps aux | grep cockroach
artem            73470   2.1  0.3  4700764  86980 s003  S     9:42AM   0:01.51 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73486   1.8  0.2  4694508  72112 s003  S     9:43AM   0:00.59 cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73371   0.7  0.5  4797672 178068 s003  S     9:39AM   0:05.85 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73492   0.0  0.0  4410620    848 s003  S+    9:43AM   0:00.00 grep cockroach
bash-3.2$ kill 73371 # node 3
bash-3.2$ cockroach start --certs-dir=certs --store=node3 --listen-addr=localhost:26259 --http 
-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11
CockroachDB node starting at 2020-04-27 13:43:27.451629 +0000 UTC (took 44.3s)
build:               CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9)
webui:               https://localhost:8080
sql:                 postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --certs-dir=certs
logs:                /Users/artem/Downloads/secure/node1/logs
temp dir:            /Users/artem/Downloads/secure/node1/cockroach-temp837519533
external I/O path:   /Users/artem/Downloads/secure/node1/extern
store[0]:            path=/Users/artem/Downloads/secure/node1
storage engine:      rocksdb
status:              restarted pre-existing node
clusterID:           42062b66-c4c5-4167-bae6-9b0e2ebc47f5

bash-3.2$ # connect to the cluster with client certs
bash-3.2$ cockroach sql --certs-dir=certs --host=localhost:26257
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26257/defaultdb> show databases;
  database_name
-----------------
  bank
  defaultdb
  postgres
  system
(4 rows)

Time: 2.002ms

root@localhost:26257/defaultdb> use bank;
SET

Time: 993µs

root@localhost:26257/bank> 
use bank;
show databases;
SELECT * FROM bank.accounts;
�[28G\q�[K
�[28GSELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 5.893ms

root@localhost:26257/bank> \q
bash-3.2$ # connect to another node 
bash-3.2$ # cockroach sql --certs-dir=certs --host=localhost:26259
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26259/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 5.105ms

root@localhost:26259/defaultdb> \q

Once complete, you may also validate the certs by creating a user with admin privileges and navigating to the certificates page in the Admin UI, such as https://<address of node with new certs>:8080/#/reports/certificates/local and view all of the available certificates.

CREATE USER roach WITH PASSWORD 'cockroach';

GRANT ADMIN TO roach;

SHOW ROLES;

    username    |  options   | member_of
----------------+------------+------------
  admin         | CREATEROLE | {}
  roach         |            | {admin}
  root          | CREATEROLE | {admin}
(3 rows)

cluster security Production (computer science)

Published at DZone with permission of Artem Ervits. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

Trending

How to Secure a Previously Insecure Cluster in Production

Learn how to secure a previously insecure cluster in production and how to mitigate any pitfalls that come with running an insecure cluster in production.

Secure Cluster Take Over

Related

Partner Resources