DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • The Production-Ready Kubernetes Service Checklist
  • Demystifying Kubernetes in 5 Minutes
  • Strengthening Your Kubernetes Cluster With Pod Security Admission
  • From Novice to Expert: Building Robust Security With Kubernetes RBAC

Trending

  • A Modern Stack for Building Scalable Systems
  • Beyond Linguistics: Real-Time Domain Event Mapping with WebSocket and Spring Boot
  • Streamlining Event Data in Event-Driven Ansible
  • Mastering Fluent Bit: Installing and Configuring Fluent Bit on Kubernetes (Part 3)
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. How to Secure a Previously Insecure Cluster in Production

How to Secure a Previously Insecure Cluster in Production

Learn how to secure a previously insecure cluster in production and how to mitigate any pitfalls that come with running an insecure cluster in production.

By 
Artem Ervits user avatar
Artem Ervits
DZone Core CORE ·
Feb. 08, 22 · Tutorial
Likes (3)
Comment
Save
Tweet
Share
4.7K Views

Join the DZone community and get the full member experience.

Join For Free

Cockroach Labs does not recommend running an insecure cluster in production. There are only a few additional steps necessary to secure an instance, so why do it? Convenience, you say. It can hurt you down the line but fret not, this article will demonstrate how to fix this. We are going to follow the standard insecure cluster start-up procedure. Once complete, we're going to flip to the documentation for a secure cluster to turn each node on with security enabled. Here's a handy video of the procedure in action:


Step by step instructions are below:

Secure Cluster Take Over

Begin with an insecure cluster.

cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=l 
ocalhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
*
bash-3.2$ cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=l 
ocalhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
bash-3.2$ cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=l 
ocalhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background
*
* WARNING: RUNNING IN INSECURE MODE!
* 
* - Your cluster is open for any client that can access localhost.
* - Any user, even root, can log in without providing a password.
* - Any user, connecting as root, can read or write any data in your cluster.
* - There is no network encryption nor authentication, and thus no confidentiality.
* 
* Check out how to secure your cluster: https://www.cockroachlabs.com/docs/v20.1/secure-a-cluster.html
*
*
* INFO: initial startup completed.
* Node will now attempt to join a running cluster, or wait for `cockroach init`.
* Client connections will be accepted after this completes successfully.
* Check the log file(s) for progress. 
bash-3.2$ cockroach init --insecure --host=localhost:26257
Cluster successfully initialized
bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11
CockroachDB node starting at 2020-04-27 13:39:56.808756 +0000 UTC (took 19.4s)
build:               CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9)
webui:               http://localhost:8080
sql:                 postgresql://root@localhost:26257?sslmode=disable
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --insecure
logs:                /Users/artem/Downloads/secure/node1/logs
temp dir:            /Users/artem/Downloads/secure/node1/cockroach-temp775384998
external I/O path:   /Users/artem/Downloads/secure/node1/extern
store[0]:            path=/Users/artem/Downloads/secure/node1
storage engine:      rocksdb
status:              initialized new cluster
clusterID:           42062b66-c4c5-4167-bae6-9b0e2ebc47f5
bash-3.2$ cockroach sql --insecure --host=localhost:26257
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26257/defaultdb> 
CREATE DATABASE bank;
CREATE DATABASE

Time: 4.644ms

root@localhost:26257/defaultdb> 
CREATE TABLE bank.accounts (id INT PRIMARY KEY, balance DECIMA 
L);
CREATE TABLE

Time: 5.005ms

root@localhost:26257/defaultdb> 
INSERT INTO bank.accounts VALUES (1, 1000.50);
INSERT 1

Time: 5.354ms

root@localhost:26257/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 1.465ms

root@localhost:26257/defaultdb> \q
bash-3.2$ # connect from another node
bash-3.2$ cockroach sql --insecure --host=localhost:26258
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26258/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 7.537ms

root@localhost:26258/defaultdb> \q

Secure cluster take over:

bash-3.2$ mkdir certs my-safe-directory
bash-3.2$ cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key
bash-3.2$ cockroach cert create-node localhost $(hostname) --certs-dir=certs --ca-key=my-safe- 
directory/ca.key
bash-3.2$ cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.ke 
y

Stop each node one by one and restart in secure mode, because each node has the same hostname, we will reuse the cert:

bash-3.2$ ps aux | grep cokroach
artem            73363   1.9  0.7  4843760 220372 s003  S     9:39AM   0:05.50 cockroach start --insecure --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73368   1.0  0.5  4787908 175784 s003  S     9:39AM   0:04.55 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73371   0.9  0.5  4787176 170988 s003  S     9:39AM   0:04.20 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73461   0.0  0.0  4417788    840 s003  S+    9:42AM   0:00.00 grep cockroach
bash-3.2$ kill 73363 # stopping node1
bash-3.2$ cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http 
-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ clear
bash-3.2$ ps aux | grep cockroach
artem            73371   4.3  0.5  4795368 174144 s003  S     9:39AM   0:04.86 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73368   3.6  0.5  4796100 177476 s003  S     9:39AM   0:05.18 cockroach start --insecure --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73470   0.9  0.2  4622236  60644 s003  S     9:42AM   0:00.39 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73476   0.0  0.0  4399356    800 s003  S+    9:42AM   0:00.00 grep cockroach
kill 73368 # node 2
bash-3.2$ cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http 
-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ clear
bash-3.2$ ps aux | grep cockroach
artem            73470   2.1  0.3  4700764  86980 s003  S     9:42AM   0:01.51 cockroach start --certs-dir=certs --store=node1 --listen-addr=localhost:26257 --http-addr=localhost:8080 --join=localhost:26257,localhost:26258,localhost:26259
artem            73486   1.8  0.2  4694508  72112 s003  S     9:43AM   0:00.59 cockroach start --certs-dir=certs --store=node2 --listen-addr=localhost:26258 --http-addr=localhost:8081 --join=localhost:26257,localhost:26258,localhost:26259
artem            73371   0.7  0.5  4797672 178068 s003  S     9:39AM   0:05.85 cockroach start --insecure --store=node3 --listen-addr=localhost:26259 --http-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259
artem            73492   0.0  0.0  4410620    848 s003  S+    9:43AM   0:00.00 grep cockroach
bash-3.2$ kill 73371 # node 3
bash-3.2$ cockroach start --certs-dir=certs --store=node3 --listen-addr=localhost:26259 --http 
-addr=localhost:8082 --join=localhost:26257,localhost:26258,localhost:26259 --background
bash-3.2$ grep 'node starting' node1/logs/cockroach.log -A 11
CockroachDB node starting at 2020-04-27 13:43:27.451629 +0000 UTC (took 44.3s)
build:               CCL v20.1.0-rc.1 @ 2020/04/10 01:53:13 (go1.13.9)
webui:               https://localhost:8080
sql:                 postgresql://root@localhost:26257?sslcert=certs%2Fclient.root.crt&sslkey=certs%2Fclient.root.key&sslmode=verify-full&sslrootcert=certs%2Fca.crt
RPC client flags:    cockroach <client cmd> --host=localhost:26257 --certs-dir=certs
logs:                /Users/artem/Downloads/secure/node1/logs
temp dir:            /Users/artem/Downloads/secure/node1/cockroach-temp837519533
external I/O path:   /Users/artem/Downloads/secure/node1/extern
store[0]:            path=/Users/artem/Downloads/secure/node1
storage engine:      rocksdb
status:              restarted pre-existing node
clusterID:           42062b66-c4c5-4167-bae6-9b0e2ebc47f5
bash-3.2$ # connect to the cluster with client certs
bash-3.2$ cockroach sql --certs-dir=certs --host=localhost:26257
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26257/defaultdb> show databases;
  database_name
-----------------
  bank
  defaultdb
  postgres
  system
(4 rows)

Time: 2.002ms

root@localhost:26257/defaultdb> use bank;
SET

Time: 993µs

root@localhost:26257/bank> 
use bank;
show databases;
SELECT * FROM bank.accounts;
�[28G\q�[K
�[28GSELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 5.893ms

root@localhost:26257/bank> \q
bash-3.2$ # connect to another node 
bash-3.2$ # cockroach sql --certs-dir=certs --host=localhost:26259
#
# Welcome to the CockroachDB SQL shell.
# All statements must be terminated by a semicolon.
# To exit, type: \q.
#
# Server version: CockroachDB CCL v20.1.0-rc.1 (x86_64-apple-darwin14, built 2020/04/10 01:53:13, go1.13.9) (same version as client)
# Cluster ID: 42062b66-c4c5-4167-bae6-9b0e2ebc47f5
#
# Enter \? for a brief introduction.
#
root@localhost:26259/defaultdb> 
SELECT * FROM bank.accounts;
  id | balance
-----+----------
   1 | 1000.50
(1 row)

Time: 5.105ms

root@localhost:26259/defaultdb> \q

Once complete, you may also validate the certs by creating a user with admin privileges and navigating to the certificates page in the Admin UI, such as https://<address of node with new certs>:8080/#/reports/certificates/local and view all of the available certificates.

CREATE USER roach WITH PASSWORD 'cockroach';

GRANT ADMIN TO roach;

SHOW ROLES;

    username    |  options   | member_of
----------------+------------+------------
  admin         | CREATEROLE | {}
  roach         |            | {admin}
  root          | CREATEROLE | {admin}
(3 rows)

Certificate Authority node certificate client certificate

cluster security Production (computer science)

Published at DZone with permission of Artem Ervits. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • The Production-Ready Kubernetes Service Checklist
  • Demystifying Kubernetes in 5 Minutes
  • Strengthening Your Kubernetes Cluster With Pod Security Admission
  • From Novice to Expert: Building Robust Security With Kubernetes RBAC

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!