Using CockroachDB Workloads With Kerberos

GSSAPI authentication is becoming increasingly popular as CockroachDB starting to make inroads in Fortune 2000 customer bases and financial services segment. That said, ecosystem coverage for GSS needs to improve for parity with other authN methods. Today, we are providing a workaround and a look at the future. By the way, do you realize this is my 15th article on Kerberos and CockroachDB?

Articles Covering CockroachDB and Kerberos

I find the topic of Kerberos very interesting and my colleagues commonly refer to me for help with this complex topic. I am by no means an expert at Kerberos, I am however familiar enough with it to be dangerous. That said, I've written multiple articles on the topic which you may find below:

1. CockroachDB With MIT Kerberos and Docker Compose
2. Executing CockroachDB table import via GSSAPI
3. CockroachDB With SQLAlchemy and MIT Kerberos
4. CockroachDB With MIT Kerberos Cert User Authentication
5. CockroachDB with Django and MIT Kerberos
6. CockroachDB With Kerberos and Custom Service Principal Name (SPN)
7. Simplifying CockroachDB Kerberos Architecture With a Load Balancer
8. CockroachDB with mixed Kerberos and cert authentication
9. CockroachDB with GSSAPI deployed via systemd
10. Selecting proper cipher for CockroachDB with GSSAPI
11. Overriding KRB5CCNAME for CockroachDB with GSSAPI
12. DNS gotchas with CockroachDB and GSSAPI
13. Using CockroachDB workload with Kerberos

Motivation

CockroachDB workload generator is a popular tool used internally and externally to provide a running metric of how CockroachDB behaves from one release to another as well as offering a comparable baseline to other database flavors. Workload relies on third-party Go libraries for PostgreSQL connectivity, two of which make up critical components for CockroachDB: lib/pq and pgx drivers. To date, all of the GSS related work has been done with lib/pq. It has stood the test of time but Go community has been moving away from it towards pgx. Additionally, lib/pq GH repo has the following statement: For users that require new features or reliable resolution of reported bugs, we recommend using pgx which is under active development. Unfortunately, pgx does not support GSS authN. Lack of GSS has come up at least twice this week alone across two different use cases. Today, we're going to cover one use case impacting our largest financial services organizations. I do have to warn you that I do not have a solution as much as a workaround that may or may not fit your requirements. The gist of the problem is that workload CLI command relies on both libraries and the real solution is to either provide pgx GSS capability and/or remove pgx and fill the gaps in workload with lib/pq alone. My personal opinion is to go with the former, but I will defer to engineering with the decision. In the meantime, let me demonstrate a workaround to potentially unblock this scenario.

High-Level Steps

• Start a three-node CockroachDB cluster in Docker with GSSAPI
• Demonstrate the problem scenario
• Workaround
• Verify
• Clean up

Step by Step Instructions

Start a cluster

There's nothing special about this tutorial than what was covered in my previous tutorials, feel free to set up a stand-alone environment to follow along or use my docker-compose environment.

git clone https://github.com/dbist/cockroach-docker.git
cd cockroach-gssapi-multinode
./up.sh

Demonstrate the Problem Scenario

I need to connect to the client container to demonstrate the issue.

docker exec -it client bash

Cockroach workload expects connection string instead of --host and --certs-dir flags. We need to pass a connection string similar to postgresql://root@localhost:26257?sslmode=disable. You can familiarize with all of the available options here. If you remember from my previous tutorials, to connect to a Kerberos-enabled cluster, we have to use something like this

cockroach sql --certs-dir=/certs --host=lb.local --user=tester

Tester user is a valid user we have in KDC.

#
# Enter \? for a brief introduction.
#
tester@lb.local:26257/defaultdb>

Let's attempt to initialize the workload using the tester user.

cockroach \
 workload \
 fixtures \
 import \
 tpcc \
 "postgresql://tester:nopassword@lb.local:26257/tpcc?sslmode=verify-full&sslrootcert=/certs/ca.crt&krbsrvname=customspn"

We get the following

Error: pq: permission denied to create database

Let's add tester to admin role. For that, we need to connect to the cluster using a root client or another admin account. I only have root client.

GRANT ADMIN TO tester WITH ADMIN OPTION;

Run the workload init again

I220218 14:12:19.281893 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
I220218 14:12:19.718693 83 ccl/workloadccl/fixture.go:478  [-] 2  imported 1017 B in district table (10 rows, 0 index entries, took 409.892ms, 0.00 MiB/s)
I220218 14:12:20.430798 87 ccl/workloadccl/fixture.go:478  [-] 3  imported 114 KiB in new_order table (9000 rows, 0 index entries, took 1.1203485s, 0.10 MiB/s)
I220218 14:12:20.509430 82 ccl/workloadccl/fixture.go:478  [-] 4  imported 53 B in warehouse table (1 rows, 0 index entries, took 1.2004323s, 0.00 MiB/s)
I220218 14:12:20.981628 89 ccl/workloadccl/fixture.go:478  [-] 5  imported 31 MiB in stock table (100000 rows, 0 index entries, took 1.6719129s, 18.30 MiB/s)
I220218 14:12:21.291022 90 ccl/workloadccl/fixture.go:478  [-] 6  imported 16 MiB in order_line table (300343 rows, 0 index entries, took 1.9820492s, 8.33 MiB/s)
I220218 14:12:21.404788 85 ccl/workloadccl/fixture.go:478  [-] 7  imported 2.2 MiB in history table (30000 rows, 0 index entries, took 2.0952243s, 1.04 MiB/s)
I220218 14:12:21.425125 84 ccl/workloadccl/fixture.go:478  [-] 8  imported 18 MiB in customer table (30000 rows, 30000 index entries, took 2.1156929s, 8.30 MiB/s)
I220218 14:12:21.708730 86 ccl/workloadccl/fixture.go:478  [-] 9  imported 1.5 MiB in order table (30000 rows, 30000 index entries, took 2.3995018s, 0.63 MiB/s)
I220218 14:12:21.992139 88 ccl/workloadccl/fixture.go:478  [-] 10  imported 7.8 MiB in item table (100000 rows, 0 index entries, took 2.6816118s, 2.90 MiB/s)
I220218 14:12:22.047743 1 ccl/workloadccl/fixture.go:354  [-] 11  imported 76 MiB bytes in 9 tables (took 2.7648271s, 27.57 MiB/s)
I220218 14:12:23.863738 1 ccl/workloadccl/cliccl/fixtures.go:355  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
I220218 14:12:23.897875 1 workload/tpcc/tpcc.go:509  [-] 13  check 3.3.2.1 took 34.0216ms
I220218 14:12:23.964014 1 workload/tpcc/tpcc.go:509  [-] 14  check 3.3.2.2 took 66.0118ms
I220218 14:12:23.985257 1 workload/tpcc/tpcc.go:509  [-] 15  check 3.3.2.3 took 21.1735ms
I220218 14:12:24.163443 1 workload/tpcc/tpcc.go:509  [-] 16  check 3.3.2.4 took 178.0787ms
I220218 14:12:24.211839 1 workload/tpcc/tpcc.go:509  [-] 17  check 3.3.2.5 took 48.3283ms
I220218 14:12:24.426848 1 workload/tpcc/tpcc.go:509  [-] 18  check 3.3.2.7 took 214.9428ms
I220218 14:12:24.472901 1 workload/tpcc/tpcc.go:509  [-] 19  check 3.3.2.8 took 45.9853ms
I220218 14:12:24.537016 1 workload/tpcc/tpcc.go:509  [-] 20  check 3.3.2.9 took 64.0062ms

We are able to advance to the workload run step.

cockroach workload \
 run tpcc \
 --duration=20m \
 --conns 10 \
 --ramp=3m \
 --workers=10 \
 --tolerate-errors \
 "postgresql://tester:nopassword@lb.local:26257/tpcc?sslmode=verify-full&sslrootcert=/certs/ca.crt&krbsrvname=customspn"

This is where everything starts to break

I220218 14:14:07.577813 1 workload/cli/run.go:408  [-] 1  creating load generator...
Initializing 10 connections...
I220218 14:14:07.587287 95 workload/pgx_helpers.go:72  [-] 2  pgx logger [error]: connect failed logParams=map[err:AuthTypeGSS is unimplemented]
W220218 14:14:07.638318 1 workload/cli/run.go:417  [-] 3  retrying after error while creating load: failed to initialize the load generator: AuthTypeGSS is unimplemented
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4  a panic has occurred!
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +duplicate metrics collector registration attempted
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +(1) attached stack trace
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  -- stack trace:
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/prometheus/client_golang/prometheus/promauto.Factory.NewCounter
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/prometheus/client_golang/prometheus/promauto/auto.go:265
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/workload/tpcc.setupTPCCMetrics
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/worker.go:99
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/workload/tpcc.(*tpcc).Ops
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/tpcc.go:754
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func2
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:419
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/workload/cli.runRun
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:436
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/workload/cli.CmdHelper.func1
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:223
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/workload/cli.HandleErrs.func1
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/cli.go:87
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/spf13/cobra.(*Command).execute
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/spf13/cobra.(*Command).ExecuteC
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/spf13/cobra.(*Command).Execute
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/cli.Run
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:283
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/cli.doMain
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:130
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | github.com/cockroachdb/cockroach/pkg/cli.Main
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:59
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | main.main
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach/main.go:26
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | runtime.main
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /usr/local/go/src/runtime/proc.go:225
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  | runtime.goexit
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +  |        /usr/local/go/src/runtime/asm_amd64.s:1371
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +Wraps: (2) duplicate metrics collector registration attempted
E220218 14:14:07.638669 1 1@util/log/logcrash/crash_reporting.go:140  [-] 4 +Error types: (1) *withstack.withStack (2) prometheus.AlreadyRegisteredError
panic: duplicate metrics collector registration attempted [recovered]
        panic: duplicate metrics collector registration attempted

goroutine 1 [running]:
github.com/cockroachdb/cockroach/pkg/util/log/logcrash.RecoverAndReportPanic(0x8c889a0, 0xc00013a010, 0xc000a42000)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/logcrash/crash_reporting.go:90 +0xa5
panic(0x4db8f00, 0xc00120e180)
        /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc001102000, 0xc00072ee40, 0x1, 0x1)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/prometheus/client_golang/prometheus/registry.go:401 +0xb7
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewCounter(0x8c60fa0, 0xc001102000, 0x52b6442, 0x8, 0x52a8b0a, 0x4, 0xc0007f8030, 0x16, 0xc000706800, 0x35, ...)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/prometheus/client_golang/prometheus/promauto/auto.go:265 +0x12f
github.com/cockroachdb/cockroach/pkg/workload/tpcc.setupTPCCMetrics(0x8c60fa0, 0xc001102000, 0x1)
        /go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/worker.go:99 +0x2cd
github.com/cockroachdb/cockroach/pkg/workload/tpcc.(*tpcc).Ops(0xc0001aa780, 0x8c889d8, 0xc000d4acc0, 0xc000a89110, 0x1, 0x7, 0xc000d4ac60, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/tpcc.go:754 +0x8c
github.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func2(0x7f41605110d8, 0xc0001aa780, 0xc000a89110, 0x1, 0x7, 0xc000d4ac60, 0xc00077c040, 0x8c889d8, 0xc000d4acc0, 0x1, ...)
        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:419 +0x207
github.com/cockroachdb/cockroach/pkg/workload/cli.runRun(0x8c3e588, 0xc0001aa780, 0xc000a89110, 0x1, 0x7, 0x52a8b0a, 0x4, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:436 +0x685
github.com/cockroachdb/cockroach/pkg/workload/cli.CmdHelper.func1(0xc000d54f00, 0xc000a89110, 0x1, 0x7, 0x4398377, 0x8c889a0)
        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:223 +0x1e4
github.com/cockroachdb/cockroach/pkg/workload/cli.HandleErrs.func1(0xc000d54f00, 0xc000a89110, 0x1, 0x7)
        /go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/cli.go:87 +0x5d
github.com/spf13/cobra.(*Command).execute(0xc000d54f00, 0xc000a890a0, 0x7, 0x7, 0xc000d54f00, 0xc000a890a0)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xb2dcda0, 0xc00013a010, 0xc0007ba4ea, 0x11)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
        /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:283
github.com/cockroachdb/cockroach/pkg/cli.doMain(0xc000d54f00, 0xc0007ba4ea, 0x11, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:130 +0x11a
github.com/cockroachdb/cockroach/pkg/cli.Main()
        /go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:59 +0x13a
main.main()
        /go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach/main.go:26 +0x25

The key error is [-] 2  pgx logger [error]: connect failed logParams=map[err:AuthTypeGSS is unimplemented]

What is left to do is rely on a certs authN for our purpose.

Workaround

What we have to do is either use our root client user to run the workload or implement a mixed authN state where some users are able to connect with certs and others will rely on KDC. This is described in my 10th article.

I happen to have a user roach which relies on certs created. All we have to do is grant it admin rights to create databases. We need to be connected to CockroachDB with an admin user.

GRANT ADMIN TO roach WITH ADMIN OPTION;

Let's also modify the hba config to allow this user authN access.

SET cluster setting server.host_based_authentication.configuration = 'host all roach all cert
host all all all gss include_realm=0';

Confirm the configuration looks like below

SHOW cluster setting server.host_based_authentication.configuration;

  server.host_based_authentication.configuration
--------------------------------------------------
  host all roach all cert
  host all all all gss include_realm=0

At this point, we should be able to init and run the workload using roach user.

cockroach \
 workload \
 fixtures \
 import \
 tpcc \
 "postgresql://roach@lb.local:26257/tpcc?sslcert=/certs%2Fclient.roach.crt&sslkey=/certs%2Fclient.roach.key&sslmode=verify-full&sslrootcert=/certs%2Fca.crt"

I220218 14:27:23.923399 1 ccl/workloadccl/fixture.go:345  [-] 1  starting import of 9 tables
I220218 14:27:24.551990 72 ccl/workloadccl/fixture.go:478  [-] 2  imported 2.2 MiB in history table (30000 rows, 0 index entries, took 598.9423ms, 3.68 MiB/s)
I220218 14:27:25.038922 69 ccl/workloadccl/fixture.go:478  [-] 3  imported 54 B in warehouse table (1 rows, 0 index entries, took 1.0871543s, 0.00 MiB/s)
I220218 14:27:25.129639 75 ccl/workloadccl/fixture.go:478  [-] 4  imported 7.9 MiB in item table (100000 rows, 0 index entries, took 1.1758183s, 6.69 MiB/s)
I220218 14:27:25.344410 77 ccl/workloadccl/fixture.go:478  [-] 5  imported 17 MiB in order_line table (300343 rows, 0 index entries, took 1.3927475s, 12.05 MiB/s)
I220218 14:27:25.467984 74 ccl/workloadccl/fixture.go:478  [-] 6  imported 123 KiB in new_order table (9000 rows, 0 index entries, took 1.5144006s, 0.08 MiB/s)
I220218 14:27:25.991409 71 ccl/workloadccl/fixture.go:478  [-] 7  imported 18 MiB in customer table (30000 rows, 30000 index entries, took 2.0387504s, 8.64 MiB/s)
I220218 14:27:26.654524 70 ccl/workloadccl/fixture.go:478  [-] 8  imported 1.0 KiB in district table (10 rows, 0 index entries, took 2.7023935s, 0.00 MiB/s)
I220218 14:27:28.058776 76 ccl/workloadccl/fixture.go:478  [-] 9  imported 31 MiB in stock table (100000 rows, 0 index entries, took 4.1047727s, 7.48 MiB/s)
I220218 14:27:28.143085 73 ccl/workloadccl/fixture.go:478  [-] 10  imported 1.6 MiB in order table (30000 rows, 30000 index entries, took 4.1898382s, 0.37 MiB/s)
I220218 14:27:28.197433 1 ccl/workloadccl/fixture.go:354  [-] 11  imported 77 MiB bytes in 9 tables (took 4.2727076s, 17.99 MiB/s)
I220218 14:27:30.235457 1 ccl/workloadccl/cliccl/fixtures.go:355  [-] 12  fixture is restored; now running consistency checks (ctrl-c to abort)
I220218 14:27:30.270919 1 workload/tpcc/tpcc.go:509  [-] 13  check 3.3.2.1 took 35.3795ms
I220218 14:27:30.332077 1 workload/tpcc/tpcc.go:509  [-] 14  check 3.3.2.2 took 61.0754ms
I220218 14:27:30.352936 1 workload/tpcc/tpcc.go:509  [-] 15  check 3.3.2.3 took 20.796ms
I220218 14:27:30.545975 1 workload/tpcc/tpcc.go:509  [-] 16  check 3.3.2.4 took 192.973ms
I220218 14:27:30.597143 1 workload/tpcc/tpcc.go:509  [-] 17  check 3.3.2.5 took 51.1032ms
I220218 14:27:30.804996 1 workload/tpcc/tpcc.go:509  [-] 18  check 3.3.2.7 took 207.7871ms
I220218 14:27:30.849415 1 workload/tpcc/tpcc.go:509  [-] 19  check 3.3.2.8 took 44.3551ms
I220218 14:27:30.906854 1 workload/tpcc/tpcc.go:509  [-] 20  check 3.3.2.9 took 57.3705ms

Verify

Finally, run the workload

cockroach \
 workload \
 run \
 tpcc \
 --duration=20m \
 --conns 10 \
 --ramp=3m \
 --workers=10 \
 --tolerate-errors \
 "postgresql://roach@lb.local:26257/tpcc?sslcert=/certs%2Fclient.roach.crt&sslkey=/certs%2Fclient.roach.key&sslmode=verify-full&sslrootcert=/certs%2Fca.crt"

I220218 14:29:17.476376 1 workload/cli/run.go:408  [-] 1  creating load generator...
Initializing 10 connections...
Initializing 0 idle connections...
Initializing 10 workers and preparing statements...
I220218 14:29:17.503484 1 workload/cli/run.go:439  [-] 2  creating load generator... done (took 27.0899ms)
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
    1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
    1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    1.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
    2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
    2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    2.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
    3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    3.0s        0            1.0            0.3    142.6    142.6    142.6    142.6 orderStatus
    3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    3.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
    4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    4.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
    4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    4.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
    5.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    5.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    5.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
    5.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    5.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
    6.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    6.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    6.0s        0            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
    6.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    6.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
    7.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    7.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    7.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
    7.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    7.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
    8.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    8.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    8.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
    8.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    8.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
    9.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
    9.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
    9.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
    9.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
    9.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
   10.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
   10.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
   10.0s        0            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
   10.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
   10.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel

Cleanup

After you're done you can tear down the demo environment

./down.sh

Again this is not perfect but at least it shows a path forward, ultimately adopting GSS authN in pgx will establish peace in the Golang Kerberos world!