Kalix CLI login issue with SSL Proxy

Hello, I am doing the evaluation. So, I was trying to use Kalix CLI to login. However, due to the company proxy server layer, the CLI login process resulted in connection error. I suspect that it is related to SSL handshaking issue due to the trusted certificate rewrite in proxy layer. Is there any open source code of the CLI so that we can compile the CLI with own CA certificates? or, can we configure the SSL trusted certificates setting inside the CLI executables?

1 Like

I am also getting this kind of error. What is needed to setup the appropriate proxy information to be able to connect successfully?

C:\Users\kd>kalix auth login
Opening browser window to authorize login:
https://console.kalix.io/login/cli/b056aa1dc514e50012c53017790d28e1127703b71a1d2cf16cb7b07c196d80c7

Waiting for UI login...
Error: unable to create token: rpc error: code = Unauthenticated desc = The supplied bearer token could not be verified
C:\Users\kd>kalix regions list
Error: unable to list regions: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing failed to do connect handshake, response: \"HTTP/1.1 407 Proxy Authentication Required\\r\\nConnection: close\\r\\nContent-Length: 2079\\r\\nContent-Type: text/html\\r\\nDate: Mon, 11 Jul 2022 14:58:49 GMT\\r\\nMime-Version: 1.0\\r\\nProxy-Authenticate: Negotiate\\r\\nProxy-Authenticate: NTLM\\r\\nProxy-Authenticate: Basic realm=\\\"Cisco IronPort Web Security Appliance\\\"\\r\\nProxy-Connection: close\\r\\n\\r\\n<!DOCTYPE HTML PUBLIC \\\"-//W3C//DTD HTML 4.01 Transitional//EN\\\"\\n\\\"http://www.w3.org/TR/html4/loose.dtd\\\">\\n<html>\\n<head>\\n<meta http-equiv=\\\"Content-Type\\\" content=\\\"text/html; charset=UTF-8\\\">\\n<title>Notification: Proxy Authorization Required</title>\\n<style type=\\\"text/css\\\">\\nbody {\\n  font-family: Arial, Helvetica, sans-serif;\\n  font-size: 14px;\\n  color:#333333;\\n  background-color: #ffffff;\\n}\\nh1 {\\n  font-size: 18px;\\n  font-weight: bold;\\n  text-decoration: none;\\n  padding-top: 0px;\\n  color: #2970A6;\\n}\\na:link {\\n    color: #2970A6;\\n  text-decoration: none;\\n}\\na:hover {\\n    color: #2970A6;\\n  text-decoration: underline;\\n}\\np.buttonlink {\\n  margin-bottom: 24px;\\n}\\n.copyright {\\n  font-size: 12px;\\n  color: #666666;\\n  margin: 5px 5px 0px 30px;\\n\\n}\\n.details {\\n  font-size: 14px;\\n  color: #969696;\\n  border: none;\\n  padding: 20px 20px 20px 20px;\\n  margin: 0px 10px 10px 35px;\\n}\\n\\n.shadow {\\n  border: 3px solid #9f9f9f;\\n  padding: 10px 25px 10px 25px;\\n  margin: 10px 35px 0px 30px;\\n  background-color: #ffffff;\\n  width: 600px;\\n\\n  -moz-box-shadow: 3px 3px 3px #cccccc;\\n  -webkit-box-shadow: 3px 3px 3px #cccccc;\\n  box-shadow: 3px 3px 3px #cccccc;\\n  /* For IE 8 */\\n  -ms-filter: \\\"progid:DXImageTransform.Microsoft.Shadow(Strength=5, Direction=135, Color='cccccc')\\\";\\n  /* For IE 5.5 - 7 */\\n  filter: progid:DXImageTransform.Microsoft.Shadow(Strength=5, Direction=135, Color='cccccc');\\n}\\n.logo {\\n  border: none;\\n  margin: 5px 5px 0px 30px;\\n}\\n</style>\\n</head>\\n\\n<body>\\n<div class=\\\"logo\\\"></div><p>&nbsp;</p>\\n<div class=\\\"shadow\\\">\\n<h1>This Page Cannot Be Displayed</h1>\\n\\n\\n<p>\\nAuthentication is required to access the Internet using this system.\\nA valid user ID and password must be entered when prompted.\\n</p>\\n\\n\\n\\n<p>\\nIf you have questions, please contact\\nService Desk \\nand provide the codes shown below.\\n</p>\\n\\n</div>\\n\\n<div class=\\\"details\\\"><p>\\nDate: Mon, 11 Jul 2022 14:58:49 GMT<br />\\nUsername: <br />\\nSource IP: 10.28.218.19<br />\\nURL: CONNECT https://api.kalix.io/<br />\\nCategory: URL Filtering Bypassed<br />\\nReason: UNKNOWN<br />\\nNotification: PROXY_AUTH_REQUIRED\\n</p></div>\\n</body>\\n</html>\\n\""
1 Like

In my testing, I got this error when using “auth login command”:

C:\kalix>kalix auth login

Here are the outcome under different test scenarios.

Before using VPN, and No PROXY setting in environment.

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup api.kalix.io: no such host"

Before using VPN, and Added HTTPS_PROXY in system environment variables.

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup myproxy.mycompany.mydomain: no such host"

After using VPN, and No PROXY setting in environment.

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup api.kalix.io: no such host" 

After using VPN, and Added HTTPS_PROXY in system environment variables.

Error: rpc error: code = Unavailable desc = connection closed

@faichun Is your company using transparent proxying, or explicit? If transparent (which is a really bad security practice), the CLI is implemented in Go, and from my searching, I believe that setting the SSL_CERT_FILE environment variable can be used on any Go app to override where the trusted certs are read from. I haven’t tested that though.

Note that even if you do this, you may still have problems. The Kalix CLI uses gRPC, including gRPC streamed connections. This requires HTTP/2. Many corporate proxies don’t support HTTP/2 yet, and even when they do, can have problems with gRPC streamed connections.

If using explict proxying, here’s the documentation for Go’s gRPC client:

Note that this requires the proxy to support the CONNECT method.

I can’t really comment on your tests since I have no idea what VPN you’re talking about, but from the looks of it, it appears your VPN has issues if it’s unable to resolve anything, including your companies proxy, from DNS.

@kenandalley You’ll need to give more information than that. How are you configuring the proxy? If it’s a transparent proxy, then the issue there appears to be that the proxy only supports HTTP/1.1, the kalix command uses gRPC and therefore requires HTTP/2.

Thank you for the suggestion. But no luck after using SSL_CERT_FILE and SSL_CERT_DIR option. So, may be not related to certificate issue. For the proxy type, I have no idea at the moment and I need time to ask. If the CLI is something go lang related, is it possible to enable debug mode so that i can see the log on what is going on?

After I enable the go logging variables, I see more lines now.

export GRPC_GO_LOG_VERBOSITY_LEVEL=99
export GRPC_GO_LOG_SEVERITY_LEVEL=info
export GRPC_TRACE=all
export GODEBUG=http2debug=2

Here is the result:

root:/# kalix auth login
INFO: 2022/07/12 05:38:16 [core] parsed scheme: ""
INFO: 2022/07/12 05:38:16 [core] scheme "" not registered, fallback to default scheme
INFO: 2022/07/12 05:38:16 [core] ccResolverWrapper: sending update to cc: {[{api.kalix.io:443  <nil> 0 <nil>}] <nil> <nil>}
INFO: 2022/07/12 05:38:16 [core] ClientConn switching balancer to "pick_first"
INFO: 2022/07/12 05:38:16 [core] Channel switches to new LB policy "pick_first"
INFO: 2022/07/12 05:38:16 [core] Subchannel Connectivity change to CONNECTING
INFO: 2022/07/12 05:38:16 [core] pickfirstBalancer: UpdateSubConnState: 0xc000355c00, {CONNECTING <nil>}
INFO: 2022/07/12 05:38:16 [core] Subchannel picks a new address "api.kalix.io:443" to connect
INFO: 2022/07/12 05:38:16 [core] blockingPicker: the picked transport is not ready, loop back to repick
INFO: 2022/07/12 05:38:16 [core] Channel Connectivity change to CONNECTING
2022/07/12 05:38:17 http2: Framer 0xc0007aac40: wrote SETTINGS len=0
INFO: 2022/07/12 05:38:17 [core] Subchannel Connectivity change to TRANSIENT_FAILURE
INFO: 2022/07/12 05:38:17 [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
INFO: 2022/07/12 05:38:17 [core] pickfirstBalancer: UpdateSubConnState: 0xc000355c00, {TRANSIENT_FAILURE connection closed}
INFO: 2022/07/12 05:38:17 [core] Channel Connectivity change to TRANSIENT_FAILURE
Error: rpc error: code = Unavailable desc = connection closed

Another finding when using a machine with direct connection. NO PROXY in the network.

INFO: 2022/07/13 15:09:08 [core] parsed scheme: ""
INFO: 2022/07/13 15:09:08 [core] scheme "" not registered, fallback to default scheme
INFO: 2022/07/13 15:09:08 [core] ccResolverWrapper: sending update to cc: {[{api.kalix.io:443  <nil> 0 <nil>}] <nil> <nil>}
INFO: 2022/07/13 15:09:08 [core] ClientConn switching balancer to "pick_first"
INFO: 2022/07/13 15:09:08 [core] Channel switches to new LB policy "pick_first"
INFO: 2022/07/13 15:09:08 [core] Subchannel Connectivity change to CONNECTING
INFO: 2022/07/13 15:09:08 [core] Subchannel picks a new address "api.kalix.io:443" to connect
INFO: 2022/07/13 15:09:08 [core] pickfirstBalancer: UpdateSubConnState: 0xc000263850, {CONNECTING <nil>}
INFO: 2022/07/13 15:09:08 [core] Channel Connectivity change to CONNECTING
2022/07/13 15:09:08 http2: Framer 0xc00052b0a0: wrote SETTINGS len=0
2022/07/13 15:09:08 http2: Framer 0xc00052b0a0: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=65536
2022/07/13 15:09:08 http2: Framer 0xc00052b0a0: read WINDOW_UPDATE len=4 (conn) incr=983041
2022/07/13 15:09:08 http2: Framer 0xc00052b0a0: read SETTINGS flags=ACK len=0
INFO: 2022/07/13 15:09:08 [core] Subchannel Connectivity change to READY
2022/07/13 15:09:08 http2: Framer 0xc00052b0a0: wrote SETTINGS flags=ACK len=0
INFO: 2022/07/13 15:09:08 [core] pickfirstBalancer: UpdateSubConnState: 0xc000263850, {READY <nil>}
INFO: 2022/07/13 15:09:08 [core] Channel Connectivity change to READY
2022/07/13 15:09:08 http2: Framer 0xc00052b0a0: wrote HEADERS flags=END_HEADERS stream=1 len=176
2022/07/13 15:09:08 http2: Framer 0xc00052b0a0: wrote DATA flags=END_STREAM stream=1 len=23 data="\x00\x00\x00\x00\x12\n\x10faichun-MBP-2019"
2022/07/13 15:09:09 http2: Framer 0xc00052b0a0: read HEADERS flags=END_HEADERS stream=1 len=245
2022/07/13 15:09:09 http2: decoded hpack field header field ":status" = "200"
2022/07/13 15:09:09 http2: decoded hpack field header field "content-type" = "application/grpc+proto"
2022/07/13 15:09:09 http2: decoded hpack field header field "content-length" = "224"
2022/07/13 15:09:09 http2: decoded hpack field header field "grpc-encoding" = "identity"
2022/07/13 15:09:09 http2: decoded hpack field header field "ratelimit-limit" = "60;w=60"
2022/07/13 15:09:09 http2: decoded hpack field header field "ratelimit-remaining" = "59"
2022/07/13 15:09:09 http2: decoded hpack field header field "access-control-allow-origin" = "*"
2022/07/13 15:09:09 http2: decoded hpack field header field "access-control-allow-headers" = "*"
2022/07/13 15:09:09 http2: decoded hpack field header field "access-control-allow-methods" = "*"
2022/07/13 15:09:09 http2: decoded hpack field header field "access-control-max-age" = "86400"
2022/07/13 15:09:09 http2: decoded hpack field header field "date" = "Wed, 13 Jul 2022 05:09:09 GMT"
2022/07/13 15:09:09 http2: decoded hpack field header field "server" = "akka-http/10.2.9"
2022/07/13 15:09:09 http2: decoded hpack field header field "via" = "1.1 google"
2022/07/13 15:09:09 http2: decoded hpack field header field "alt-svc" = "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
2022/07/13 15:09:09 http2: Framer 0xc00052b0a0: read DATA stream=1 len=224 data="\x00\x00\x00\x00\xdb\nJlogin/cli/0bc6de5f976cfa5dbcddf170cefcf6a896f915c4fb0c6008dd867da706a72d26\x12\x10faichun-MBP-2019\x1a\x0e159.196.169.71\"\x06\b\xa1\xa9\xb9\x96\x06*chttps://console.kalix.io/login/cli/0bc6de5f976cfa5dbcddf170cefcf6a896f915c4fb0c6008dd867da706a72d26"
2022/07/13 15:09:09 http2: Framer 0xc00052b0a0: read HEADERS flags=END_STREAM|END_HEADERS stream=1 len=12
2022/07/13 15:09:09 http2: decoded hpack field header field "grpc-status" = "0"
2022/07/13 15:09:09 http2: Framer 0xc00052b0a0: read PING len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2022/07/13 15:09:09 http2: Framer 0xc00052b0a0: wrote WINDOW_UPDATE len=4 (conn) incr=224
2022/07/13 15:09:09 http2: Framer 0xc00052b0a0: wrote PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2022/07/13 15:09:09 http2: Framer 0xc00052b0a0: wrote PING flags=ACK len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
Opening browser window to authorize login:

so, when comparing the normal (no PROXY in environment) and abnormal (with PROXY in environment) situation, I can see that the difference starting from the log related to http2, i.e.

2022/07/12 05:38:17 http2: Framer 0xc0007aac40: wrote SETTINGS len=0