feat(spanner): support MultiEndpoint

[nimf]

This PR adds optional gRPC-GCP channel pool (which is available in Cloud Spanner Java client) with its new MultiEndpoint feature.

The MultiEndpoint accepts a grouped lists of endpoints and creates a gRPC-GCP channel pool for each unique endpoint. The purposes of MultiEndpoint are:

• Fallback to an alternative endpoint (host:port) when the original
  endpoint is completely unavailable.
• Be able to route a Cloud Spanner call to a specific group of endpoints.
• Be able to reconfigure endpoints in runtime.

Here is an example of creating a client with MultiEndpoint

gmeConfig := &grpcgcp.GCPMultiEndpointOptions{
	MultiEndpoints: map[string]*multiendpoint.MultiEndpointOptions{
		"default": {
			Endpoints: []string{
				"psc-east4:443",
			},
		},
	},
	Default: "default",
}

client, gme, err := NewMultiEndpointClient(ctx, database, gmeConfig)

Endpoints reconfiguration

Then, with the above example, to route the calls to a different endpoint we can:

gmeConfig = &grpcgcp.GCPMultiEndpointOptions{
	MultiEndpoints: map[string]*multiendpoint.MultiEndpointOptions{
		"default": {
			Endpoints: []string{
				"psc-west2:443",
			},
		},
	},
	Default: "default",
}

err := gme.UpdateMultiEndpoints(gmeConfig)

And all the following calls in the client will go to the west2 endpoint instead of east4.

Using several MultiEndpoints

We can configure several MultiEndpoints, for example:

gmeConfig := &grpcgcp.GCPMultiEndpointOptions{
	MultiEndpoints: map[string]*multiendpoint.MultiEndpointOptions{
		"default": {
			Endpoints: []string{
				"psc-east4:443",
			},
		},
		"west": {
			Endpoints: []string{
				"psc-west2:443",
			},
		},
	},
	Default: "default",
}

client, gme, err := NewMultiEndpointClient(ctx, database, gmeConfig)

Then we can route a call to a specific MultiEndpoint ("default" is used by default) using context:

westCtx := grpcgcp.NewMEContext(ctx, "west")

iter := client.Single().Query(westCtx, spanner.NewStatement(sql))

The call above will use an endpoint from the "west" MultiEndpoint group.

Fallback

If we add more than one endpoint to a MultiEndpoint, like in the example below:

gmeConfig := &grpcgcp.GCPMultiEndpointOptions{
	MultiEndpoints: map[string]*multiendpoint.MultiEndpointOptions{
		"default": {
			Endpoints: []string{
				"psc-east4:443",
				"psc-east4-fallback:443",
				"psc-west2:443",
				"psc-west2-fallback:443",
			},
			RecoveryTimeout: recoveryTimeout,
			SwitchingDelay:  switchingDelay,
		},
		"west": {
			Endpoints: []string{
				"psc-west2:443",
				"psc-west2-fallback:443",
				"psc-east4:443",
				"psc-east4-fallback:443",
			},
			RecoveryTimeout: recoveryTimeout,
			SwitchingDelay:  switchingDelay,
		},
	},
	Default: "default",
}

client, gme, err := NewMultiEndpointClient(ctx, database, gmeConfig)

Each MultiEndpoint will use the first endpoint in the list as long as any connection to that endpoint is alive.
When all connections to the first endpoint are broken, the MultiEndpoint will wait for recoveryTimeout and if the connections are still down, fail over to the next endpoint and so forth.
When a connection to one of the previous endpoints in the list is recovered, the MultiEndpoint will switch back to that endpoint after switchingDelay.

[codyoss]

please updated your title to be in the form of feat(spanner): ... and include a little more detail in the commit title.

[dfawley]

Note that mutating a var args parameter (opts) is dangerous. This code should be making a copy first.

https://go.dev/play/p/UAEsaz4xxXg

This is a contrived example to illustrate what's happening, but there is real risk in mutating a variadic argument, especially for exported APIs.

[nimf]

Also fixing in #11151

[dfawley]

This could potentially be even worse. If the DialFunc is run concurrently in the ChannelPool, then this function will be overwriting memory in one goroutine while reading it in another, which could cause serious problems, and should be expected to trigger the race detector if nothing else. It looks like that doesn't happen today - grpc clients are created serially, but I could see a user complaining about startup performance and sending a pr that does them each in their own goroutine, and then this becomes broken as a result. A copy should ideally be made before mutating the opts slice.

[nimf]

Oops, that wasn't intended. Thank you for pointing this out. Sending PR #11151 to fix this.