Documentation for passing the cluster configuration to update_cluster

[go-dustin]

When trying to update a running cluster, you are required to pass in the cluster configuration. The documentation isn't clear on what that should be and expects you to know what the proto-buffer should be. Which creates a bit of circular problem, if you don't know what the proto-buffer is supposed to be you have no point of reference, so you won't know what to pass. We could use an example explaining how this works.

"If a dict is provided, it must be of the same form as the protobuf message Cluster"

response = client.update_cluster(project_id, region, cluster_name, cluster, update_mask)

---

In the meantime, can someone please explain how to this works? Ideally I'd be able to get the config from an existing cluster and then pass that with the update_mask to increase/decrease the number of workers.

[bradmiro]

Hey @go-dustin, thanks for your feedback here. I had to do some digging myself to figure out the correct way to do this and agree this is a great opportunity for us to improve our documentation. The following worked for me, in this case to double the number of workers on the cluster.

from google.cloud import dataproc_v1 as dataproc

project_id = <project_id>
region = <region>
cluster_name = <cluster>

client = dataproc.ClusterControllerClient(
    client_options={"api_endpoint": f"{region}-dataproc.googleapis.com:443"}
)

cluster = client.get_cluster(
    project_id=project_id, region=region, cluster_name=cluster_name
)

new_num_instances = cluster.config.worker_config.num_instances * 2
mask = { 
    "paths":  {
        "config.worker_config.num_instances": str(new_num_instances)
    }
}
cluster.config.worker_config.num_instances = new_num_instances // Must update the cluster info itself

operation = client.update_cluster(
    project_id=project_id, 
    region=region, 
    cluster=cluster,
    cluster_name=cluster_name,
    update_mask=mask
)

updated_cluster = operation.result()

print(updated_cluster.config.worker_config.num_instances == new_num_instances) // Should be true

Let me know if this helps or if I can answer any further questions :)

[go-dustin]

@bradmiro this was exactly what i needed thank you very much!

[bradmiro]

Glad to hear it @go-dustin! Closing this, feel free to reopen if you run into more issues!

[go-dustin]

This answered my question but shouldn't we leave this open so the documentation gets updated?

[bradmiro]

Hey @go-dustin , sorry for the delay in reply. I am tracking this internally to get our documentation updated :) thanks again!

[loferris]

I'm creating a PR to add the above fix to our samples documentation!