entry.to_api_repr() causes exception Can not find message descriptor by type_url: type.googleapis.com/google.iam.v1.logging.AuditData

[SimSama]

Hi Team,

Following the code example to list entries, we want the response data in json format.

We used the function of LogEntry to_api_repr() and works for most logs.

e.g. print(entry.to_api_repr())

This causes an exception for certain log types.

Reproduce:

1. Edit iam permission for a user
2. Poll list of logs using python client e.g. client.list_entries
3. Print each log in json format using to_api_repr() function.
   e.g.

for entry in entries: 
    print(entry.to_api_repr())

For this particular event you'll get an exception below:
Can not find message descriptor by type_url: type.googleapis.com/google.iam.v1.logging.AuditData

I unfortunately can't post the log, I can later perhaps if required, have to strip out some data first.

[minherz]

@SimSama from the issue description it is impossible to reproduce it although it sounds like a problem in converting audit log entry to JSON. However, it will help us if you will be able to provide a minimal program that reproduce the error with as much details about the environment as possible. We cannot reproduce it based on the currently provided information.

[jeremiahRedacted]

I was able to replicate this using this short script that just uses a service account to list logs in a log bucket then print them. The installed version of google-cloud-logging is 2.6.0

from google.cloud.logging import Client
from google.oauth2 import service_account
import json


def authenticate_to_gsuite_service_account(token_file):
    creds = service_account.Credentials.from_service_account_file(
        token_file)
    return creds

def main():
    creds = authenticate_to_gsuite_service_account('creds_file.json')

    client = Client(
        project='PROJECT_NAME',
        credentials=creds
    )

    entries = client.list_entries()

    for x in entries:
        try:
            print(json.dumps(x.to_api_repr()))
        except TypeError:
            print(x)
            raise

if __name__ == '__main__':
    main()

That returns a bunch of good logs, then when it gets to a log using type.googleapis.com/google.iam.v1.logging.AuditData it throws this error:

Traceback (most recent call last):
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 405, in _CreateMessageFromTypeUrl
    message_descriptor = pool.FindMessageTypeByName(type_name)
KeyError: "Couldn't find message google.iam.v1.logging.AuditData"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Volumes/Working/working/test/gcp_log_buckets/access_gcp_logs.py", line 80, in <module>
    main()
  File "/Volumes/Working/working/test/gcp_log_buckets/access_gcp_logs.py", line 59, in main
    print(json.dumps(x.to_api_repr()))
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/cloud/logging_v2/entries.py", line 332, in to_api_repr
    proto_payload = MessageToDict(self.payload_pb)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 175, in MessageToDict
    return printer._MessageToJsonObject(message)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 214, in _MessageToJsonObject
    return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 344, in _AnyMessageToJsonObject
    return self._RegularMessageToJsonObject(sub_message, js)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 251, in _RegularMessageToJsonObject
    js[name] = self._FieldToJsonObject(field, value)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 285, in _FieldToJsonObject
    return self._MessageToJsonObject(value)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 214, in _MessageToJsonObject
    return methodcaller(_WKTJSONMETHODS[full_name][0], message)(self)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 333, in _AnyMessageToJsonObject
    sub_message = _CreateMessageFromTypeUrl(type_url, self.descriptor_pool)
  File "/Volumes/Working/working/test/gcp_log_buckets/venv/lib/python3.8/site-packages/google/protobuf/json_format.py", line 407, in _CreateMessageFromTypeUrl
    raise TypeError(
TypeError: Can not find message descriptor by type_url: type.googleapis.com/google.iam.v1.logging.AuditData.

Printing the protobuf itself looks like this (with some sanitization...):

ProtobufEntry(log_name='projects/PROJECT_NAME/logs/cloudaudit.googleapis.com%2Factivity', labels=None, insert_id='XXXXXXXX', severity='NOTICE', http_request=None, timestamp=datetime.datetime(2021, 9, 9, 19, 1, 20, 107560, tzinfo=datetime.timezone.utc), resource=Resource(type='project', labels={'project_id': 'PROJECT_ID'}), trace=None, span_id=None, trace_sampled=None, source_location=None, operation=None, logger=<google.cloud.logging_v2.logger.Logger object at 0x10e40e7c0>, payload=type_url: "type.googleapis.com/google.cloud.audit.AuditLog"
value: "\022\000\032 \n\036user@domain.net\"\230\001\n\rxx.xx.xx.xx\022\202\001Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.61 Safari/537.36,gzip(gfe):\000B\000:#cloudresourcemanager.googleapis.comB\014SetIamPolicyJ\277\001\n\037projects/PROJECT_NAME\022%resourcemanager.projects.setIamPolicy\030\001*s\n#cloudresourcemanager.googleapis.com\022\037projects/PROJECT_NAME\032+cloudresourcemanager.googleapis.com/ProjectJ\277\001\n\037projects/PROJECT_NAME\022%resourcemanager.projects.setIamPolicy\030\001*s\n#cloudresourcemanager.googleapis.com\022\037projects/PROJECT_NAME\032+cloudresourcemanager.googleapis.com/ProjectZ\037projects/PROJECT_NAMEz\241\001\n3type.googleapis.com/google.iam.v1.logging.AuditData\022j\022h\nf\010\001\022\023roles/logging.admin\032MserviceAccount:SERVICE_ACCCOUNT@PROJECT_NAME.iam.gserviceaccount.com\202\001\326\n\n@\n\005@type\0227\0325type.googleapis.com/google.iam.v1.SetIamPolicyRequest\n$\n\010resource\022\030\032\026PROJECT_NAME\n\353\t\n\006policy\022\340\t*\335\t\n\302\t\n\010bindings\022\265\t2\262\t\nu*s\n\027\n\004role\022\017\032\rroles/browser\nX\n\007members\022M2K\nI\032GserviceAccount:SERVICE_ACCOUNT0PROJECT_NAME.iam.gserviceaccount.com\n{*y\n\035\n\004role\022\025\032\023roles/compute.admin\nX\n\007members\022M2K\nI\032GserviceAccount:SERVICE_ACCOUNT@PROJECT_NAME.iam.gserviceaccount.com\n\206\001*\203\001\n[\n\007members\022P2N\nL\032JserviceAccount:service-123456@compute-system.iam.gserviceaccount.com\n$\n\004role\022\034\032\032roles/compute.serviceAgent\n\203\001*\200\001\n$\n\004role\022\034\032\032roles/compute.storageAdmin\nX\n\007members\022M2K\nI\032GserviceAccount:SERVICE_ACCOUNT@PROJECT_NAME.iam.gserviceaccount.com\n\263\001*\260\001\n\225\001\n\007members\022\211\0012\206\001\nC\032AserviceAccount:123456-compute@developer.gserviceaccount.com\n?\032=serviceAccount:123456@cloudservices.gserviceaccount.com\n\026\n\004role\022\016\032\014roles/editor\na*_\n5\n\007members\022*2(\n&\032$user:user@domain.com\n&\n\004role\022\036\032\034roles/iam.serviceAccountUser\n\231\001*\226\001\n\025\n\004role\022\r\032\013roles/owner\n}\n\007members\022r2p\n%\032#user:user@domain.com\n&\032$user:user@domain.com\n\037\032\035user:user@domain.com\nt*r\nX\n\007members\022M2K\nI\032GserviceAccount:SERVICE_ACCOUNT@PROJECT_NAME.iam.gserviceaccount.com\n\026\n\004role\022\016\032\014roles/viewer\n\201\001*\177\n\035\n\004role\022\025\032\023roles/logging.admin\n^\n\007members\022S2Q\nO\032MserviceAccount:USERNAME@PROJECT_NAME.iam.gserviceaccount.com\n\026\n\004etag\022\016\032\014BwXLlFlcfX0=\212\001\222\n\n3\n\005@type\022*\032(type.googleapis.com/google.iam.v1.Policy\n\026\n\004etag\022\016\032\014BwXLlJ/UlRo=\n\302\t\n\010bindings\022\265\t2\262\t\nu*s\n\027\n\004role\022\017\032\rroles/browser\nX\n\007members\022M2K\nI\032GserviceAccount:USERNAME@PROJECT_NAME.iam.gserviceaccount.com\n{*y\n\035\n\004role\022\025\032\023roles/compute.admin\nX\n\007members\022M2K\nI\032GserviceAccount:USERNAME@PROJECT_NAME.iam.gserviceaccount.com\n\206\001*\203\001\n[\n\007members\022P2N\nL\032JserviceAccount:service-123456@compute-system.iam.gserviceaccount.com\n$\n\004role\022\034\032\032roles/compute.serviceAgent\n\203\001*\200\001\nX\n\007members\022M2K\nI\032GserviceAccount:USERNAME@PROJECT_NAME.iam.gserviceaccount.com\n$\n\004role\022\034\032\032roles/compute.storageAdmin\n\263\001*\260\001\n\225\001\n\007members\022\211\0012\206\001\nC\032AserviceAccount:123456-compute@developer.gserviceaccount.com\n?\032=serviceAccount:123456@cloudservices.gserviceaccount.com\n\026\n\004role\022\016\032\014roles/editor\na*_\n&\n\004role\022\036\032\034roles/iam.serviceAccountUser\n5\n\007members\022*2(\n&\032$user:user@domain.com\n\201\001*\177\n\035\n\004role\022\025\032\023roles/logging.admin\n^\n\007members\022S2Q\nO\032MserviceAccount:USERNAME@PROJECT_NAME.iam.gserviceaccount.com\n\231\001*\226\001\n\025\n\004role\022\r\032\013roles/owner\n}\n\007members\022r2p\n%\032#user:user@domain.com\n&\032$user:user@domain.com\n\037\032\035user:user@domain.com\nt*r\nX\n\007members\022M2K\nI\032GserviceAccount:USERNAME@PROJECT_NAME.iam.gserviceaccount.com\n\026\n\004role\022\016\032\014roles/viewer"

[daniel-sanche]

Thanks for the report. We are currently officially supporting the type.googleapis.com/google.cloud.audit.AuditLog and type.googleapis.com/google.appengine.logging.v1.RequestLog protos, but it looks like type.googleapis.com/google.iam.v1.logging.AuditData should be added to the list. I'll try to look at this for the next release

[SimSama]

Sorry for the extreme delay on my end. Thanks Jeremiah for the save. I ended up working around it by using the pub/sub client instead. That seemed like the more efficient way to aggregate data for my purposes. Also very minor item, but would it be better to just call it JSON format as opposed to just "api representation"? this line

x.to_api_repr()

Add a func or rename function to:

x.to_json_format()

Thanks all,

[jeremiahRedacted]

Also of note. Installing google-cloud-iam-logging and importing from google.cloud import iam_logging_v1 in addition to what I imported above will also apparently fix it without having to add in any addition code to the x.to_api_repr() bit.