Support for logging formatter class

[keegan2149]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Is your feature request related to a problem? Please describe.
N/A
Describe the solution you'd like
Support for python logging formatter class
Describe alternatives you've considered
I would like to add the function name and line number to the log entires. The only alternative is to add formatting to each log entry.
Additional context

[daniel-sanche]

I'll look into the best way to implement this likely next quarter. Although we will likely add automatic source_entry detection to the handlers first, which seems like a more direct solution to your use case

[willjhenry]

Does this issue apply to setting a formatter string, for example:

from google.cloud import logging as cloudlogging
lg_client = cloudlogging.Client()
lg_handler = lg_client.get_default_handler()
lg_handler.setLevel(logging_level)
formatter = logging.Formatter('%(levelname)s - %(funcName)s  - %(lineno)d - %(message)s')
lg_handler.setFormatter(formatter)

This works when I run it locally. The log message is successfully sent to my google cloud logging logs, with the correct format. However, when I run the code live (in a cloud function) no formatting is applied.

[willjhenry]

Never mind, I was looking at the logs generated by the default integration of cloud functions and the python logging module. The logs generated by this client library were formatted fine. (Now I need to figure out how to disable the default logging so I can just use this client library and not have duplicate logs)

[MadJlzz]

I am not sure if this issue reflects what I am thinking but it would be great to be able to get the formatter class or the default handler independently from the Client().

As far as I understood, creating a Client() creates a connection (gRPC?) that I think is an overhead.

In fact, the documentation states that for some services (like Cloud Functions or AppEngine), log entries are automatically parsed by Cloud Logging from stdout and stderr if they follow this structure.

It might save some ressources for applications to avoid sending requests for logging as LogEntries are already parsed by Cloud Logging.

Because I am deploying to other cloud provider as well ; I would be able then to format my logs in a certain way whether I am using GCP or not.

Does that make sense?

[daniel-sanche]

@MadJlzz good timing, that's exactly what I've been working on this week: #228

Starting with the next release, Cloud Functions and Cloud Run should print JSON logs directly to standard output while other environments will continue to use gRPC/HTTP clients to send logs. (Of course this applies when using the automatic logging integration, you will still be able to instantiate different handlers directly if desired)

Because I am deploying to other cloud provider as well ; I would be able then to format my logs in a certain way whether I am using GCP or not.

Can you explain this part a bit more? Do you want your logs on other cloud providers to show up in GCP Cloud Logging? Or are you just interested in using the same JSON format without sending to GCP?

[MadJlzz]

Perfect timing indeed! The fact that we won't need to write our own formatter is a good news 👍

AppEngine will not be concerned? It's also using the same pattern of reading stdout and stderr, no?

I don't know how the library will be exposed but I foresee something like this:

import logging

from google.cloud.logging import CloudLoggingHandler

# set up logging by overriding default handlers of Python logging module.
logging.basicConfig(level=logging.DEBUG)

# add the handler to the root logger
logging.getLogger('').addHandler(CloudLoggingHandler())

Can you explain this part a bit more? Do you want your logs on other cloud providers to show up in GCP Cloud Logging? Or are you just interested in using the same JSON format without sending to GCP?

It's more for being able to adapt the format depending on the platform we deploy!

1. Reading JSON logs when developing locally is really painfull so we would like to use a huma redeable format.

2. On-premise people are usually using their own logs aggregator that may filter log entries if not properly formatted...

3. On other Cloud providers, they might use another format to automatically aggregate logs from their services.

The upper code block would then look like this:

import logging

from google.cloud.logging import CloudLoggingHandler

# set up logging by overriding default handlers of Python logging module.
logging.basicConfig(level=logging.DEBUG)

root_logger = logging.getLogger('')

# our context module knows if we are on GCP, AWS, Azure, On-premise or even locally.
# it appends automatically to the given logger the propper handler.
context.addLoggingHandler(root_logger)

# and then in each .py file
logger = logging.getLogger(__name__)

# log record properly formatted, thanks to the handlers ;)
logger.error('uh oh, something really bad happend...')