Python Logging

Logging Challenges

Include contextual information
Make the transition from dev-only to machine-readable logs
Consistet wording

Structlog

Structlog is a third-party logging library. It helps to solve the above challenges.

import structlog  
logger = structlog.get_logger()  
...  
logger.debug("Start uploading file.", file_name=file_name,retries=0)  
...  
logger.error("Failed uploading file.", file_name=file_name,retries=retries)  
...

Pasted image 20230507130030.png

We can also bind the variables to the context to avoid rewriting them every time.

import structlog  
logger = structlog.get_logger()  
...  
log = logger.bind(file_name=file_name)  
log.debug("Start uploading file.", retries=0)  
...  
log.error("Failed uploading file.", retries=retries)  
...

Then, we can convert our logs to machine-readable logs.

import structlog  
from structlog.dev import ConsoleRenderer  
from structlog.processors import JSONRenderer  
  
# Timestamper preprocessor that to add unified timestamps to each log  
timestamper = structlog.processors.TimeStamper(fmt="iso", utc=True)  
  
# Structlog preprocessors  
structlog_processors = [  
structlog.stdlib.add_log_level  
structlog.processors.add_log_level,  
structlog.contextvars.merge_contextvars,  
structlog.processors.StackInfoRenderer(),  
structlog.dev.set_exc_info,  
timestamper,  
]

Now we want to define a function that either returns structlogs ConsoleRendereror JSONRenderer, based on the environment settings and append this renderer to the processors.

# __init__.py  
  
from structlog.dev import ConsoleRenderer  
from structlog.processors import JSONRenderer  
  
def get_renderer() -> Union[JSONRenderer, ConsoleRenderer]:  
"""  
Get renderer based on the environment settings  
:return structlog renderer  
"""  
if os.get("DEV_LOGS", True):  
return ConsoleRenderer()  
return JSONRenderer()  
  
  
## Structlog  
structlog.configure(  
processors=structlog_processors + [get_renderer()],  
wrapper_class=structlog.stdlib.BoundLogger,  
context_class=dict,  
logger_factory=structlog.PrintLoggerFactory(),  
cache_logger_on_first_use=False,  
)

Limitations

Structlog can't solve all the problems on its own. We also have to use a set of conventions:

<entity>_id <- ( required) every entity (e.g. file) that is processed needs to be added to the logger
<entity>_name <- (optional) every entity (e.g. file) can be added with the suffix _name for a better readability
<entity>_<additional_key> <- (optional) additional properties of entities are allowed to be added with the key name as a suffix
<custom_usecase_specific_field> ← (optional) Sometimes, you may need to add case-specific information. In such cases, you can use additional arbitrary keys. However, it is recommended to follow the naming conventions from 1–3, if possible.

Python Logging

Logging Challenges

Structlog

Limitations

META