loading
loading
How to Use Python UDF Logging for Pipeline Observability
Cong Du
Software Engineer
Ming Ma
Software Engineer
We are announcing a new logging feature that is now available in Python User-Defined Function (UDF) in BigQuery. Developers can now use Python’s standard logging library directly inside BigQuery UDFs to emit structured logs seamlessly into Cloud Logging—with zero external plugins or custom connectors required.
While UDF logging enables a wide range of operational monitoring and debugging workflows, to demonstrate this feature we are showcasing an example for building fault-tolerant, resilient data pipelines. In the example below, we are capturing row-level diagnostic telemetry during query execution, so that teams can easily isolate problematic records, trigger real-time alerts, and triage issues via SQL without halting entire workloads.
In this post, we’ll show you how to get started with Python UDF logging and walk through an e-commerce event ingestion pipeline as a practical, end-to-end example.
Architecture overview
The workflow consists of three stages:
- Inline validation: The Python UDF parses incoming JSON payloads. Valid records pass to the destination table; corrupted or non-compliant rows trigger logging.error() and return NULL to prevent query failure.
- Real-time alerting: A Cloud Monitoring log-based metric notifies your team when error rates breach SLA thresholds.
- SQL triage & replay: Data engineers query the quarantined payloads in BigQuery using standard SQL over the linked _AllLogs view, diagnose the root cause, and re-ingest recovered data.
Although this specific demonstration relies on Python for data validation and transformation, you can seamlessly integrate Python-driven logging and validation into pipelines that perform SQL-based transformations.
Quickstart walkthrough: Step-by-step
Step 1: Create the generic Python UDF
This UDF parses JSON and checks for a required field. If parsing fails or the required key is missing, it logs structured diagnostic JSON and returns NULL.
Step 2: Run ingestion with logging enabled
In this example, we simulate an e-commerce event ingestion pipeline processing incoming user actions (click, view, checkout). Our goal is to extract valid events into our clean analytics table while gracefully quarantining broken or missing-schema payloads.
Enable logging in your session with SET @@udf.properties:
Ingestion Results
The query succeeds without interruption while bad rows (2 and 3) are routed to Cloud Logging.
Step 3: Set up real-time alerts on Cloud Logging
You can configure alerts to notify your on-call team as soon as malformed payloads exceed acceptable thresholds.
In the Google Cloud Console (1-Click)
- In the Google Cloud Console, open Logs Explorer.
- Paste the log filter for your UDF routine:
3. Click Create alert in the top action bar.
4. Set your notification channels (e.g., Slack, PagerDuty, or Email) and save the policy.
Step 4: Triage bad rows with BigQuery Log Analytics
"Note: To query your logs via BigQuery SQL, ensure you have enabled BigQuery Log Analytics on your Cloud Logging bucket and created a linked dataset."
Query the quarantined payloads using BigQuery SQL via the linked _AllLogs view:
Log Analytics output
Recovering data
Because full raw payloads are preserved in Cloud Logging, you can recover quarantined records directly with SQL once upstream data is fixed:
Final clean_events Table Result:
All valid and recovered data is now unified in the production table without any data loss.
Other use cases
- Shadow execution & safe migrations: Run legacy and new business logic side-by-side in a UDF. Return the legacy result to production while logging calculation differences to Cloud Logging.
- PII redaction auditing: Mask sensitive tokens (such as SSNs or account numbers) before writing to BigQuery, while emitting anonymized audit events to Cloud Logging.
- ML inference drift: Log warnings when embedded scoring logic encounters out-of-vocabulary inputs or low model confidence.
Best practices
- Enable debug output: Include SET @@udf.properties = JSON '{"enableDebugOutput": true}'; in the session where you want logs captured.
- Avoid sensitive data in logs: Cloud Logging is visible to users with Logging Viewer permissions. Do not log credentials or secrets.
- Cost management: On high-throughput pipelines, log selectively (e.g., only on errors or via sampling) to manage logging ingestion volume.document fetched 2026-10-24 11:20 · canonical https://cloud.google.com/blog/products/data-analytics/python-udf-logging-for-pipeline-observability-in-bigquery