loading
loading
From weeks to minutes: The new agentic era of data pipelines Piotr Wieczorek Lead Product Manager Alexandre Moueddene Software Engineer Data pipelines are the backbone of the modern enterprise, yet a barrier to entry exists for orchestrating them, making this critical capability unavailable to many data professionals. Following our announcements at Google Cloud NEXT ’26, where we introduced the Orchestration Pipelines framework, we are fundamentally changing this dynamic. To bring this powerful framework directly to practitioners, we offer the Data Agent Kit — a unified, freely available, and open-source collection of data engineering and data science tools that integrate directly into your preferred IDE or CLI (such as VS Code, Claude Code, or Codex). The Data Agent Kit seamlessly embeds the Orchestration Pipelines framework into your workflow in two distinct ways. First, it provides a dedicated Data Engineering tab for comprehensive pipeline management. Second, it includes a specialized agentic skill designed to author, deploy, and troubleshoot production-grade Apache Airflow® DAGs using natural language. By pairing these specialized agent skills with a declarative YAML DSL, all data personas — from analysts to ML engineers — can bypass complex Python Airflow boilerplate. This framework decouples high-level orchestration logic from underlying compute execution, democratizing access to powerful MLOps capabilities across your entire data organization. In this post, we will walk through an exemplary MLOps use case to demonstrate how easily this can be achieved. Setting up your environment Before authoring your first Orchestration Pipeline, you need to set up your local development environment. Getting started takes less than two minutes. 1. Install and configure the extension To install the extension in your preferred IDE or CLI — such as VS Code, VS Code forks, Antigravity, Claude Code, Antigravity CLI, or Codex — and authenticate it with your Google Cloud account, follow the step-by-step setup guide in the official documentation: Google Cloud Data Agent Kit installation guide 2. Verify orchestration pipeline skills Once installed, verify that the required agent skills are active: - Open the ‘Google Cloud Data Agent Kit’ panel on the VS Code activity bar. - Navigate to ‘Settings’ then ‘Skills’. - Ensure the ‘gcp-pipelines-orchestration’ skill is enabled. This skill provides the agent with deep contextual knowledge of pipeline syntax, variable substitution, secret management, and automated incident diagnosis for Airflow runs. 3. Building your first pipeline To start authoring, building, and validating orchestration pipelines directly inside the any VS Code compatible IDE using natural language prompts, follow the official building guide: Build pipelines guide An example business problem: Proactive supply chain management Let’s walk through an example business problem. In the logistics and retail sector, customer satisfaction hinges on accurate delivery estimates. When an order is delayed without warning, customer churn can spike and support costs can escalate. To address this, we are building an end-to-end MLOps architecture that predicts the exact transit time (in days) based on warehouse location, customer location, and order characteristics. By predicting these delays before shipping, operations teams can proactively notify customers or automatically upgrade shipping tiers before Service Level Agreements (SLAs) are breached. To make this architecture fully reproducible, we use the bigquery-public-data.thelook_ecommerce public dataset in BigQuery. For demo purposes, we split this static dataset into training and inference sets. In a real-life scenario, inference would be performed on new, incoming data. This dataset provides authentic operational complexity: - Geographical data: Latitude and longitude for both customer addresses ( users ) and distribution centers (distribution_centers ). - Temporal data: Granular order lifecycle timestamps ( created_at ,shipped_at ,delivered_at ). - Order attributes: Product categories, pricing, and fulfillment status ( orders ,order_items ). By combining this dataset with BigQuery, Managed Service for Apache Spark serverless, Gemini Enterprise Agent Platform, and dbt, we will demonstrate how to build an automated, self-healing MLOps loop that handles training, daily batch inference, and model drift evaluation. The agentic workflow: From prompt to pipeline in minutes With the extension configured, we can bypass boilerplate Python for DAG authoring entirely. Inside VS Code, we opened the Data Agent Kit chat and provided a single natural language prompt to define our continuous MLOps feedback loop: Note: The detailed prompt was crafted with repeatability in mind specifically for this blog post. In real-life scenarios, you can achieve the same result in a more conversational way, pipeline by pipeline. The complete prompt and all generated files are available in the Orchestration-pipelines GitHub repository. Note: While frontier models equipped with the Orchestration Pipelines skill can often scaffold complete workflows in a single step, LLM responses naturally vary based on model versions, workspace context, and token depth. If a specific parameter, dataset path, or dependency is omitted in the initial pass, simply provide a short follow-up prompt. Within minutes, the Data Agent Kit generated the underlying PySpark scripts, dbt configurations, and the three declarative YAML pipelines. Please find below the generated YAML pipelines and a visual diagram of them. This pipeline is a simplified example designed to showcase Orchestration Pipelines capabilities. In practice, recommended production MLOps setups will vary depending on your specific use cases and operational needs. Pipeline 1: The training engine This pipeline serves as our heavy-compute engine. The agent generated a YAML definition that first queries BigQuery to extract historical completed orders. It then dynamically provisions a Managed Spark serverless cluster to calculate geographical distances and train a model for production use. Finally, it pushes the trained model to Gemini Enterprise Agent Platform Model Registry. Pipeline 2: Daily inference For our daily operational workflow, this lightweight pipeline applies the trained model to all currently in-transit orders. It queries the dataset via BigQuery job, executes inference job via Gemini Enterprise Agent Platform, and writes the results back to a BigQuery table to flag potential SLA breaches for the customer support team. Pipeline 3: Automated evaluation and branching The daily evaluation pipeline acts as our automated quality gate. It triggers dbt models to join our predictions with actual delivery timestamps, calculating absolute errors and SLA breaches. Using built-in logic, the pipeline automatically evaluates these metrics. If the model’s error rate exceeds our acceptable threshold, it conditionally triggers the ‘training-pipeline’ to generate a fresh model. Automated deployment to Managed Service for Apache Airflow Authoring pipeline logic is only half the battle; deploying it securely and reliably to production is where data teams historically lose valuable time. With Orchestration Pipelines, deployment is streamlined through standard CI/CD practices. Rather than manually writing deployment scripts or configuring complex environment boundaries, the Data Agent Kit automatically generates the necessary continuous integration workflows (such as GitHub Actions) for your workspace. This means you can simply click commit, and the framework will seamlessly package and deploy your Orchestration Pipeline bundle directly to your Managed Airflow environment. For a comprehensive guide on integrating these automated workflows into your existing CI/CD pipelines, review the official guide: Deploying Orchestration Pipelines. Day-two operations: Monitoring and agentic troubleshooting Maintaining these pipelines is just as intuitive as building them. By bringing the orchestration control plane directly into your IDE, the Data Agent Kit provides real-time monitoring of your Managed Airflow runs without requiring you to constantly context-switch between browser tabs. The Data Agent Kit provides real-time monitoring of your Managed Airflow runs directly within your IDE. The Data Agent Kit visualises the created pipeline. Inevitably, infrastructure or data issues occur—perhaps a Managed Spark cluster hits an out-of-memory exception due to a seasonal data spike, or a BigQuery quota is reached. Resolving these issues no longer requires digging through thousands of lines of raw execution logs. If a pipeline fails, the Data Agent Kit provides out-of-the-box agentic troubleshooting. With the click of a "Troubleshoot" button in your IDE, the Data Engineering Agent analyzes the failure context. It can accurately distinguish between infrastructure quota limits and code-level bugs, instantly providing a root-cause summary and suggesting an inline fix (such as scaling up the compute template). Agentic troubleshooting instantly diagnoses pipeline failures, identifies infrastructure bottlenecks, and suggests inline fixes. Summary: Accelerating time to value Building a resilient MLOps architecture — extracting historical data, executing dbt transformations, provisioning Managed Spark ML compute, integrating Gemini Enterprise Agent Platform for model registry and inference, and configuring cross-DAG conditional triggers — traditionally takes platform engineering teams weeks of writing complex Python Operator logic. With Orchestration Pipelines and the Data Agent Kit, this entire lifecycle was authored, deployed, and easily maintained in a matter of minutes. By replacing boilerplate infrastructure code with a declarative, agent-ready standard, we are ensuring your data organization spends less time orchestrating pipelines and more time delivering tangible business value. Get Started Today: - Review the Orchestration Pipelines documentation. - Install the Data Agent Kit in your preferred IDE or CLI and configure your workspace. - Learn more about the broader ecosystem in our recent blog post: Data Agent Kit brings data skills and tools to your IDE or CLI. - Explore reference architectures in the Data Agent Kit documentation.
document fetched 2026-09-01 02:49 · canonical https://cloud.google.com/blog/products/data-analytics/build-data-pipelines-in-less-time-with-data-agent-kit