Fleet-wide computer vision lifecycle at the edge with SageMaker SDK v3

extract

Artificial Intelligence Fleet-wide computer vision lifecycle at the edge with SageMaker SDK v3 Fleet-wide computer vision model lifecycle management is a problem that shows up the moment a quality inspection proof of concept succeeds. One camera and one model work fine. Then the plant asks for the same result on 40 lines, each with several cameras, and every camera sees the world differently. Each has a different angle, different glare off the same stainless surface, and different ambient light depending on which bay door is open. In production deployments we have worked on, a single model trained on pooled footage underperforms enough individual cameras that each camera ends up with its own fine-tuned model. Now the real problem is no longer training one model well. It’s training, evaluating, versioning, and shipping thousands of them without a human in the loop for each one. This post updates the pattern introduced in the 2023 series Build an end-to-end MLOps pipeline for visual quality inspection at the edge for the current Amazon SageMaker Python SDK v3. It replaces the since-deprecated Amazon SageMaker Edge Manager with Amazon SageMaker Neo for edge optimization. I walk through a dual-branch Amazon SageMaker Pipeline that trains an anomaly detector and a defect classifier per camera, with quality gates that block underperforming models. An in-pipeline SageMaker Neo compilation step ensures every registered model ships with a device-optimized artifact, and an AWS Step Functions orchestration layer paces fleet training. You deploy everything with the AWS Cloud Development Kit (AWS CDK), and a full pipeline run on synthetic data costs about $0.07 in compute. The complete sample is available in the aws-samples GitHub repository. Solution overview The following diagram shows the sample’s fleet-wide model lifecycle. An orchestration layer queues re-training requests and limits concurrent work. A per-camera training pipeline prepares data, trains and evaluates an anomaly detector and a classifier, and produces versioned model artifacts for edge devices. The training half is a single SageMaker Pipeline, parameterized by camera, that runs 13 steps per execution. Three entry steps run in parallel: Preprocess, Fetch Latest Anomaly, and Fetch Latest Classifier. The pipeline preprocesses that camera’s images and fetches the camera’s most recent approved model for each branch, then trains two models. The first is a small convolutional autoencoder that learns what normal product looks like and flags anomalies by reconstruction error. The second is a ResNet18 classifier fine-tuned to confirm the defect. Each branch exports its model to ONNX, evaluates the exported artifact against held-out data, and passes through a quality gate. Past the gate, the model is compiled with SageMaker Neo for the configured edge target, and only then registered in the SageMaker Model Registry. As a result, every approved package has a device-ready compiled artifact. Packages carry the camera ID as metadata, and each branch’s warm-start fetch filters on it, so re-training always resumes from that camera’s own history. The orchestration half paces the fleet. Re-training requests land on an Amazon Simple Queue Service (Amazon SQS) FIFO queue, and an AWS Lambda function starts a Step Functions execution per request. Each execution acquires a slot through a DynamoDB semaphore before starting its pipeline, so a burst of requests drains at a steady rate instead of all launching at once. Completions publish to Amazon Simple Notification Service (Amazon SNS), and operational failures reach the same topic through an Amazon EventBridge rule with the original error preserved. The output of the system is versioned, quality-gated, per-camera model artifacts. The Model Registry holds the portable ONNX, and S3 holds the Neo-compiled build of that model, which is a device-optimized shared library loaded by the DLR runtime rather than ONNX. Both are ready for whatever delivery mechanism your plant uses. Prerequisites To deploy the sample you need the following: - An AWS account with permissions to create AWS Identity and Access Management (IAM) roles. - An Amazon Simple Storage Service (Amazon S3) bucket. - An Amazon Elastic Container Registry (Amazon ECR) repository. - The SageMaker AI, AWS Step Functions, AWS Lambda, Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), Amazon EventBridge, and Amazon DynamoDB resources in the stacks. - Python 3.10, 3.11, or 3.12, Docker. - The AWS CDK CLI version 2.1131.0 or later. Previous CLI versions fail with a cloud assembly schema mismatch against recent versions of aws-cdk-lib. No service quota increase is needed. The orchestration layer admits at most five pipeline executions at a time by default, which keeps concurrent SageMaker training jobs inside the standard account quota. If you deploy to an AWS Region other than your CLI default, set all three of AWS_REGION, AWS_DEFAULT_REGION, and CDK_DEFAULT_REGION on the deploy command itself rather than in your shell profile, for example, AWS_REGION=us-east-2 AWS_DEFAULT_REGION=us-east-2 CDK_DEFAULT_REGION=us-east-2 cdk deploy --all. The CDK CLI, Boto3, and the AWS Command Line Interface (AWS CLI) resolve the Region from overlapping but not identical variable sets, and a mismatch produces confusing partial deployments. Prefixing the command sets all three for that invocation only, so your own default Region is left unchanged. Defining the training pipeline with SDK v3 The code in this section is excerpted from the CDK application in the repository rather than a standalone script, and it runs during cdk deploy, when the stack synthesizes the pipeline definition. Environment setup and the deploy commands come later, under Deploying and running the sample. SDK v3 reorganized the SageMaker Python SDK into modular packages, and the imports show the new shape of the library: Installing sagemaker>=3.0.0,<4 pulls in sagemaker-core, sagemaker-train, sagemaker-serve, and sagemaker-mlops as separate packages. Training configuration that v2 spread across Estimator constructor arguments now lives in typed config objects passed to ModelTrainer. This is the anomaly branch’s trainer, verbatim from the sample: anomalib_input_data wires four channels (train, validation, test, pretrained_model) from upstream step properties, and run_prefix scopes every S3 path to the pipeline execution ID so concurrent fleet runs never collide. Both are in the repository. Two v3 behaviors matter when the pipeline lives inside a CDK application, and neither appears in the changelog as a breaking change. First, SDK v3 validates the pipeline’s execution role against live IAM while building the pipeline definition. This validation happens at synthesis time rather than execution time. A CDK app can’t create the role and the pipeline in one deployment, because the role doesn’t exist yet when the SDK validates it. This is separate from cdk bootstrap, which provisions the CDK toolkit resources once per account and Region and might already be done in your account. The sample’s own two-phase deploy is driven by a context flag that happens to share that name. An initial pass with -c bootstrap=true creates the role, bucket, and ECR repository, and the second pass synthesizes the pipeline against the now-existing role. The role Amazon Resource Name (ARN) embedded in the pipeline definition must also be a concrete string, not an unresolved CDK token. Second, the v3 role validator simulates ECR pull permissions without a resource ARN, so an execution role scoped to a single ECR repository fails validation even though it would work at runtime. The sample’s role carries a broader ECR grant for this reason. Lastly, one v3 habit the sample uses: constructing the pipeline with PipelineSession(default_bucket=your_bucket) routes SDK-managed uploads, such as step code, into the pipeline’s own bucket, so every artifact lives in one place and the execution role’s permissions cover a single bucket. (Granting the role access to the SDK’s default bucket works too. This approach is tidier.) Quality gates that match how defect data behaves Every trained model is evaluated before registration, and the evaluation runs against the exported ONNX artifact rather than the in-memory PyTorch model. That distinction matters: the artifact you gate is the artifact you ship, so an export problem can never slip a broken model past a healthy-looking training run. The anomaly branch gates on AUROC of at least 0.8. The classifier branch gates on two metrics at once, and the choice of metrics is shaped by the defining property of defect data: heavy class imbalance. In this sample roughly 90 percent of images are good, which is realistic for a production line. Against that distribution, plain accuracy is meaningless, because a model that predicts “good” for every image scores about 0.9 while catching zero defects. Balanced accuracy, the mean of per-class recall, corrects for that, but it has its own blind spot. A model with perfect recall on good parts and only 20 percent recall on defects still scores (1.0 + 0.2) / 2 = 0.6. So the gate requires balanced accuracy of at least 0.6 and defect recall of at least 0.5, together: A ConditionStep ANDs its conditions, so both must pass. The thresholds are pipeline parameters, adjustable per execution without redeploying. If a gate fails, a FailStep marks the execution failed and that branch’s model never reaches the registry (the two branches gate independently, so the other branch’s model may still register). The fleet keeps serving the previous approved model for that camera. The training code addresses the same imbalance from the other side. The classifier is a fine-tuned resnet18 with ImageNet weights and a two-class head. At per-camera dataset sizes, fine-tuning pretrained weights is what keeps the model from collapsing onto the majority class. The training code balances training batches with a WeightedRandomSampler using inverse class frequencies, while the loss stays unweighted. Combining a weighted loss with weighted sampling over-corrects toward the minority class. Compiling for the edge with SageMaker Neo, inside the pipeline Notice the gate’s if_steps in the preceding code: past the gate, compilation runs first and registration depends on it. Both training scripts export ONNX with dynamic batch axes. The quality gates score those exported artifacts under ONNX Runtime, and then a CompileAnomalibNeo or CompileClassifierNeo step hands the gated artifact to SageMaker Neo. The invariant is gate, then compile, then register. A model only becomes an Approved package if it also compiled for the fleet’s edge target, so the registry never advertises a model the devices cannot run. The compile step encapsulates three error-prone details. Neo requires the input tarball to contain only the model file, but training jobs emit model.tar.gz with the checkpoint and metrics alongside the ONNX model. The step repackages an ONNX-only tarball first. Neo also requires concrete input shapes while the exports declare a dynamic batch axis, so batch size 1 is pinned in the compilation request. And because pipeline steps can retry, the compilation job name derives deterministically from the execution ID, so a retry resumes polling the job it already started instead of orphaning it. The target is a pipeline parameter: NeoTargetDevice selects a named device such as jetson_xavier or rasp4b, and its default compiles for the NeoTargetOs/NeoTargetArch platform, LINUX/X86_64 by default. The compiled package contains the optimized compiled.so plus the DLR runtime for loading it. The default output runs on any x86_64 Linux host, so you can validate it before committing to device-specific targets. Compilation itself incurs no charge. You pay for the processing instance that drives it and the S3 storage of the outputs. For compiling an already-registered model for additional device types later, without re-training, the repository includes an ad-hoc script that runs the same flow against the Model Registry. Per-camera models in the Model Registry Each branch registers into its own model package group, and every package carries the camera ID and its evaluation metrics. From the sample, with the metrics wiring elided: Camera identity in customer_metadata_properties is what turns two shared package groups into a fleet catalog. Packages from all cameras live in the same groups, and per-camera history is recovered by filtering on the metadata. Each branch has a fetch step that pages through approved packages newest-first until it finds the requesting camera’s latest model, because ListModelPackages cannot filter on customer metadata server-side. On a camera’s first run the fetch finds nothing and training falls back to its cold-start initialization (ImageNet weights for the classifier, random initialization for the autoencoder). Every later run resumes from that camera’s own prior model, which is what makes frequent, cheap, incremental re-training viable across a fleet. Note what these registry entries are and are not: they catalog versioned, quality-gated artifacts, but their container reference is the training image, which serves no inference endpoints. They exist for a fleet workflow where a downstream system pulls artifacts and ships them to devices, not for CreateEndpoint. Orchestrating fleet training requests Every AWS account has service quotas on concurrent SageMaker training jobs, and each pipeline execution here launches several jobs, so fleet re-training needs pacing. Something has to keep a burst of 500 requests from becoming 500 simultaneous pipeline launches. The sample uses a semaphore, a counter of available slots that works like a parking lot with five spaces. Before the Step Functions state machine starts a pipeline, it takes a slot by atomically incrementing a counter in DynamoDB, with a condition that refuses the increment once the cap is reached: If all slots are taken, the execution waits and retries. When its pipeline finishes, it decrements the counter to release the slot. One detail is worth carrying into your own designs. The DynamoDB condition expression enforces the cap in a single atomic operation, rather than the state machine reading the counter and deciding, which would leave a race window between the read and the write. The repository implements the rest of the operational hardening, including releasing slots on every failure path and forwarding failed executions to Amazon SNS through Amazon EventBridge. Its README covers sizing the wait bounds. The cap of five is a stack parameter, set low so the sample deploys and runs in any standard account without requesting quota increases. Adjust it to whatever your account’s training-job quotas support. Past a single account’s quotas, scaling fleet training is its own topic. The standard levers are raising the service quotas, spreading training across instance types or Regions, and distributing the fleet across multiple accounts. You can also hold the backlog in SQS and admit work only as capacity frees up. Those choices depend on your fleet’s size and shape. The sample gives you the working skeleton to extend. Deploying and running the sample The walkthrough in the repository README is the authoritative version. The following short form starts from a clean checkout and sets up the environment before the first deploy: By default the pipeline generates synthetic production line images, so you can validate the whole system end to end with no dataset and no data approvals. A full run executes all 13 steps, including both Neo compilations, in about 15 minutes and costs roughly $0.07 in billed ml.m5.xlarge time (US East (Ohio) Region (us-east-2), July 2026, on-demand). The orchestration resources together cost well under $0.10 per month at sample scale. To exercise the pipeline on real images, the repository includes a downloader for the BCCD dataset. The dataset is a collection of 364 annotated blood smear microscopy images originally open sourced on GitHub and reorganized into Pascal VOC format under an MIT license. The same dataset is also distributed by Roboflow as public domain. The script maps platelet presence to the defect class, optionally splits the images per camera to simulate a fleet, and uploads training tarballs to the pipeline bucket. The accompanying notebook runs the full pipeline on the real data and does side-by-side ONNX inference on a normal and an anomalous sample. Treating platelets as defects is an illustrative stand-in for a manufacturing defect distribution, not a clinical claim. Cleaning up Model packages are created at pipeline runtime, so AWS CloudFormation cannot delete them. Remove them with the AWS CLI first, then destroy the stacks with cdk destroy --all -c bootstrap=true --force. The README lists the exact commands. Conclusion In this post, I showed a working SageMaker SDK v3 reference architecture for the fleet-scale computer vision lifecycle. The architecture includes a dual-branch training pipeline where every model is quality-gated on its exported ONNX artifact, compiled with SageMaker Neo, and only then registered. A Model Registry layout uses camera metadata to turn shared package groups into per-camera warm-startable history, and a Step Functions orchestration layer paces fleet training within account quotas. For teams whose needs are met by managed options such as Amazon Lookout for Vision, that is the simpler path. This pattern targets the per-camera, fleet-scale constraint that managed options do not cover. The design rules transfer beyond computer vision: gate on balanced accuracy plus minority-class recall rather than accuracy, evaluate and compile the artifact you ship, and make operational failures notify rather than terminate silently. The sample deploys in minutes and a full validation run costs about seven cents of compute, so a low-cost next step is to clone the repository and run it. From there, point the BCCD downloader at a real bucket, set NeoTargetDevice for your hardware, or size the orchestration bounds to your fleet. For the SDK v3 migration itself, the SageMaker Python SDK documentation and the two-part series announcing the new SDK are the references I kept open throughout. If you want help adapting this pattern to your production environment, reach out to your AWS account team.