Wound care · Vision models on SageMaker
SkinScan
AI wound-care imaging, end to end: one DINOv3 model for classification and segmentation, cm²-accurate measurement, Bedrock agents for staging and recommendations, and a Next.js clinician portal.
- Role
- Senior Software Engineer, Cloud & AI · model training and deployment through to the iOS capture app
- Organization
- Virtual Health Hub
- Timeframe
- 2024 – 2026
- Swift (iOS)
- Next.js
- Node.js / Express
- PyTorch
- Hugging Face Transformers
- DINOv3 ViT-L/16
- AprilTag
- Amazon SageMaker
- Amazon Bedrock Agents
- Amazon Rekognition
- Amazon Comprehend Medical
- Amazon Cognito
Context
Wound care runs on repeat assessment: what kind of wound, how large it is, and whether it is changing. Phone photos make that hard. A handheld photo has no scale, and distance, angle, and lighting shift every visit, so two photos of one wound rarely agree. The scope was the whole path, not a model: capture on a phone, classify and segment, measure in real units, propose staging and recommendations, and hand structured results to a clinician. Three constraints are visible in the design. Wound images and notes are PHI. A ViT-L-class model needs real compute, and idle compute is pure cost. The available clinical data was severely imbalanced across wound types; a single source would teach the wrong priors.
The build
One model, two outputs.
A unified DINOv3 ViT-L/16, trained with PyTorch and Hugging Face Transformers, returns 10-class wound classification and pixel-level segmentation in a single forward pass. One backbone means one artifact to version and one pass per image.
An 11-source clinical dataset.
Eleven clinical sources were curated and mapped to one 10-class taxonomy to eliminate severe class imbalance. Transfer learning from a self-supervised backbone carries a lot, but it cannot learn a class the training set barely contains.
SageMaker async endpoints that scale to zero.
The model runs behind Amazon SageMaker asynchronous inference with autoscaling down to zero instances. Requests queue, outputs land in S3, and no instance runs while nothing is waiting. A wound capture can tolerate a short wait; an always-on instance cannot be made cheap.
AprilTag calibration for cm² measurement.
An AprilTag of known physical size sits in frame. Detecting it gives the pixel-to-centimetre scale for cm²-accurate measurements. Without a reference object, a phone photo has no scale and wound area is a guess.
Swift iOS capture app with AR guidance and Rekognition pre-screening.
A native Swift app guides capture with AR. Amazon Rekognition pre-screens each capture, so poor inputs are caught before an inference job is queued.
3 Bedrock agents plus Comprehend Medical.
Three Amazon Bedrock agents handle wound staging and recommendations. Amazon Comprehend Medical extracts ICD-10-CM codes, RxNorm medication concepts, and PHI entities from clinical text. The vision model classifies and segments; the agents handle staging and recommendations; Comprehend Medical structures the clinical text.
Node/Express backend with Cognito RBAC and SSE; Next.js clinician portal.
Amazon Cognito handles identity and the Express layer enforces role-based access, so PHI is scoped by role. The backend streams results over server-sent events as they arrive.
Outcome
10
wound classes, one model
11
clinical sources curated into one dataset
1
forward pass for classification and segmentation
3
Bedrock agents for staging and recommendations
- One deployed model covers 10 wound classes with classification and segmentation in a single pass, and AprilTag calibration gives cm²-accurate measurement.
- Severe class imbalance in the training data was eliminated by curating an 11-source clinical dataset into one taxonomy.
- Inference runs on SageMaker async endpoints that scale to zero when idle, so compute follows capture volume instead of running around the clock.
- The founder trained the model and built the platform end to end: iOS capture app, Node/Express backend, Next.js clinician portal, 3 Bedrock agents, Comprehend Medical coding, and SageMaker deployment.
Engineering notes
The unified model is a trade
One shared ViT-L/16 backbone with a classification head and a segmentation head means one pass per image and one artifact to version, at the cost of coupling the two tasks to one backbone. The data work was substantial. Eleven clinical sources were curated into one dataset and mapped to one 10-class taxonomy to eliminate severe class imbalance.
SageMaker serves the model
Asynchronous endpoints fit the workload: requests queue, outputs land in S3, and the endpoint scales to zero instances when the queue is empty. The cost of that is cold start. The first capture after an idle period waits for an instance to come up. That is acceptable for a clinical review flow and would not be for a chat box; results are streamed over SSE when they land. Rekognition pre-screening sits in front as a cheap gate, so inference time is spent on usable images.
Wound images and clinical notes are PHI
Cognito handles identity and the Express layer enforces RBAC, so access is scoped by role, not by endpoint. Comprehend Medical detects PHI entities alongside its ICD-10-CM and RxNorm extraction. The platform was built HIPAA-aligned. Aligned, not certified, is the accurate word for it. The three Bedrock agents produce staging and recommendations.
Relevant if
A vision model from dataset to deployed endpoint. Multi-source dataset curation to fix class imbalance, DINOv3 ViT transfer learning with classification and segmentation in one forward pass, and SageMaker async inference that scales to zero when idle.
Clinical AI on images and notes that contain PHI. Bedrock agents for staging and recommendations, and Comprehend Medical for ICD-10-CM, RxNorm, and PHI, behind Cognito RBAC, built HIPAA-aligned.
Real measurements from a phone camera. AprilTag calibration for cm²-accurate measurements, AR-guided Swift capture, and Amazon Rekognition pre-screening of each capture.