Google's PhotoScan Reads Insulin Resistance From a Smartphone Photo

Google Research just published work on PhotoScan, a deep learning pipeline that turns ordinary front and side smartphone photos into clinically meaningful body composition metrics, then uses those metrics to flag insulin resistance. The headline result: on an independent clinical cohort, a demographics plus PhotoScan classifier hit an AUROC of 0.760 for identifying insulin resistant subjects, compared to 0.773 when using gold-standard DXA scans and 0.692 for demographics alone.

Why BMI keeps missing the metabolic story

Insulin resistance predates the clinical onset of type 2 diabetes by years, quietly impairing vascular health, liver function, and energy metabolism long before fasting blood sugar rises into diagnostic ranges. The standard screening signal, BMI, is a blunt instrument because it cannot distinguish visceral fat from subcutaneous fat, or trunk fat from hip fat.

Two ratios do a much better job. The Android-to-Gynoid fat ratio (A/G) compares fat stored in the trunk versus hips and thighs, while the Visceral-to-Subcutaneous fat area ratio (V/S) distinguishes the highly metabolic internal fat around organs from the subcutaneous fat under the skin. Both correlate strongly with insulin resistance. The catch is that measuring them normally requires Dual-Energy X-Ray Absorptiometry (DXA) scans, which are precise but expensive, require specialized clinical infrastructure, and expose patients to low doses of radiation.

The three-stage training recipe

PhotoScan is an investigational framework built around a ResNet-50 image backbone. The team trained it in three phases rather than trying to learn everything from scarce paired photo-to-DXA data:

1. Pre-training on a UK Biobank subset of 35,323 participants that contains both MRI images and DXA body composition ground truth, using a ResNet-50 initialized with ImageNet weights to predict body composition from 2D frontal and lateral projections generated from 3D MRI scans.
2. Fine-tuning on the PhotoBIA cohort of 677 adults with real-world smartphone photos paired with DXA ground truth, using 5-fold cross-validation. An automated landmark detection pipeline picked optimal frontal and lateral pose frames directly from 360-degree participant videos.
3. Independent validation on the MetabolicMosaic cohort of 132 participants from a 30-week longitudinal trial in San Francisco with complete paired DXA, PhotoScan, BIA, anthropometrics, fasting blood labs, and Fitbit tracking.

The model fuses image features with participant sex, height, weight, and BMI through a final dense layer, and outputs probability density functions rather than point estimates for each target metric.

Beating the smartwatch on its home turf

The first interesting result is that a photo-based model outperforms the electrical measurement wearables have relied on for years. On the PhotoBIA cohort, fine-tuned PhotoScan reached a mean absolute error of 2.15 for body fat percentage, while the BIA-based model achieved an MAE of 2.91. On the independent MetabolicMosaic cohort, the errors held steady at 2.13 for BF%, 0.085 for A/G, and 0.085 for V/S.

That consistency between fine-tuning and held-out cohorts matters. Body composition models often fall apart when moved off the training distribution because body shape varies with demographics. Here the model transferred cleanly, with only minor A/G and V/S changes attributable to a higher proportion of female participants in the validation cohort.

Near-DXA insulin resistance prediction

Body composition is only the intermediate output. The clinical target is insulin resistance classification, and this is where the numbers get striking. The team trained a gradient boosting classifier and compared five feature sets. The baseline demographic model (age, sex, BMI) hit an AUROC of 0.692, adding PhotoScan features pushed it to 0.760 with a Net Reclassification Index of 0.593, and the clinical DXA feature set topped out at AUROC 0.773 with NRI 0.748.

For readers unfamiliar with NRI, it quantifies how many people a new model correctly reclassifies compared to a baseline. A value of 0.593 means PhotoScan meaningfully moves subjects between risk categories, not just shuffles probabilities.

The comparison with smartwatch BIA is the most damning part of the results. Adding BIA to demographics yielded no improvement in AUROC or NRI for insulin resistance classification, because BIA only provides body fat percentage estimation, whose feature importance is significantly lower than A/G and V/S ratios in the PhotoScan model. In other words, a total body fat number is not the useful signal. Where the fat sits is.

What this unlocks and what it does not

PhotoScan is explicitly framed as a research prototype, not a product. The authors position it as a middle ground between DXA imaging, which is the most accurate but lacks scalability, and wearable BIA sensors, which are convenient but limited to basic body fat percentage. Practical use cases the work points toward:

• Passive metabolic risk screening on phones people already own, without radiation or clinic visits
• Longitudinal tracking of visceral fat trends during interventions like GLP-1 therapy or lifestyle changes
• Enrichment of wearable data pipelines that already track heart rate, sleep, and activity, but lack a structural body composition signal
• Population studies where DXA scans are cost-prohibitive but insulin resistance prevalence matters

The important caveats: the fine-tuning cohort is 677 people, validation is 132 people from a single San Francisco trial, and the paper is investigational rather than a cleared medical device. Generalization across ethnicities, ages, and clothing conditions is still open. The technical paper is on arXiv if you want the full architecture and evaluation details.

The broader shift

This lands in the middle of a clear trend at Google Research toward passive, phone-based physiological sensing. Recent work on insulin resistance prediction from wearables and continuous heart-rate monitoring from consumer devices points in the same direction: the phone becomes a sensor stack, and the ML pipeline turns raw signals into clinical-grade estimates. The assumption worth updating is that meaningful body composition assessment requires specialized hardware. If a ResNet-50 fine-tuned on a few hundred paired samples can approach DXA-level insulin resistance classification, the bottleneck moves from imaging tech to data collection ethics and regulatory pathways.