CoreML conversion?

[ckse93]

Hi, I am a developer, working on real time 3d conversion of a video, I was wondering if there is a coreML version of this model somewhere. having 9ms compute time is insane, and it is a lot more than enough to process real time 3d conversion. I am unfortunately very novice in this line of work and the learning curve is extensive. If you can release a model that can process a frame by frame, I would be incredibly grateful! Thank you for your amazing work

[jvcleave]

I was able to get it to build as an .mlpackage but for me it is a lot slower than DepthAnythingv2's mlpackage. This is the first one I have ever converted so it may be missing something critical. It doesn't have the convenience methods to convert from CVPixelBuffer like DepthAnything so that may be slowing me down too

Anyway, in case it helps here is what I did on a M1 Max to get it to build:

Searched for and changed all align_corners=True → align_corners=False

Replaced mode="bicubic" with mode="bilinear" somewhere

This allowed it to compile using this convert_to_coreml.py script

import os
import torch
import coremltools as ct
import logging
logging.basicConfig(level=logging.DEBUG)

from coremltools.converters.mil import mil



from video_depth_anything.video_depth import VideoDepthAnything

# Load the full model (ViT-S + Decoder)
model = VideoDepthAnything(
    encoder="vits",
    features=64,
    out_channels=[48, 96, 192, 384],
)

# Load pre-trained checkpoint for small model
ckpt = torch.load("checkpoints/video_depth_anything_vits.pth", map_location="cpu")
model.load_state_dict(ckpt, strict=True)
model.eval()

# Dummy input: [B, T, C, H, W] → [1, 1, 3, 512, 512]
dummy_input = torch.rand(1, 1, 3, 518, 518)

# Trace the entire model
print("🔍 Tracing full VideoDepthAnything model...")
traced = torch.jit.trace(model, dummy_input)

# Convert to Core ML
print("Converting to Core ML...")
mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(name="input", shape=dummy_input.shape)],
    convert_to="mlprogram",
    compute_units=ct.ComputeUnit.CPU_AND_GPU
)


# Save to .mlpackage
mlmodel.save("VideoDepthAnything_Full_ALL.mlpackage")
print("Done! Model saved.")

adding this reference as it has more detail https://github.com/huggingface/coreml-examples/blob/main/tutorials/depth-anything-coreml-guide.ipynb