[RFC] Support load converted model from hub

[guangy10]

Feature request: Support load converted model from hub

Define the use-case

The fundamental use case behind is to cache the converted model and allow users to fetch it efficiently w/o always going through the exporting process.

Preferred path

During the trip to Paris last year, we briefly mentioned the idea of creating an Executorch Community and built a prototype of it with help from you guys. Here it is: https://huggingface.co/executorch-community. Ideally we would like host the .pte model, its configuration, and a snapshot of the recipe in the Executorch Community. Do you think it's a more appropriate place to scale?

Process

What is the process for maintaining the generated artifacts? It's open to discuss but I think GGUF and ONNX already have similar features, so I'm curious if anything we can learn, particular around the life cycle of the generated artifacts:

• when is it triggered in the first place?
• how is it updated or override?
• how is the compatibility (backward and forward) policy of consuming the generated artifacts enforced?

[guangy10]

@echarlaix is already publishing a PR to support this here: #13

I'm creating this GitHub Issue dedicated for more in-depth discussion around the process and where to land the artifacts

[guangy10]

cc: @byjlw @mergennachin @GregoryComer

[byjlw]

This is what i was thinking for format for how the models would be organized and named. what do you think?

Guidelines

Model Cards

• Each high level item will target a particular model and target device.
• Model card will include:
  • Boiler Plate Details on ET
  • Link to the original model
  • Recipe for converting the original model to all the quantized versions
  • Code snip-it on how to use the model
  • Prompt format
  • Table for all models within this repo with size and expected results along as a recommendation on which one to use:
    • FileName
    • QuantType
    • Size
    • Description
  • Besides .pte file, Corresponding .pt2 file and its recipe (Future)

Files

• Files will include:
  • .gitattributes
  • .pte files for all the quantized versions of the model
  • config.json
  • export.ipynb
  • README.md
  • params.json
  • tokenizer.model

Repo Naming

Names should include model, number of parameters and target device
{Model}-{Params}-{target-device}
Example: executorch-community/Llama-3.2-1B-Instruct-ARM-CPU

Model File Naming

{Model}-{Params}-{quantscheme}-{bitsize}.pte
Example: llama-3.2-1B-SQ-4-8.pte

[guangy10]

This is what i was thinking for format for how the models would be organized and named. what do you think?

@byjlw Thanks for sharing. It looks good to me about the naming and file organization. Do you have particular thoughts on the user experience that you would expect to enable by adding support to this request?

[byjlw]

to these questions?
when is it triggered in the first place? First time this model+target device gets created. If we are using the naming i proposed above then we can check to see if it exists already. If it does we do nothing, if it doesn't we create it. The challenge is partials. Like we only do the model for a single quant when there are 5 possible. See the second part
how is it updated or override? Things should be immutable more or less. The only real reason to change things is if there was a bug in the export. Model cards could get updated in instances where we add support to another quant and we need to upload just that quantized model. In this case nothing is overwritten. The model card gets updated and another file gets added to the repository
how is the compatibility (backward and forward) policy of consuming the generated artifacts enforced? Do we need this? I'm curious in what situations this would break.

[guangy10]

to these questions? when is it triggered in the first place? First time this model+target device gets created. If we are using the naming i proposed above then we can check to see if it exists already. If it does we do nothing, if it doesn't we create it. The challenge is partials. Like we only do the model for a single quant when there are 5 possible. See the second part how is it updated or override? Things should be immutable more or less. The only real reason to change things is if there was a bug in the export. Model cards could get updated in instances where we add support to another quant and we need to upload just that quantized model. In this case nothing is overwritten. The model card gets updated and another file gets added to the repository

@byjlw I'd like to have the discussion to be concrete about how the API should behave. For example, if you look at the PR #13 @echarlaix created, the API from_pretrained() will load a cached model if 1) the model_id is passed 2) the export flag is set False.
Now the question needs to be clarified is: Do we expect to always publish new or override if exists with the following call, e.g. from CI or invoked by users:

model = ExecuTorchModelForCausalLM.from_pretrained(model_id, export=True, recipe=recipe, task=task)

Likely no IMO, then we need to clarify how/when it should be published. To me it can be easily managed by an additional flag, but would like to know how it works for other backends. I think @echarlaix can help explain.

how is the compatibility (backward and forward) policy of consuming the generated artifacts enforced? Do we need this? I'm curious in what situations this would break.

@byjlw When the artifacts is published, and users download it directly and try to run inference with it, they will face BC/FC. Without stating it clearly, users may hit it as a surprise. For example, when one is trying to run a model from 6 months ago on a runtime built on latest ET trunk. I know we already have a policy for the API, but I can't recall if we have one for the pte file. cc: @mergennachin

The basic I can propose is to attach the BC/FC policy as a general guidance if we agree on this.
Advanced support of course will have a better UX if there is a mechanism to enforce the policy automatically

[echarlaix]

@byjlw I'd like to have the discussion to be concrete about how the API should behave. For example, if you look at the PR #13 @echarlaix created, the API from_pretrained() will load a cached model if 1) the model_id is passed 2) the export flag is set False.
Now the question needs to be clarified is: Do we expect to always publish new or override if exists with the following call, e.g. from CI or invoked by users:

model = ExecuTorchModelForCausalLM.from_pretrained(model_id, export=True, recipe=recipe, task=task)

Likely no IMO, then we need to clarify how/when it should be published. To me it can be easily managed by an additional flag, but would like to know how it works for other backends. I think @echarlaix can help explain.

We shouldn't need to provide the task since #12, same for export as we could directly check whether a .pte file a is present no ? Added something in #13 let me know if that works for you

[guangy10]

@byjlw I'd like to have the discussion to be concrete about how the API should behave. For example, if you look at the PR #13 @echarlaix created, the API from_pretrained() will load a cached model if 1) the model_id is passed 2) the export flag is set False.
Now the question needs to be clarified is: Do we expect to always publish new or override if exists with the following call, e.g. from CI or invoked by users:

model = ExecuTorchModelForCausalLM.from_pretrained(model_id, export=True, recipe=recipe, task=task)

Likely no IMO, then we need to clarify how/when it should be published. To me it can be easily managed by an additional flag, but would like to know how it works for other backends. I think @echarlaix can help explain.

We shouldn't need to provide the task since #12, same for export as we could directly check whether a .pte file a is present no ? Added something in #13 let me know if that works for you

Sounds good. @echarlaix wdyt about publishing the converted model to executorch-community and load from there? any other comment about this proposal?

[echarlaix]

Sounds good. @echarlaix wdyt about publishing the converted model to executorch-community and load from there? any other comment about this proposal?

For that what do you think about creating a space as done for ONNX : https://huggingface.co/spaces/onnx-community/convert-to-onnx or https://huggingface.co/spaces/onnx/export. This space could directly push the exported model under the executorch-community org, an other option could be to push directly the resulting model under the user namespace (so for example if I'm using the space, the model will be pushed under https://huggingface.co/echarlaix, this can be useful for cases where many different options exist as for example for nncf quantization https://huggingface.co/spaces/OpenVINO/nncf-quantization)