Error while exporting hugging face model on macbook

[arjunsingri]

optimum-cli export executorch --model "PrunaAI/zackli4ai-llama3.2-1b-train-aware-quantization-bnb-8bit-smashed" --task "text-generation" --recipe "xnnpack" --output_dir="meta_llama3_2_1b"
Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.
Could not load bitsandbytes native library: dlopen(/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib, 0x0006): tried: '/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file), '/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file)
Traceback (most recent call last):
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/cextension.py", line 85, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/cextension.py", line 72, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/ctypes/__init__.py", line 460, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: dlopen(/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib, 0x0006): tried: '/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file), '/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/bitsandbytes/libbitsandbytes_cpu.dylib' (no such file)
CUDA is required but not available for bitsandbytes. Please consider installing the multi-platform enabled version of bitsandbytes, which is currently a work in progress. Please check currently supported platforms and installation instructions at https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backend
Traceback (most recent call last):
  File "/opt/anaconda3/envs/optimum_custom2/bin/optimum-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/optimum/commands/optimum_cli.py", line 208, in main
    service.run()
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/optimum/commands/export/executorch.py", line 66, in run
    main_export(
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/optimum/exporters/executorch/__main__.py", line 123, in main_export
    model = task_func(model_name_or_path, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/optimum/exporters/executorch/tasks/causal_lm.py", line 54, in load_causal_lm_model
    return AutoModelForCausalLM.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/transformers/modeling_utils.py", line 3620, in from_pretrained
    hf_quantizer.validate_environment(
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/transformers/quantizers/quantizer_bnb_8bit.py", line 81, in validate_environment
    validate_bnb_backend_availability(raise_exception=True)
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/transformers/integrations/bitsandbytes.py", line 559, in validate_bnb_backend_availability
    return _validate_bnb_cuda_backend_availability(raise_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/optimum_custom2/lib/python3.12/site-packages/transformers/integrations/bitsandbytes.py", line 537, in _validate_bnb_cuda_backend_availability
    raise RuntimeError(log_msg)
RuntimeError: CUDA is required but not available for bitsandbytes. Please consider installing the multi-platform enabled version of bitsandbytes, which is currently a work in progress. Please check currently supported platforms and installation instructions at https://huggingface.co/docs/bitsandbytes/main/en/installation#multi-backend

[guangy10]

By looking at the stacktrace, it's not even reaching to the export stage. Is your model suppose to use CUDA always?

[arjunsingri]

Is CUDA usage encoded in the model? I thought we should be able to specify whether we want to use CUDA or not during the export. Please correct me if I am wrong.

[guangy10]

No the export process will be running on cpu only. The stacktrace shows you haven’t reach to the export step yet. The installed pytorch is a cpu version. So if you try to use the cuda via torch it will fail.

[arjunsingri]

I didn't explicitly try to use cuda. I just ran the command I have pasted above. Is there a way I can explicitly say "don't use cuda"?