Description
Checks
- This template is only for bug reports, usage problems go with 'Help Wanted'.
- I have thoroughly reviewed the project documentation but couldn't find information to solve my problem.
- I have searched for existing issues, including closed ones, and couldn't find a solution.
- I am using English to submit this issue to facilitate community communication.
Environment Details
docker pull nvcr.io/nvidia/tritonserver:24.12-py3，torch=2.5.1 trtllm=0.16.0，RTX4090
Steps to Reproduce
root@3210a7dbf083:/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm# bash run.sh 5 5
运行到这个步骤报错下面的，错误，，能帮忙看下什么问题吗？
✔️ Expected Behavior
3_spk.wav', 'array': array([-3.05175781e-05, 0.00000000e+00, -6.10351562e-05, ...,
-5.79833984e-04, -5.18798828e-04, -6.10351562e-04]), 'sampling_rate': 16000}, 'reference_text': '而在新闻领域的奇葩说目前仍空缺。', 'target_audio_path': 'unseen3_text6', 'target_text': '哎，我现在兜里只剩八十块钱了，咱们俩就随便看一下，标准件就完了。'}, {'audio_filepath': {'path': 'unseen4_spk.wav', 'array': array([-1.22070312e-04, -9.15527344e-05, 0.00000000e+00, ...,
0.00000000e+00, 3.96728516e-04, 5.18798828e-04]), 'sampling_rate': 16000}, 'reference_text': '在夏日阴凉的树荫下，鸭妈妈孵着鸭宝宝。', 'target_audio_path': 'unseen4_text7', 'target_text': '按照科学的知识来说这是因为蓝色更加容易散射哦，宝宝以后就会学到的哦。'}, {'audio_filepath': {'path': 'unseen4_spk.wav', 'array': array([-1.22070312e-04, -9.15527344e-05, 0.00000000e+00, ...,
0.00000000e+00, 3.96728516e-04, 5.18798828e-04]), 'sampling_rate': 16000}, 'reference_text': '在夏日阴凉的树荫下，鸭妈妈孵着鸭宝宝。', 'target_audio_path': 'unseen4_text8', 'target_text': '他一度梦想参军，但是因没身份而作罢，整日在村内瞎晃。'}, {'audio_filepath': {'path': 'unseen5_spk.wav', 'array': array([0.00021362, 0.00045776, 0.00045776, ..., 0.00079346, 0.00054932,
0.00057983]), 'sampling_rate': 16000}, 'reference_text': '在夏日阴凉的树荫下，鸭妈妈孵着鸭宝宝。', 'target_audio_path': 'unseen5_text9', 'target_text': '哎，不是有一句话这样说嘛，你永远不知道明天和意外哪个先来。'}, {'audio_filepath': {'path': 'unseen5_spk.wav', 'array': array([0.00021362, 0.00045776, 0.00045776, ..., 0.00079346, 0.00054932,
0.00057983]), 'sampling_rate': 16000}, 'reference_text': '在夏日阴凉的树荫下，鸭妈妈孵着鸭宝宝。', 'target_audio_path': 'unseen5_text10', 'target_text': '当时有点感动还是什么？'}, {'audio_filepath': {'path': 'unseen6_spk.wav', 'array': array([ 1.83105469e-04, 1.83105469e-04, 3.35693359e-04, ...,
0.00000000e+00, -3.05175781e-05, 3.05175781e-05]), 'sampling_rate': 16000}, 'reference_text': '负责指挥的将军在一旁交代着注意事项，每 个人在上面最多只能待九十秒，九十秒之后，听到铃响，必须立即返回。', 'target_audio_path': 'unseen6_text12', 'target_text': '派大军前去争抢，赫赫 查拉二话不说，喊出了巴顿的名字，结果手套开始传递给斯科特半路遭到灭霸拦截，真正的杀神从天而降。'}, {'audio_filepath': {'path': 'unseen6_spk.wav', 'array': array([ 1.83105469e-04, 1.83105469e-04, 3.35693359e-04, ...,
0.00000000e+00, -3.05175781e-05, 3.05175781e-05]), 'sampling_rate': 16000}, 'reference_text': '负责指挥的将军在一旁交代着注意事项，每 个人在上面最多只能待九十秒，九十秒之后，听到铃响，必须立即返回。', 'target_audio_path': 'unseen6_text11', 'target_text': '这些飞机仍然由位于华 盛顿州埃弗里特的工厂生产。'}, {'audio_filepath': {'path': 'unseen7_spk.wav', 'array': array([-3.05175781e-05, -1.22070312e-04, -1.22070312e-04, ...,
-3.05175781e-04, -2.74658203e-04, -6.10351562e-05]), 'sampling_rate': 16000}, 'reference_text': '然后大家看书的时候就发现，那些什么心灵 鸡汤啊，心灵鸡血啊，还有什么毒鸡汤啊，还挺管用的。', 'target_audio_path': 'unseen7_text14', 'target_text': '而中国却只是将其作为红旗反导系统下 的一个补充而已。'}, {'audio_filepath': {'path': 'unseen7_spk.wav', 'array': array([-3.05175781e-05, -1.22070312e-04, -1.22070312e-04, ...,
-3.05175781e-04, -2.74658203e-04, -6.10351562e-05]), 'sampling_rate': 16000}, 'reference_text': '然后大家看书的时候就发现，那些什么心灵 鸡汤啊，心灵鸡血啊，还有什么毒鸡汤啊，还挺管用的。', 'target_audio_path': 'unseen7_text13', 'target_text': '以后我们大概不会再见面了。'}, {'audio_filepath': {'path': 'unseen8_spk.wav', 'array': array([-6.10351562e-05, -1.52587891e-04, -2.74658203e-04, ...,
-3.05175781e-05, 3.05175781e-05, -3.05175781e-05]), 'sampling_rate': 16000}, 'reference_text': '空喊不得不再找到卖西瓜的老大爷，老大爷 告诉空喊，光不可能凭空变出来。', 'target_audio_path': 'unseen8_text15', 'target_text': '这一段戏同样也表达了亚瑟说的，我原本以为我的人生是一出 悲剧，但其实它是一出戏剧。'}, {'audio_filepath': {'path': 'unseen8_spk.wav', 'array': array([-6.10351562e-05, -1.52587891e-04, -2.74658203e-04, ...,
-3.05175781e-05, 3.05175781e-05, -3.05175781e-05]), 'sampling_rate': 16000}, 'reference_text': '空喊不得不再找到卖西瓜的老大爷，老大爷 告诉空喊，光不可能凭空变出来。', 'target_audio_path': 'unseen8_text16', 'target_text': '常常暴跳如雷，红着眼睛用力攥着洛洛的手，把洛洛的手攥得 生疼也不撒手。洛洛吓坏了，她很害怕，那个平日里对她那么好的男友为什么会变得这么可怕？'}, {'audio_filepath': {'path': 'unseen9_spk.wav', 'array': array([ 3.05175781e-05, -3.05175781e-05, -3.05175781e-05, ...,
0.00000000e+00, 3.05175781e-05, -3.05175781e-05]), 'sampling_rate': 16000}, 'reference_text': '布鲁斯特在赛后举办了一个盛大的派对，以 感谢朋友们一个月以来的陪伴。', 'target_audio_path': 'unseen9_text17', 'target_text': '或者看一下他的状态。'}, {'audio_filepath': {'path': 'unseen9_spk.wav', 'array': array([ 3.05175781e-05, -3.05175781e-05, -3.05175781e-05, ...,
0.00000000e+00, 3.05175781e-05, -3.05175781e-05]), 'sampling_rate': 16000}, 'reference_text': '布鲁斯特在赛后举办了一个盛大的派对，以 感谢朋友们一个月以来的陪伴。', 'target_audio_path': 'unseen9_text13', 'target_text': '以后我们大概不会再见面了。'}]
task-0: 0/26
Traceback (most recent call last):
File "/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/client_grpc.py", line 470, in
asyncio.run(main())
File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/client_grpc.py", line 425, in main
ans_list = await asyncio.gather(*tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/client_grpc.py", line 304, in send
response = await triton_client.infer(model_name, inputs, request_id=str(sequence_id), outputs=outputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tritonclient/grpc/aio/init.py", line 686, in infer
raise_error_grpc(rpc_error)
File "/usr/local/lib/python3.12/dist-packages/tritonclient/grpc/_utils.py", line 77, in raise_error_grpc
raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] Failed to process the request(s) for model 'f5_tts_0_0', message: TritonModelException: Failed to open the cudaIpcHandle. error: invalid resource handle
At:
/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(168): forward_vocoder
/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(268): execute
root@3210a7dbf083:/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm# bash run.sh 6 6
Model is none, using default model F5TTS_Base
Start stage: 6, Stop stage: 6, Model: F5TTS_Base
Testing http client
Traceback (most recent call last):
File "/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/client_http.py", line 141, in
audio = result["outputs"][0]["data"]
~~~~~~^^^^^^^^^^^
KeyError: 'outputs'
root@3210a7dbf083:/workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm#
triton_tirllm
E0629 04:58:23.089858 1958 pb_stub.cc:714] "Failed to process the request(s) for model 'f5_tts_0_0', message: RuntimeError: CUDA error: invalid resource handle\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1\nCompile with TORCH_USE_CUDA_DSA to enable device-side assertions.\n\n\nAt:\n /usr/local/lib/python3.12/dist-packages/torch/nn/functional.py(5096): pad\n /usr/local/lib/python3.12/dist-packages/torch/functional.py(702): stft\n /usr/local/lib/python3.12/dist-packages/torchaudio/functional/functional.py(126): spectrogram\n /usr/local/lib/python3.12/dist-packages/torchaudio/transforms/_transforms.py(110): forward\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1747): _call_impl\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1736): _wrapped_call_impl\n /usr/local/lib/python3.12/dist-packages/torchaudio/transforms/_transforms.py(619): forward\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1747): _call_impl\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1736): _wrapped_call_impl\n /workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(157): get_vocos_mel_spectrogram\n /workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(216): execute\n"
E0629 04:59:08.475925 1958 pb_stub.cc:714] "Failed to process the request(s) for model 'f5_tts_0_0', message: TritonModelException: Failed to open the cudaIpcHandle. error: invalid resource handle\n\nAt:\n /workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(168): forward_vocoder\n /workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(268): execute\n"
E0629 05:00:16.182553 1958 pb_stub.cc:714] "Failed to process the request(s) for model 'f5_tts_0_0', message: RuntimeError: CUDA error: invalid resource handle\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1\nCompile with TORCH_USE_CUDA_DSA to enable device-side assertions.\n\n\nAt:\n /usr/local/lib/python3.12/dist-packages/torch/nn/functional.py(5096): pad\n /usr/local/lib/python3.12/dist-packages/torch/functional.py(702): stft\n /usr/local/lib/python3.12/dist-packages/torchaudio/functional/functional.py(126): spectrogram\n /usr/local/lib/python3.12/dist-packages/torchaudio/transforms/_transforms.py(110): forward\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1747): _call_impl\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1736): _wrapped_call_impl\n /usr/local/lib/python3.12/dist-packages/torchaudio/transforms/_transforms.py(619): forward\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1747): _call_impl\n /usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py(1736): _wrapped_call_impl\n /workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(157): get_vocos_mel_spectrogram\n /workspace/F5-TTS/src/f5_tts/runtime/triton_trtllm/model_repo/f5_tts/1/model.py(216): execute\n"
❌ Actual Behavior
No response