Repository navigation
feature request: edge device CUDA backend #1440
Description
Does it make sense for executorch to have a mobile cuda backend? There are many edge devices in the Jetson lineup from nvidia that have a cuda gpu, but can benefit from not wanting to link an enormous libtorch dependence.
Activity
[-]edge cuda backend[/-][+]feature request: edge device CUDA backend[/+]bionictoucan commentedon Feb 12, 2024
+1 to this, would be nice to get comparable performance to TensorRT without having to export models to ONNX etc. first!
hietalajulius commentedon Feb 29, 2024
+1
mergennachin commentedon Feb 29, 2024
@mindbeast @bionictoucan @hietalajulius
Hi, thanks for the comment.
Yes, that makes sense in general.
Right now, for ExecuTorch, we are integrating Vulkan into ExecuTorch. The reason is that it is a suitable solution for mobile GPUs. Enabling mobile use-cases is our primary goal at the moment.
We will revisit Cuda, but perhaps, in the second half in the year. Curious, what are your current product needs?
DzAvril commentedon Sep 11, 2024
Apologies for opening a similar feature request in #5263.
@mergennachin We want to deploy LLMs in cars, but Python-based inference frameworks like vLLM and SGLang are not suitable for edge devices.
Nearly five months have passed, is there any progress on this?
digantdesai commentedon Sep 11, 2024
Thank you for following up @DzAvril.
I guess this is using a platform similar to Jetson?
No update yet on CUDA backend for ET at the moment. We can get back to you here once we plan something.
DzAvril commentedon Sep 12, 2024
@digantdesai Yes, Jetson Orin for now, and possibly Thor in the future.
Looking forward to your update.
DuinoDu commentedon Jan 11, 2025
For mobile cuda backend, does
torch_tensorrtsatisfy the requirement?mindbeast commentedon Jan 23, 2025
@DuinoDu My expectation is that compatibility is poor with torch_tensorrt. I expect a more compliant backend in executorch would help a lot of developers.