[None][fix] Release GIL around cudaLaunchHostFunc to fix guided-decoding host-func deadlock#17
Open
sopwg612 wants to merge 1 commit into
Conversation
…ing host-func deadlock launchHostFunc held the Python GIL across cudaLaunchHostFunc. The CUDA driver serializes host-function dispatch on a stream, so in the eager (non-capturing) path a launch issued while a callback is in flight busy-waits in the driver. That in-flight callback runs cudaHostFuncTrampoline, which reacquires the GIL to execute Python (e.g. xgrammar in guided decoding). With the launching thread holding the GIL, the callback can never acquire it -> circular deadlock. Under Eagle3 speculative decoding + xgrammar guided decoding this wedged the engine with zero forward progress (observed via py-spy on both TP ranks). This is a regression from NVIDIA#13251, which correctly dropped the call_guard (so the nanobind arg construct/destruct happens under the held GIL) but also removed the inner GIL release around the driver enqueue that NVIDIA#10028 had added. Fix: re-release the GIL only around cudaLaunchHostFunc, and only on the eager path (freeUserData == true). During graph capture no callback runs, so the GIL is kept to hold capture on a single thread. The release scope ends before the nanobind refcount work, preserving the object-lifetime fix from NVIDIA#13251. Adds a regression guard in tests/unittest/bindings/test_hostfunc.py. Signed-off-by: Sihan Wang <sihan@deepinfra.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
launchHostFuncheld the Python GIL acrosscudaLaunchHostFunc. The CUDA driver serializes host-function dispatch on a stream, so on the eager (non-capturing) path a launch issued while a callback is in flight busy-waits in the driver. That in-flight callback runscudaHostFuncTrampoline, which reacquires the GIL to execute Python (e.g. xgrammar in guided decoding). With the launching thread holding the GIL, the callback can never acquire it -> circular deadlock, zero forward progress, no crash.Triggered by Eagle3 speculative decoding + xgrammar guided decoding. Captured live via py-spy
--nativeon both TP ranks of a wedged engine:sched_yield-> libcuda ->launch_hostfunc->GuidedDecoder.execute->eagle3.forward~gil_scoped_release(reacquiring GIL) -> xgrammarfill_next_token_bitmask->GuidedDecoder.buildRoot cause
Regression from NVIDIA#13251, which correctly removed the
call_guard(so the nanobind arg construct/destruct happens under the held GIL) but also removed the inner GIL release around the driver enqueue that NVIDIA#10028 had added.Fix
Re-release the GIL only around
cudaLaunchHostFunc, and only on the eager path (freeUserData == true). During graph capture no callback runs, so the GIL is kept to hold capture on a single thread. The release scope ends before the nanobind refcount work, preserving NVIDIA#13251's object-lifetime fix (nvbugs-5643631). Reached only viaCapturableGuidedDecoder(one-model spec + guided); non-guided / non-spec paths are untouched.Validation
--native; both ranks show the identical cycle above.test_hostfunc_gil_release_no_deadlock. NOTE: not executed in this PR (needs a GPU + a C++ rebuild); the production repro is the primary validation.Scope
Fork hotfix targeting
deep-main-v1.3.0rc16. The same patch is suitable for NVIDIA upstream (main) as a follow-up.