NeuralEngine, tuned to RDNA4.
A from-scratch GPU inference engine in C++ and HIP/ROCm, written to replace an ONNX Runtime and DirectML pipeline. It runs a full object detector on the GPU in under a millisecond on a Radeon RX 9060 XT, with zero memory allocation at runtime. Powers Globality AI, Neural Vision — shipped 2026.
Pure GPU, no runtime in the way.
No ONNX Runtime. No DirectML. The detector is hand-written HIP compute kernels on a single DX12 and HIP interop queue. The captured frame never leaves the card: preprocess, backbone, detection head, and non-max suppression all run device-side, and the result is read through host-mapped pinned memory with no device-to-host copy.
Weights are baked into GPU buffers at startup from a custom .nef format. Activations live in a pre-allocated ping-pong pool. The runtime path allocates nothing: total GPU footprint is 25.53 MB, split into 1.40 MB of weights and 24.12 MB of activations.
Written for Radeon, not ported to it.
The engine targets RDNA4 directly through HIP and ROCm 6.4, AMD's answer to CUDA. It is built around the architecture, not abstracted from it: 32-wide waves instead of the 64 of older GCN, packed FP16 math measured at 21.76 TFLOPS, and tile sizes chosen for the RX 9060 XT's 16 compute units and 4 MB of L2.
Some of the hardest wins were Windows-specific. On WDDM, a stray hipMemsetAsync routes to the copy engine and stalls the whole stream for roughly half a millisecond when compute kernels follow it. Replacing it with a one-thread compute kernel kept every step on the compute engine. An NVIDIA path exists on paper, but this runs on a dual-AMD machine and is tuned for it.
Four parts, all on the GPU.
Every stage is custom. No third-party inference runtime touches a frame.
Fused NHWC HIP kernels: 1x1 and 3x3 convolutions, depthwise 3x3 and 5x5, residual blocks, upsample, and the detection head. Tiled GEMM with packed-FP16 dot2 on the hot path, naive fallbacks elsewhere.
A ping-pong double-buffered activation pool with zero runtime allocation. The whole model sits in 25.53 MB on the GPU, weights and activations combined, allocated once at init.
Forward pass and postprocess captured as one 52-node HIP graph, submitted to WDDM a single time per frame. Output is host-mapped, read with no device-to-host copy.
A custom .nef weight file: FP16, batch-norm folded, CRC32 checked. A Python exporter writes it from PyTorch; a C++ loader validates and uploads it at startup.
2.22 ms down to one.
The first working build ran at 2.22 ms. Each bar below is a measured steady-state average for the forward pass at 512x512. The gains came from layout, from keeping work on one engine, and from cutting copies, not from changing the math.
Forward pass, 512x512, warm steady-state average. Total pipeline projects to about 1.03 ms on real input.
Three problems, worth the weeks.
Routing activations through skip connections. The ping-pong buffer scheme only handles linear chains. Residual adds, the BiFPN feature pyramid, and the detection head each read a tensor that a later layer had already overwritten. Four separate routing bugs, every one fixed by parking the needed feature map in a scratch region until the layer that wanted it could read it.
The WDDM engine-switch stall. A single asynchronous memset was quietly costing half a millisecond by bouncing work onto the copy engine mid-stream. Finding it meant reading GPU timeline traces, not the source. The fix was a few lines; the diagnosis was days.
Zero-copy postprocess. Decode, non-max suppression, the survivor count, and the final gather all run on the GPU and write straight into host-mapped memory. The CPU reads the detections after one stream sync, with no staging buffer and no readback. Postprocess overhead lands at 0.07 ms on real input.
Most of the speed did not come from clever math. It came from refusing to allocate, refusing to copy, and keeping every byte on the GPU.
From here.
The engine is built and benchmarked. The model that runs on it is the next phase: a small single-class detector distilled from a YOLOv8s teacher, trained on roughly 16,000 labelled frames, holding mAP50 within one percent of the teacher's 0.98. Until that finishes the engine runs on randomly initialised weights, which is right for timing and wrong for detections. I would rather say that plainly than fake a number.
The engine is wired to live DXGI capture inside Globality AI, the computer-vision tool it was built to drive. Next up is a portable path to other GPUs. If you want to talk about inference on AMD, reach me on Discord.