Research
The Synoros Engine
The engineer's PyTorch.
Status: active research prototype, started June 2026. No release, no license, still moving fast.
The idea
The two dominant deep learning frameworks each tax you in a different direction. tinygrad asks you to rebuild almost everything from a small set of primitives, and you pay for that abstraction in performance and in day-to-day ergonomics. PyTorch is batteries-included, and that generality is exactly what makes it hostile to research that leaves the standard fixed-rank, Euclidean case: manifolds, Lorentzian metrics, causal cones. Its breadth costs performance the moment your tensors and ops stop looking like everyone else's.
The engine's bet is to own the parts around the kernels instead: the computation graph, automatic differentiation, memory placement, and tensor liveness, all in a small, low-overhead C core. Each operation then calls whichever kernel is actually best for its shape and device, hand-written, vendor-supplied, or generated. The wins it chases are dispatch overhead, small and odd shapes that vendor heuristics handle poorly, and whole-graph memory planning.
Determinism is law
Reruns are byte-identical, always. Every fast execution path, the interpreter, kernel fusion, a vendor kernel, the JIT, is checked against a double-precision oracle evaluator within float32 tolerance, and a fast-but-wrong kernel is never allowed to stand in for a correct one. Floating-point reduction order is fixed, and no path relies on nondeterministic parallelism that would make two runs of the same graph diverge. This held across a vendor boundary in practice: the same transformer produced a bit-identical training loss on an AMD RDNA4 card and an NVIDIA Ampere card, from the same source, on the first try.
The optimization loop
The engine's optimization loop runs in four steps against its flat, integer-indexed instruction tape. Track computes each tensor's liveness, from when it's produced to its last use. Place uses that liveness to size the working set against the cache hierarchy, fusing elementwise chains so intermediates stay in registers or L1 instead of round-tripping to DRAM, and pinning hot weights. Measure profiles where time and memory actually go, per node and per subgraph, on real hardware. Recompile regenerates hot subgraphs with better tiling and fusion once the profile justifies it, then measures again. The engine profiles the graph on real silicon and rewrites its own hot paths to fit.
Benchmarks
Single-threaded CPU, C engine vs. PyTorch 2.12.0 eager (CPU, one thread):
| Scenario | C engine | PyTorch eager, CPU, 1 thread | Result |
|---|---|---|---|
| Tiny MLP, forward + backward | 1,029.7 ns/iter | 118,271.6 ns/iter | 114.9× faster |
| Elementwise chain (1024×1024, depth 6) | 3,256,963.1 ns/iter | 6,747,035.0 ns/iter | 2.07× faster |
| Matmul, 512×512 × 512×512 | 2,473,745.1 ns/iter | 1,998,919.5 ns/iter | 1.24× slower |
The MLP number measures dispatch overhead: the tensors are tiny, so PyTorch's Python/autograd dispatch cost dominates its runtime and the C engine's near-zero dispatch cost wins by a wide margin. The elementwise result is a real, if smaller, win from avoiding materialized intermediates and using a vectorized kernel. At a normal GEMM size, PyTorch's vendor BLAS backend is still faster than the engine's hand-tiled AVX2 kernel. It does not out-tune vendor GEMM libraries.
On an AMD RDNA4 card (RX 9060 XT), running a GPT-2-style transformer against PyTorch 2.12.1+rocm7.1:
| Workload | Model size | Result vs. PyTorch ROCm |
|---|---|---|
| Inference, sequence 128 | ~11M params | 1.45× faster |
| Inference, sequence 256 | ~98M params | 1.61× faster |
| Inference, sequence 512 | ~98M params | 2.23× faster |
| Training step, sequence 256 | ~98M params | 1.13× faster |
| LoRA training step (batch 128, dim 768, rank 8) | — | 1.6× faster (0.291 vs. 0.464 ms) |
The inference win holds and widens as the model and sequence length grow: 1.45× at the smallest configuration up to 2.23× at the largest one measured. The training-step and LoRA-step wins are smaller. They come from measuring which vendor GEMM kernel is fastest per shape and pinning it, cutting device-side operation count below PyTorch's own, and removing unnecessary parameter copies.
On NVIDIA, the engine loses. On an RTX 4060 Ti running the same transformer, the engine's CUDA path took roughly 4× longer than PyTorch compiled with torch.compile on the forward pass, and roughly 3.5× longer on the training step, even though the engine already calls cuBLAS for its matrix multiplies and already vectorizes its elementwise kernels. The gap is fusion coverage, not GEMM performance: PyTorch's compiler collapses layernorm, softmax, and the GELU MLP into a handful of dispatches, while the engine still runs each as several small, unfused kernels.
Research notes
Two research directions have come out of building the engine. Both are unpublished drafts.
Disciplined Hybrid Programming
A proposed certificate calculus for optimization problems that mix exact, convex structure with learned neural components in the same expression graph. Each node would carry tags, curvature, a Lipschitz constant, an error bound, and the calculus's composition rules would let a verifier certify the quality of the whole hybrid problem's solution, not just bound its output. So far it is a spec for a research program, not a theorem.
The DHP seed theorem
A first concrete instantiation of the calculus above, applied to the engine's own optimization choices: it frames kernel, tiling, and fusion selection as a certified search over a discrete structure and continuous parameters, each candidate checked against a correctness tolerance and a measured cost model with a stated calibration error. The theorem is drafted, not yet fully proven.
Where it's going
Further out, the bet is that the next decade of ML compute shifts substrate, toward monolithic 3D carbon-nanotube chips or silicon photonics. Both favor local, recurrent, compute-near-memory dataflow over one big flat matrix multiply. The engine owns the graph, autodiff, and memory, and stitches kernels beneath a stable interface, so if that shift comes, a new device class is another kernel provider underneath rather than a rewrite of everything above. There is nothing concrete to show for this yet; the GPU and API work comes first.
If you have a workload you'd like to see it run, get in touch.