DP-KFC: data-free preconditioning for privacy-preserving deep learning
Deep networks have highly anisotropic loss landscapes; DP-SGD injects isotropic noise. Fixing that mismatch usually costs private data or public data. It turns out the geometry was never in the data — it's in the architecture.
The geometric mismatch
DP-SGD earns its guarantee with two spherical operations: clip every per-sample gradient to a global ℓ₂ ball, then add isotropic Gaussian noise. But loss landscapes are anything but spherical — curvature varies by orders of magnitude across parameter directions. The result is an isotropy mismatch: low-sensitivity directions drown in noise while high-sensitivity ones are over-clipped, producing wildly heterogeneous signal-to-noise ratios across layers. In a transformer trained with standard DP-SGD, some layers' signal simply collapses.
Why the standard escapes fail
The classical answer is preconditioning: rescale updates by curvature. In a private setting, both ways of estimating that curvature are compromised:
- From private data — every statistic the preconditioner needs is a query against the data, so it either consumes a serious share of the privacy budget or arrives so noisy that it destabilises the update more than it helps.
- From public data — the precondition-then-privatise strategies lean on a public proxy dataset, which introduces unquantified bias under distribution shift, fails precisely in the specialised domains where DP matters most (there is no "public ImageNet" for mammograms), and complicates privacy auditing.
The Fisher Information Matrix decouples into architectural sensitivity — recoverable by probing the network with synthetic noise — and input correlations — approximable from modality-level frequency statistics. Neither component needs the private data. Mean-field theory makes this precise: the trace of each layer's Fisher block is governed by depth, width, initialisation, and nonlinearity — properties of the architecture, not the dataset.
The method
DP-KFC constructs KFAC preconditioners by probing the network with structured synthetic inputs that carry no privacy cost. For images, that structure is pink noise: natural images universally follow a 1/f power spectrum, so colored noise with the same spectral decay reproduces the local and global correlations a convolution or attention layer actually sees. For text, structured random token sequences play the same role. Labels are sampled uniformly at random — KFAC's factors depend on error magnitudes, not label correctness.
A forward and backward pass over the synthetic batch yields per-layer Kronecker factors; eigendecomposition and damping give the inverse-square-root preconditioner. Applied to gradients before clipping and noising — the scale-then-privatise principle — it makes the private update approximately isotropic, so DP-SGD's spherical clipping and spherical noise finally match the geometry they operate in. The privacy analysis is untouched: nothing in the preconditioner ever saw the data.
Results
In strong privacy regimes (ε ≤ 3), DP-KFC consistently outperforms DP-SGD and adaptive baselines across modalities — CNNs and transformers, vision and text. Layer-wise SNR profiles show it matching the geometry of data-dependent preconditioners without touching any external data. And it matches private-data preconditioners while public-data proxies degrade by up to 4.8% under domain mismatch — the curvature really can be estimated without spending budget or importing distribution shift.
That last comparison is the practical point: in regulated domains like medical imaging, where public proxies don't exist and every ε counts, data-free is not a convenience — it is the only preconditioning that doesn't undermine the guarantee it serves. A companion line of work (MICCAI 2026, under review) pushes the same architecture-driven principle into sample-efficient differentially private medical image segmentation.