Estimator selection experiment

Absolute error against the exact value and wall time of every candidate DiscrepancyEstimator for energydistance/mmd, measured on the splits from support points · energy, herding · energy (scored by energy distance) and herding · gaussian (scored by MMD, median bandwidth resolved once per dataset), on the four Phase 2b datasets at N = 10,000, over 5 rng seeds per (dataset, split, estimator) cell.

Per-row results

datasetsplitkernelestimatormean abs errormax abs errormean time (s)
mixture-2dsupport points · energyEnergyKernelSubsample(2000, 8)0.0006160.0007590.408
mixture-2dsupport points · energyEnergyKernelRandomSlices(64)9.18e-61.47e-50.0244
mixture-2dsupport points · energyEnergyKernelRandomSlices(256)6.85e-61.66e-50.0861
mixture-2dsupport points · energyEnergyKernelRandomSlices(1024)2.12e-65.09e-60.401
mixture-2dherding · energyEnergyKernelSubsample(2000, 8)0.0006630.0007430.305
mixture-2dherding · energyEnergyKernelRandomSlices(64)1.1e-71.89e-70.0199
mixture-2dherding · energyEnergyKernelRandomSlices(256)7.56e-81.99e-70.109
mixture-2dherding · energyEnergyKernelRandomSlices(1024)2.89e-84.27e-80.355
mixture-2dherding · gaussianGaussianKernelSubsample(2000, 8)0.0001170.0001610.337
mixture-2dherding · gaussianGaussianKernelRandomFeatures(512)1.78e-75.36e-70.0769
mixture-2dherding · gaussianGaussianKernelRandomFeatures(2048)8.25e-81.13e-70.324
normal-10dsupport points · energyEnergyKernelSubsample(2000, 8)0.001770.00190.14
normal-10dsupport points · energyEnergyKernelRandomSlices(64)8.01e-50.000140.016
normal-10dsupport points · energyEnergyKernelRandomSlices(256)3.42e-57.65e-50.0805
normal-10dsupport points · energyEnergyKernelRandomSlices(1024)2.23e-54.14e-50.337
normal-10dherding · energyEnergyKernelSubsample(2000, 8)0.001710.001970.161
normal-10dherding · energyEnergyKernelRandomSlices(64)2.34e-53.93e-50.0169
normal-10dherding · energyEnergyKernelRandomSlices(256)1.83e-53.18e-50.0784
normal-10dherding · energyEnergyKernelRandomSlices(1024)7.19e-61.0e-50.309
normal-10dherding · gaussianGaussianKernelSubsample(2000, 8)0.0001480.0001720.289
normal-10dherding · gaussianGaussianKernelRandomFeatures(512)2.71e-74.19e-70.0533
normal-10dherding · gaussianGaussianKernelRandomFeatures(2048)1.66e-72.11e-70.32
uniform-5dsupport points · energyEnergyKernelSubsample(2000, 8)0.001190.001340.176
uniform-5dsupport points · energyEnergyKernelRandomSlices(64)9.32e-50.0001110.0187
uniform-5dsupport points · energyEnergyKernelRandomSlices(256)2.75e-55.39e-50.0769
uniform-5dsupport points · energyEnergyKernelRandomSlices(1024)1.74e-53.86e-50.308
uniform-5dherding · energyEnergyKernelSubsample(2000, 8)0.001140.001260.151
uniform-5dherding · energyEnergyKernelRandomSlices(64)3.67e-67.9e-60.0192
uniform-5dherding · energyEnergyKernelRandomSlices(256)2.71e-64.92e-60.0826
uniform-5dherding · energyEnergyKernelRandomSlices(1024)6.3e-71.32e-60.31
uniform-5dherding · gaussianGaussianKernelSubsample(2000, 8)0.0001380.0001680.278
uniform-5dherding · gaussianGaussianKernelRandomFeatures(512)8.02e-81.05e-70.0689
uniform-5dherding · gaussianGaussianKernelRandomFeatures(2048)2.86e-84.11e-80.318
t3-3dsupport points · energyEnergyKernelSubsample(2000, 8)0.0008140.0009570.157
t3-3dsupport points · energyEnergyKernelRandomSlices(64)4.08e-61.53e-50.0184
t3-3dsupport points · energyEnergyKernelRandomSlices(256)6.08e-61.52e-50.0736
t3-3dsupport points · energyEnergyKernelRandomSlices(1024)1.99e-64.5e-60.312
t3-3dherding · energyEnergyKernelSubsample(2000, 8)0.0008210.001090.162
t3-3dherding · energyEnergyKernelRandomSlices(64)1.87e-62.56e-60.0174
t3-3dherding · energyEnergyKernelRandomSlices(256)4.97e-71.0e-60.081
t3-3dherding · energyEnergyKernelRandomSlices(1024)2.59e-75.2e-70.312
t3-3dherding · gaussianGaussianKernelSubsample(2000, 8)0.0001530.0001750.29
t3-3dherding · gaussianGaussianKernelRandomFeatures(512)1.74e-73.13e-70.064
t3-3dherding · gaussianGaussianKernelRandomFeatures(2048)2.09e-73.85e-70.338

Decision

Rule: an estimator becomes the automatic splitquality fallback if, at no more than Subsample(2000, 8)'s mean wall time, its worst-case max error over every (dataset, split) row is at most one third of Subsample(2000, 8)'s; otherwise Subsample(2000, 8) stays the fallback. Aggregates below take the worst (maximum) max-abs-error and the mean mean-time over every row for that kernel.

EnergyKernel

estimatormax abs error (worst over rows)mean time (s) (over rows)
Subsample(2000, 8)0.001970.208
RandomSlices(64)0.000140.0189
RandomSlices(256)7.65e-50.0835
RandomSlices(1024)4.14e-50.331

Decision: RandomSlices(64)

GaussianKernel

estimatormax abs error (worst over rows)mean time (s) (over rows)
Subsample(2000, 8)0.0001750.298
RandomFeatures(512)5.36e-70.0658
RandomFeatures(2048)3.85e-70.325

Decision: RandomFeatures(512)