[Gnpd]

Release 0.2.3: format v4, security hardening and round-trip fixes

Summary

This PR prepares openmodels 0.2.3. It has three main themes:

1. Security: loading a model file no longer imports modules or calls functions the file names, unless they are on an allowlist.
2. Wire format v4: each estimator records the package it comes from, and dict values and keys keep their types.
3. Round-trip fixes: many loaded models couldn't predict, transform or refit, or quietly behaved differently from the original. These are fixed.

The full list of changes is in CHANGELOG.md. The format reference is in docs/format.md.

⚠️ Breaking changes

• Function allowlist. Functions referenced by a file (FunctionTransformer.func, SelectKBest.score_func, ...) are no longer imported from whatever module the file names. A function is accepted only if:

  • it is found in an already-imported module of numpy, scipy, scikit-learn or a registered custom estimator's package, and
  • the module it is defined in (its __module__) is one of those too. This blocks re-exports such as sklearn.datasets._base.makedirs, which is os.makedirs.

  Private names and __main__ are always refused. To load files that use your own functions, pass SklearnSerializer(trusted_function_modules=["mypkg"]).

• Builtins such as abs now raise DeserializationError unless "builtins" is trusted.

• Bound methods of an object (e.g. a fitted estimator's fit_transform) now raise SerializationError at save. Saving them by name lost the object they belong to.

• Clearer errors. SerializationManager now raises SerializationError/DeserializationError, with the original exception as __cause__, where it used to leak KeyError, AttributeError or TypeError.

🔒 Security

• Function references are checked against the allowlist above. Before, the file's module name was passed straight to __import__.
• A Gaussian-process kernel_type must name a concrete Kernel class. Before, any callable in sklearn.gaussian_process.kernels could be called with the file's parameters.
• A SciPy distribution must name a scipy.stats distribution generator. Before, any public scipy.stats function could be called.
• Structured dtypes are parsed with ast.literal_eval, never eval.
• SECURITY.md has a new section, "Functions referenced by a model".

📦 Wire format v4

• Every estimator records estimator_package next to estimator_class. Classes are looked up by package and class name, so a custom class can share a scikit-learn class's name (e.g. chemotools' MinMaxScaler) without replacing it. v1–v3 files are still looked up by bare name, now with a warning when the name is ambiguous.
• Dicts whose values need restoring are typed per key. Non-string keys (e.g. class_weight={0: 1.0}) are saved as text, with their types stored next to the value types.
• Lists of arrays record one dtype per array when needed, so a float32 MLP stays float32.
• Compatibility:
  • 0.2.3 reads every older file.
  • 0.2.2 reads v4 files with a warning, but typed dict values load as plain JSON and non-string keys as strings.
  • 0.2.2 can't read per-element dtype lists (e.g. a float32 MLP saved by 0.2.3).

🐛 Notable fixes

• Neighbors estimators: the search tree is rebuilt on load with the estimator's own metric and leaf_size. Non-euclidean kd_tree/ball_tree models returned wrong neighbors before. BallTree models can now be saved, and KNeighborsRegressor/RadiusNeighborsRegressor can predict after load.
• KernelDensity: now round-trips exactly, including its metric and sample weights.
• Gaussian-process kernels: every kernel now round-trips (Matern, RationalQuadratic, CompoundKernel, anisotropic RBF, ...).
• ColumnTransformer:
  • transforms pandas DataFrames after load;
  • keeps slices and make_column_selector;
  • can be saved with scikit-learn 1.6 when columns are selected by name (the _RemainderColsList error).
• Composite models saved as pickle (Pipeline, FeatureUnion, Voting*, Stacking*) load as estimators instead of raw dicts.
• NumPy values: NumPy types used as params (dtype=np.int64), NumPy scalars (np.float32(0.5)), ufuncs, scipy.special functions and NumPy array functions (np.std, np.linalg.norm) now load as the original objects instead of floats or raw dicts.
• SciPy distributions: both continuous and discrete ones round-trip inside param_distributions.
• Private state: state needed by predict_proba, staged_predict*, transform, get_feature_names_out and score is now saved or rebuilt for HistGradientBoosting, LDA, KNNImputer, OneClassSVM, NMF, HDBSCAN, GridSearchCV/RandomizedSearchCV, and 24 estimators that use get_feature_names_out.
• Refitting: params that scikit-learn accepts only as tuples come back as tuples, so loaded MinMaxScaler, CountVectorizer, ... can be refitted.
• custom_estimators: a list of (name, class) pairs now registers its classes. Before, it silently registered nothing.

Some fixes only apply to files saved with 0.2.3 (the changelog marks them "re-save older files"). Others also repair files written by earlier versions.

🧪 Testing

• New test modules cover every fix, including:
  • test_inference_parity.py, which compares every inference method of the original and loaded models;
  • test_resume_training.py, which records which estimators can resume training after load;
  • test_format_v4.py;
  • regression tests for the function allowlist, including the re-export bypass.
• The roundtrip_fit and run_test_model test helpers now check more strictly, so state that isn't saved can no longer go unnoticed.
• Local run (Windows, Python 3.11, scikit-learn 1.9.1, NumPy 2.4, SciPy 1.17):
  11189 passed, 551 skipped, 95 xfailed
• mypy openmodels: clean. flake8/black are clean on openmodels/.