Repository navigation
[BUGFIX] Honor Hardtanh min_val and max_val - #344
Open
andrenepomuceno wants to merge 1 commit into
Open
andrenepomuceno wants to merge 1 commit into
andrenepomuceno wants to merge 1 commit into
Conversation
The trainer exports torch.nn.Hardtanh as {"type": "Hardtanh", "min_val": ...,
"max_val": ...}, but the bounds were ignored and the activation always clamped
to [-1, 1]. Parse the optional bounds and build a parameterized Hardtanh when
they are present. Bare "Hardtanh" still uses the [-1, 1] singleton.
Fixes sdatkinson#342
dfernandes83
added a commit
to dfernandes83/NeuralAmpModelerCore
that referenced
this pull request
Oct 6, 2026
…s() larger than max buffer), sdatkinson#343 (keep activation in the residual when layer1x1 is inactive), sdatkinson#344 (honor Hardtanh min_val/max_val)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #342
Problem
The trainer exports
torch.nn.Hardtanhwith its bounds,{"type": "Hardtanh", "min_val": ..., "max_val": ...}(nam/models/_activations.py, ~L253–258 in neural-amp-modeler 0.13).ActivationConfig::from_jsonignored both keys, andHardtanhalways clamped to[-1, 1], so a model trained with other bounds rendered differently from the trainer.Change
This follows the existing
LeakyReLUpattern: a shared singleton for the default, and a dedicated instance when parameters are given.ActivationConfig::from_jsonreads the optionalmin_val/max_valforHardtanh. These are the existingActivationConfigfields thatLeakyHardtanhalready uses.ActivationHardTanhgets a(min_val, max_val)constructor, andhard_tanhgets a matching overload. The defaults are unchanged (-1,1).get_activation(config)returns a parameterizedActivationHardTanhwhen either bound is set (a missing bound defaults to ±1). Otherwise it returns the"Hardtanh"singleton as before.A bare
"Hardtanh"string, or an object without bounds, behaves exactly as before. No other path special-cases Hardtanh: the fast-tanh and LUT swaps only touch Tanh/Sigmoid/SiLU, and the A2 fast path only matches LeakyReLU.Tests
test_hardtanh_config: sets the bounds onActivationConfigdirectly and checks clamping at both bounds and pass-through inside them.test_from_json_hardtanh_bounds: parses the trainer's export format and checks the parsed bounds and the resulting activation.test_from_json_hardtanh_default:"Hardtanh"and{"type": "Hardtanh"}leave the bounds unset, resolve to the singleton, and clamp to[-1, 1].test_hardtanh_bounds_apply_realtime_safe: applying a bounded Hardtanh allocates nothing (allocation tracking).Cross-check: a randomly initialised neural-amp-modeler 0.13 WaveNet with inactive
layer1x1and a bounded Hardtanh, rendered withtools/render, matches the PyTorch forward pass (pad_start=True, 4096 samples) to a max abs error of 7.5e-8 with this PR and #341 (inactive layer1x1 residual) applied, down from 1.9e-3 onmain.run_testspasses in both the default and-DNAM_USE_INLINE_GEMMbuilds (g++ 11, Linux). Before the fix, the new tests fail.Note
Like
LeakyHardtanh, this does not check thatmin_val < max_val. PyTorch asserts it at construction, so a trainer-exported file always satisfies it.