Skip to content

← Flax NNX step 89 of 100

Medium Primitives

Repeat NNX Linear Initialization

Jump to the task

A newly constructed NNX Linear layer can reproduce another layer's initial parameters when both use the same configuration and fresh RNG streams with the same seed. This exercise checks the resulting forward calculation. It does not read checkpoint data or perform a sharded restore.

Start each RNG stream at the same seed

first = nnx.Linear(D, F, rngs=nnx.Rngs(seed))
second = nnx.Linear(D, F, rngs=nnx.Rngs(seed))
output = second(x)

In the same environment, these constructors produce matching kernel and bias values. They create separate model objects. Reusing one advancing nnx.Rngs object for both constructors is different: the second layer receives later keys from the stream.

Reproducibility depends on the layer configuration, initializers, RNG behavior and software environment. It is not a promise of bit-for-bit equality across all versions or backends.

Initialization cannot recover learned parameters

If training changes first.kernel or first.bias, constructing second from the original seed recreates the initial values. It does not restore the changes. A real checkpoint restores saved values and may also contain optimizer state, counters and other state needed to resume training.

NNX separates model structure from array state through operations such as nnx.split and nnx.merge. Restoring compatible state into a model is a different operation from constructing fresh random parameters. Real sharded restoration also needs appropriate array types and placement; none of those steps is tested by this exercise.

Comparing one forward output cannot prove that every parameter matches. For example, zero input hides all kernel differences in a Linear layer. Actual restore validation should check state structure and arrays as well as suitable model outputs, using the expected dtype and numerical tolerance.

Problem

Implement orbax_load_simulate(seed, x, features):

  1. Convert seed and output width F to integers; take D = x.shape[-1].
  2. Construct two default nnx.Linear(D, F) layers, each with its own freshly created nnx.Rngs(seed), to compare initialization locally.
  3. Apply the second layer to x with no activation function.
  4. Return the result flattened in row-major order.

x is a finite float32 JAX matrix of shape (B, D), with positive B and D. Width is a positive integer-valued scalar and seed is nonnegative and integer-valued; both may arrive as floats. Keep default initializers and biases. Leave x unchanged.

The output is a JAX vector of shape (B * F,) representing x @ kernel + bias, with each row's features kept together. Negative outputs stay negative. Tests check these seeded values; they cannot prove that two models were constructed or that any checkpoint was restored. The diagram below is an independent array-placement example, not a step performed by the function.

Loading visualization…