Dae Young Kim
About / Categories / Tagged
Ensuring Correct Execution Context in Ray Tune’s Distributed Environment
How ray.tune.Tuner
works?
Let’s say we define a tuner and its trainable like this:
MODEL_CONFIG = { ... }
RAY_HP_SPACE = { ... }
def trainable(config):
...
tuner = tune.Tuner(
trainable,
tune_config=tune.TuneConfig(
...
),
param_space=RAY_HP_SPACE,
)
These are the three steps happens inside during the hyperparameter search using Ray Tune.
- Tuner Captures the Context: When you initialize the Tuner, it stores (or “captures”) the
trainable
function as part of its configuration. That function’s closure includes any local variables (like the trial-specific configuration passed to it) and references to global variables (likeMODEL_CONFIG
). - Serialization and Distribution: Ray Tune then serializes (pickles) the Tuner—including the captured
trainable
function—and sends it to a remote worker process. - Independent Remote Environment: The remote worker process has its own environment, independent of the driver. If
MODEL_CONFIG
was captured as a global variable and is not correctly defined or available in the worker’s environment, it may end up beingNone
(or have an unexpected value). This leads to errors when the worker callstrainable
Key Concepts
- Trainable: an object that you can pass into a Tune run.
- Closure: a programming concept where a function “remembers” the environment in which it was created. This means that the function retains access to variables from its surrounding scope—even after that scope has finished executing.
Reference
© 2025 Dae Young Kim ― Powered by Jekyll and Textlog theme