From e1847b3db70aa097e6b972f0875de84c450f48c0 Mon Sep 17 00:00:00 2001
From: kyo <113977534+kyo-takano@users.noreply.github.com>
Date: Mon, 27 Jan 2025 10:19:28 +0000
Subject: [PATCH 1/2] **v0.2.0**: Imrprove UX and performance
- Change optimizer from L-BFGS to vanilla BFGS after quantitative experiments
- Add an LLM example data & notebook
- Improve documentation regarding the initial parameter grid & optimization tips
- Make all arguments optional except `param_grid`; previously `project_dir` and `seed_ranges` were unnecessarily required
- Debug dtype issues around visualization
---
.gitignore | 1 +
README.md | 161 ++++-
chinchilla/_logger.py | 2 +-
chinchilla/_metrics.py | 1 +
chinchilla/_utils.py | 1 +
chinchilla/core.py | 254 +++++--
chinchilla/database.py | 3 +-
chinchilla/simulator.py | 1 -
chinchilla/visualizer.py | 79 ++-
docs/TIPS.md | 129 ++--
docs/api-reference.md | 329 +++++----
docs/changes.md | 12 +-
docs/imgs/LBFGS--asymmetric.png | Bin 42306 -> 0 bytes
docs/imgs/LBFGS--seeds-too-small.png | Bin 41205 -> 0 bytes
docs/imgs/LBFGS--symmetric.png | Bin 36794 -> 0 bytes
docs/imgs/LBFGS--underfit.png | Bin 43496 -> 0 bytes
docs/imgs/algorithm.init-improved.png | Bin 0 -> 43603 bytes
docs/imgs/algorithm.init-original.png | Bin 0 -> 51379 bytes
docs/imgs/optim--asymmetric.jpg | Bin 0 -> 41080 bytes
docs/imgs/optim--seeds-too-small.jpg | Bin 0 -> 42173 bytes
docs/imgs/optim--symmetric.jpg | Bin 0 -> 40501 bytes
docs/imgs/optim--underfit.jpg | Bin 0 -> 44502 bytes
docs/imgs/parametric_fit.png | Bin 0 -> 451711 bytes
docs/imgs/sweep_param_grid.improved.png | Bin 0 -> 63715 bytes
docs/imgs/sweep_param_grid.original.png | Bin 0 -> 68856 bytes
examples/efficientcube.ipynb | 44 +-
examples/llm/df.csv | 235 +++++++
examples/llm/main.ipynb | 722 ++++++++++++++++++++
examples/llm/simulation--optim.png | Bin 0 -> 33928 bytes
examples/llm/simulation--parametric_fit.png | Bin 0 -> 311186 bytes
pyproject.toml | 4 +-
31 files changed, 1619 insertions(+), 359 deletions(-)
mode change 100644 => 100755 README.md
delete mode 100644 docs/imgs/LBFGS--asymmetric.png
delete mode 100644 docs/imgs/LBFGS--seeds-too-small.png
delete mode 100644 docs/imgs/LBFGS--symmetric.png
delete mode 100644 docs/imgs/LBFGS--underfit.png
create mode 100644 docs/imgs/algorithm.init-improved.png
create mode 100644 docs/imgs/algorithm.init-original.png
create mode 100644 docs/imgs/optim--asymmetric.jpg
create mode 100644 docs/imgs/optim--seeds-too-small.jpg
create mode 100644 docs/imgs/optim--symmetric.jpg
create mode 100644 docs/imgs/optim--underfit.jpg
create mode 100755 docs/imgs/parametric_fit.png
create mode 100644 docs/imgs/sweep_param_grid.improved.png
create mode 100644 docs/imgs/sweep_param_grid.original.png
create mode 100755 examples/llm/df.csv
create mode 100644 examples/llm/main.ipynb
create mode 100644 examples/llm/simulation--optim.png
create mode 100644 examples/llm/simulation--parametric_fit.png
diff --git a/.gitignore b/.gitignore
index 2b80308..58d7b24 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,6 +6,7 @@ __pycache__/
# Distribution / packaging
dist/
*.egg-info/
+build/
# Misc.
.pytest_cache/
diff --git a/README.md b/README.md
old mode 100644
new mode 100755
index fc4afa4..9166f18
--- a/README.md
+++ b/README.md
@@ -1,6 +1,15 @@
# `chinchilla`
-`chinchilla` is a research toolkit designed to estimate scaling laws and train compute-optimal models for various deep learning tasks.
+![Parametric fit on LLM training runs](docs/imgs/parametric_fit.png)
+
+`chinchilla` is a research toolkit designed to estimate scaling laws & train compute-optimal models for various deep learning tasks.
+
+## Features
+
+- **Scaling Law Estimation**: Fit a loss predictor based on multiple training runs.
+- **Compute-Optimal Allocation**: Train the best possible model within a given compute budget.
+- **Progressive Scaling**: Iteratively update the scaling law estimation and scale up the compute.
+- **Simulation Mode**: Test scaling law estimations in hypothetical scenarios.
@@ -11,7 +20,6 @@
-- Researching the neural scaling law itself
- Scaling compute for
- Large Language Models (LLM)
- Vision Transformers (ViT)
@@ -20,18 +28,21 @@
- Knowledge distillation
- Evaluating compute efficiencies of new algorithms & architectures
- Researching the neural scaling law itself
+
|
- **Probably Not For**:
+ Probably **NOT** For...
|
- Fine-tuning tasks
- Data-scarce domains
+- etc.
|
+
@@ -39,57 +50,123 @@
> This work builds upon the scaling law formulation proposed in [the original Chinchilla paper](https://deepmind.google/discover/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/) by DeepMind (2022),
> with some modifications detailed in [./docs/changes.md](https://github.com/kyo-takano/chinchilla/tree/master/docs/changes.md).
-## Features
+## Installation
-- **Scaling Law Estimation**: Fit a loss predictor based on multiple training runs.
-- **Compute-Optimal Allocation**: Train the best possible model within a given compute budget.
-- **Progressive Scaling**: Iteratively update the scaling law estimation and scale up the compute.
-- **Simulation Mode**: Test scaling law estimations in hypothetical scenarios.
+**From PyPI**
-## Basics
+```bash
+pip install -U chinchilla
+```
-### Definitions
+**From Source**
+
+```bash
+git clone https://github.com/kyo-takano/chinchilla.git
+cd chinchilla
+pip install -e .
+```
+
+## Prerequisite: Chinchilla formulation
+
+Just in case you are not familiar, here is the formulation of the scaling law estimation:
+
+
+
+
+Variables
- $N$: The number of parameters
- $D$: The number of data samples
- $C$: Total compute in FLOPs ($C\approx 6\ ND$)
-- $L(N,\ D) = E + A/N ^ \alpha + B / D ^ \beta$: A loss predictor parameterized by $\{E, A, B, \alpha, \beta\}$ and $C$
+- $L(N,\ D) = E + A / N ^ \alpha + B / D ^ \beta$: A loss predictor parameterized by $\{E, A, B, \alpha\}$ and $\beta$
+
+ ---
+
+ **Intuition**:
+ - $E$ corresponds to the **irreducible loss** that can only be atained with an ideal model with infinite compute
+ - $A / N ^ \alpha$ accconts for the additional loss coming from insufficiency of model size;
+ - $ B / D ^ \beta$, insufficiency of data amount.
+
+
+
+
-### Compute-Optimal Allocation
+
+
+Objective
1. Optimize the parameters $\{E, A, B, \alpha, \beta\}$ to better predict losses $L_i$ from $(N_i, D_i)$
2. Solve $\underset{N,\ D}{argmin}\ L(N,\ D\ |\ C)$, which can be derived from $\{A, B, \alpha, \beta\}$
-### `chinchilla` Procedure
+
-- `seed`: Sample X training runs $(N_i, D_i, L_i)$, referred to as **seeds**
-- For i = 0 to K:
- - `fit`: Optimize the scaling law parameters to fit $L(N,\ D)$ on the training runs
- - `scale`: Configure a new model with a **scaled** compute
- - Evaluate the allocation by training a model
- - `append`: Add the result to the database of training runs
+## Usage
-## Installation
+### 1. Fitting the scaling law on existing dataset
-> [!WARNING]
->
-> `chinchilla` requires Python >= 3.8
+> [!NOTE]
+> An example of this usage can be found [here](examples/llm/)
-**From Source** (Recommended for Customization)
+First, prepare a CSV looking like this and save it as `df.csv`:
-```bash
-git clone https://github.com/kyo-takano/chinchilla.git
-cd chinchilla
-pip install -e .
+```csv
+C,N,D,loss
+1.3972367362937152e+18,73824672,3154403320,3.405928
+1.7656304230443515e+18,89818214,3276303602,3.325255
+2.0558971596900728e+18,105811837,3238291053,3.300442
+...
```
-**From PyPI**
+Second, define a grid of initial parameters to fit like:
-```bash
-pip install -U chinchilla
+```python
+import numpy as np
+from chinchilla import Chinchilla
+cc = Chinchilla(
+ "./", # Assuming `df.csv` is under ./
+ param_grid=dict(
+ E=np.linspace(1, 2, 5),
+ a=np.linspace(1, 10, 5), # a: log(A)
+ b=np.linspace(1, 10, 5), # b: log(B)
+ alpha=np.linspace(0.1, 0.7, 5),
+ beta=np.linspace(0.1, 0.7, 5),
+ ),
+)
```
-## Usage
+Finally, call `cc.fit()` & you'll get the parameters fit on your dataset, which you can easily access as `cc.params`
+
+```python
+>>> cc.fit()
+>>> cc.params
+{'E': 1.7004437920205586,
+ 'A': 185.388090185727,
+ 'B': 1627.0012474587165,
+ 'alpha': 0.28923265350161337,
+ 'beta': 0.3556020928031086}
+ ```
+
+By calling `cc.scale` with FLOPs specified like
+
+```python
+cc.allocate_compute(C=1e24)
+```
+
+You can get an estimatedly compute-optimal allocation of compute to $N$ and $D$.
+
+### 2. Scaling from scratch
+
+> [!NOTE]
+> An example of this usage can be found [here](examples/llm)
+
+> **Procedure**:
+>
+> - `seed`: Sample X training runs $(N_i, D_i, L_i)$, referred to as **seeds**
+> - For i = 0 to K:
+> - `fit`: Optimize the scaling law parameters to fit $L(N,\ D)$ on the training runs
+> - `scale`: Configure a new model with a **scaled** compute
+> - Evaluate the allocation by training a model
+> - `append`: Add the result to the database of training runs
Below is an example to get started with `chinchilla`.
@@ -143,7 +220,9 @@ Ensure you define functionally equivalent versions of:
- `YourModelClass`: Your model class definition.
- `train_and_evaluate`: Function to train and evaluate your model.
-## Simulation
+
+
+ Simulation Mode
You can also visualize how `chinchilla` would perform under the given setup and a hypothetical scaling law, optionally with a **_noise term_**:
@@ -166,17 +245,25 @@ cc.simulate(
)
```
-Please see [API Reference](https://github.com/kyo-takano/chinchilla/tree/master/docs/api-reference.md) for more.
+
## Examples
-Find a practical application of `chinchilla` in the [`examples`](https://github.com/kyo-takano/chinchilla/tree/master/examples) directory (more to come):
+Find practical applications/examples of `chinchilla` in the [`examples`](https://github.com/kyo-takano/chinchilla/tree/master/examples) directory (more to come):
-- [Training Compute-Optimal Rubik's Cube Solvers](https://github.com/kyo-takano/chinchilla/blob/master/examples/efficientcube.ipynb) (100 PetaFLOPs)
+- [Allocating $10^{24}$ FLOPs to a single LLM](https://github.com/kyo-takano/chinchilla/blob/master/examples/llm) [NEW]
+
+- [Scaling Rubik's Cube Solvers from Scratch](https://github.com/kyo-takano/chinchilla/blob/master/examples/efficientcube.ipynb)
## Documentation
-For a detailed API Reference, tips, differences from the original Chinchilla paper, etc., please browse to [./docs](https://github.com/kyo-takano/chinchilla/tree/master/docs).
+- [API Reference](https://github.com/kyo-takano/chinchilla/tree/master/docs/api-reference.md)
+
+- [Tips](https://github.com/kyo-takano/chinchilla/tree/master/docs/TIPS.md)
+
+- [Math](https://github.com/kyo-takano/chinchilla/tree/master/docs/math.md)
+
+- [Differences from the original Chinchilla](https://github.com/kyo-takano/chinchilla/tree/master/docs/changes.md)
## Contributing
diff --git a/chinchilla/_logger.py b/chinchilla/_logger.py
index e5c3915..a0fb187 100755
--- a/chinchilla/_logger.py
+++ b/chinchilla/_logger.py
@@ -1,5 +1,5 @@
"""
-Contains a utility function `get_logger`. This module also filters out noisy debug messages
+Contains a utility function `get_logger`. This module also filters out noisy debug messages
from `matplotlib` and suppresses redundant warnings from `numpy` and `matplotlib`.
"""
diff --git a/chinchilla/_metrics.py b/chinchilla/_metrics.py
index 4cb8ac9..bcd8442 100644
--- a/chinchilla/_metrics.py
+++ b/chinchilla/_metrics.py
@@ -1,4 +1,5 @@
"""A few loss & weight functions you can use on demand."""
+
from __future__ import annotations # PEP 604 backport
import numpy as np
diff --git a/chinchilla/_utils.py b/chinchilla/_utils.py
index 912cc03..96d67a3 100755
--- a/chinchilla/_utils.py
+++ b/chinchilla/_utils.py
@@ -1,4 +1,5 @@
"""Utility functions."""
+
from __future__ import annotations # PEP 604 backport
import itertools
diff --git a/chinchilla/core.py b/chinchilla/core.py
index 24c9d2f..111cdda 100755
--- a/chinchilla/core.py
+++ b/chinchilla/core.py
@@ -23,7 +23,7 @@
# 128bit/96bit: more precise than 64bit at the cost of approx. 2x more time
DTYPE = np.longdouble
# 64bit: yields a *slightly different*, plausibly less precise result;
-# Recommendable exclusively for agile testing
+# Recommended exclusively for agile testing
# DTYPE = np.double
# Clip values by lower precision for stability with `loss_fn` and `weight_fn`.
@@ -54,11 +54,13 @@ class Chinchilla:
alpha: float
beta: float
+ algorithm = "BFGS"
+
def __init__(
self,
- project_dir: str,
- param_grid: dict[str, np.ndarray | list | tuple],
- seed_ranges: dict[str, np.ndarray | list | tuple],
+ project_dir: str = "./",
+ param_grid: dict[str, np.ndarray | list | tuple] = {},
+ seed_ranges: dict[str, np.ndarray | list | tuple] = {},
model_search_config: dict[str, Callable | dict] | None = None,
loss_fn: Callable = asymmetric_mae, # Fits to the floor (\approx. lower bound) of the distribution $L(N, D)$
weight_fn: Callable | None = None, # You nay weight loss prediction errors with any input
@@ -93,7 +95,28 @@ def __init__(
# input validation
ParamGrid(**param_grid)
- SeedRanges(**seed_ranges)
+ if seed_ranges:
+ SeedRanges(**seed_ranges)
+ # Convert dict to AttrDict for easy access
+ seed_ranges = AttrDict(seed_ranges)
+
+ """Initialize configurations"""
+ # Seed
+ self.seed_ranges = AttrDict(
+ # User-specified
+ C=[float(c) for c in seed_ranges.C], # tuple/list of large integers (>2 ** 63) can result in errors
+ N_to_D=seed_ranges.N_to_D,
+ # Pre-compute the bounds of allocations for the seed models
+ N=[
+ np.sqrt(seed_ranges.C[0] / (6 * seed_ranges.N_to_D[1])), # lower bound
+ np.sqrt(seed_ranges.C[1] / (6 * seed_ranges.N_to_D[0])), # upper bound
+ ],
+ D=[
+ np.sqrt(seed_ranges.C[0] * seed_ranges.N_to_D[0] / 6), # lower bound
+ np.sqrt(seed_ranges.C[1] * seed_ranges.N_to_D[1] / 6), # upper bound
+ ],
+ )
+
if model_search_config:
ModelSearchConfig(**model_search_config)
else:
@@ -110,26 +133,6 @@ def __init__(
if weight_fn and not callable(loss_fn):
raise TypeError("`weight_fn` must be callable or None")
- # Convert dict to AttrDict for easy access
- seed_ranges = AttrDict(seed_ranges)
-
- """Initialize configurations"""
- # Seed
- self.seed_ranges = AttrDict(
- # User-specified
- C=[float(c) for c in seed_ranges.C], # tuple/list of large integers (>2 ** 63) can result in errors
- N_to_D=seed_ranges.N_to_D,
- # Pre-compute the bounds of allocations for the seed models
- N=[
- np.sqrt(seed_ranges.C[0] / (6 * seed_ranges.N_to_D[1])), # lower bound
- np.sqrt(seed_ranges.C[1] / (6 * seed_ranges.N_to_D[0])), # upper bound
- ],
- D=[
- np.sqrt(seed_ranges.C[0] * seed_ranges.N_to_D[0] / 6), # lower bound
- np.sqrt(seed_ranges.C[1] * seed_ranges.N_to_D[1] / 6), # upper bound
- ],
- )
-
# Fit
self.model_search_config = model_search_config
self.param_grid = param_grid
@@ -192,8 +195,8 @@ def from_config(cls, config_path: str, **kwargs) -> Chinchilla:
def _create_shortcuts(self) -> None:
"""Sets up shortcut methods."""
# Bypass instance methods to class methods; override the class method once constructed
- self.allocate_compute = lambda C: Chinchilla.allocate_compute(C, self.get_params())
- self.predict_loss = lambda N, D: Chinchilla.predict_loss(N, D, self.get_params())
+ self.allocate_compute = lambda C: Chinchilla.allocate_compute(C, self.params)
+ self.predict_loss = lambda N, D: Chinchilla.predict_loss(N, D, self.params)
# Submodules; consult each class for what it does
self.append = self.database.append
@@ -231,6 +234,9 @@ def seed(self) -> tuple[tuple[int, float], dict[str, int] | None]:
Raises:
ValueError: If a valid configuration could not be found after a certain number of trials.
"""
+ if not hasattr(self, "seed_ranges"):
+ raise ValueError("When sampling seeds, you need to specify `seed_ranges` argment at initialization")
+
get_model_config = self.model_search_config is not None
_max_iters = 2**10
@@ -249,22 +255,24 @@ def seed(self) -> tuple[tuple[int, float], dict[str, int] | None]:
else:
raise ValueError(f"We could not find a valid configuration in {_max_iters} trials.")
- self.logger.debug(f"[{ordinal(len(self.database.df)+1)}]\t{C:.2e} FLOPs => {N:.2e} params * {D:.2e} samples")
+ self.logger.debug(f"[{ordinal(len(self.database.df) + 1)}]\t{C:.2e} FLOPs => {N:.2e} params * {D:.2e} samples")
return (N, D), model_config
def fit(self, parallel: bool = True, simulation: bool = False) -> None:
"""
- Uses [L-BFGS optimization (SciPy implementation)](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html)
+ Uses [BFGS optimization (SciPy implementation)](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-bfgs.html)
to find the best-fitting parameters for the scaling law based on the collected data.
+ Note that this choice of optimizer is different from the original paper, which instead used L-BFGS (without explicit bounds).
+ We choose BFGS for its absolute advantage in terms of accuracy and efficiency (see [this discussion](https://github.com/kyo-takano/chinchilla/blob/master/docs/changes.md#4-algorithm-l-bfgs-b--bfgs) for more details)
Args:
- parallel (bool, optional): Whether to run L-BFGS optimization over the initialization grid in parallel processing.
+ parallel (bool, optional): Whether to run BFGS optimization over the initialization grid in parallel processing.
simulation (bool, optional): Indicates whether the fitting is part of a simulation. Defaults to False.
Raises:
ValueError: If there are not enough data points to perform the fitting.
- TypeError: If the numerical precision is insufficient for the L-BFGS algorithm.
+ TypeError: If the numerical precision is insufficient for the BFGS algorithm.
"""
_df = self.database.df.copy()
@@ -275,7 +283,9 @@ def fit(self, parallel: bool = True, simulation: bool = False) -> None:
if DTYPE().itemsize < 8: # In bytes
raise TypeError(
- "The current operation requires a numerical precision of at least 64-bit as used in the L-BFGS algorithm. Lower precisions such as np.float32 or below are not supported for this operation. Please ensure you're using np.float64 or higher precision to avoid this error."
+ "The current operation requires a numerical precision of at least 64-bit as used in the BFGS algorithm. "
+ "Lower precisions such as np.float32 or below are not supported for this operation. "
+ "Please ensure you're using np.float64 or higher precision to avoid this error."
)
# Pre-compute the series repeatedly accessed by `self._evaluate_params`
@@ -283,7 +293,7 @@ def fit(self, parallel: bool = True, simulation: bool = False) -> None:
# raise NotImplementedError("When specifying `weight_fn`, you are expected to edit the source code by deleting this error and specify how to compute yourself.")
self.logger.warning(
"`weight_fn` receives `cc.dataframe.df.C` as its default argument. "
- "If you want to weigh L-BFGS losses by something else, please edit the source code."
+ "If you want to weight BFGS losses by something else, please edit the source code."
)
weights = self.weight_fn(_df.C.values.astype(DTYPE))
weights /= weights.mean()
@@ -310,23 +320,11 @@ def fit(self, parallel: bool = True, simulation: bool = False) -> None:
initial_guesses = list(itertools.product(*self.param_grid.values()))
initial_guesses /= self._autoscale_range
- def _optimize_params(i):
- x0 = initial_guesses[i]
- # res = sciop.minimize(self._evaluate_params, x0, method="L-BFGS-B", tol=1e-7) # Note: `tol` -> `ftol`
- # lbfgs_loss = res.fun
- # if np.isfinite(lbfgs_loss):
- # return res.x * self._autoscale_range, lbfgs_loss
- # https://github.com/scipy/scipy/blob/v1.12.0/scipy/optimize/_lbfgsb_py.py
- x, lbfgs_loss, _ = sciop.fmin_l_bfgs_b(
- self._evaluate_params,
- x0,
- approx_grad=True,
- maxiter=1_000_000,
- maxfun=1_000_000,
- # Default values generally perform fine
- )
- if np.isfinite(lbfgs_loss):
- return x, lbfgs_loss
+ def _optimize_params(x0):
+ result = sciop.minimize(self._evaluate_params, x0, method=self.algorithm)
+ L_bfgs = result.fun
+ if np.isfinite(L_bfgs):
+ return result.x, L_bfgs
with multiprocessing.Pool(os.cpu_count()) as pool:
with Progress(
@@ -344,19 +342,19 @@ def _optimize_params(i):
results = []
if parallel:
self.logger.debug(f"{os.cpu_count()=}")
- for res in pool.imap_unordered(_optimize_params, range(len(initial_guesses))):
+ for res in pool.imap_unordered(_optimize_params, initial_guesses):
if res:
results.append(res)
progress.update(task, advance=1.0)
else:
- for i in range(len(initial_guesses)):
- res = _optimize_params(i)
+ for x0 in initial_guesses:
+ res = _optimize_params(x0)
if res:
results.append(res)
progress.update(task, advance=1.0)
if not results:
- raise ValueError("No valid result from L-BFGS. `loss_fn` you have specified is possibly broken.")
+ raise ValueError("No valid result from BFGS. `loss_fn` you have specified is possibly broken.")
best_fit = min(results, key=lambda x: x[1])
self.E, self.A, self.B, self.alpha, self.beta = best_fit[0] * self._autoscale_range
@@ -368,7 +366,7 @@ def _optimize_params(i):
N, D = _df.N.values, _df.D.values
# N, D = N.astype(float), D.astype(float) # In case N and D were type object in pandas/numpy
y_pred = self.predict_loss(N, D)
- self.visualizer.LBFGS(y_pred, _df.loss.values, simulation=simulation)
+ self.visualizer.optim(y_pred, _df.loss.values, simulation=simulation)
self.logger.info(
f"Loss predictor:\n\n L(N, D) = {self.E:#.4g} + {self.A:#.4g} / (N ^ {self.alpha:#.4g}) + {self.B:#.4g} / (D ^ {self.beta:#.4g})\n"
@@ -434,7 +432,11 @@ def scale(
if C is None:
# Use the preset `scaling_factor` if not overridden
scaling_factor = scaling_factor or self.scaling_factor
- C = max(self.seed_ranges.C[1], int(self.database.df.C.max())) * scaling_factor
+ if hasattr(self, "seed_ranges"):
+ C = max(self.seed_ranges.C[1], int(self.database.df.C.max())) * scaling_factor
+ else:
+ # You can only use the existing max when `seed_ranges` is not specified
+ C = int(self.database.df.C.max()) * scaling_factor
N, D = self.allocate_compute(C)
if get_model_config:
@@ -446,9 +448,13 @@ def scale(
else:
model_config = None
- self.logger.info(f"[{ordinal(len(self.database.df)+1)}]\t{C:.2e} FLOPs => {N:.2e} params * {D:.2e} samples")
- self.plot(next_point=dict(C=C, N=N, D=D), simulation=simulation)
-
+ self.logger.info(f"[{ordinal(len(self.database.df) + 1)}]\t{C:.2e} FLOPs => {N:.2e} params * {D:.2e} samples")
+ self.plot(
+ next_point=dict(
+ C=np.array(C, dtype=np.float64), N=np.array(N, dtype=np.float64), D=np.array(D, dtype=np.float64)
+ ),
+ simulation=simulation,
+ )
return (N, D), model_config
def step(
@@ -471,7 +477,7 @@ def step(
Args:
num_seeding_steps (int, optional): The threshold number of seed training runs before starting to scale the compute budget.
- parallel (bool, optional): Whether to run L-BFGS optimization over the initialization grid in parallel processing. To be passed to `fit`.
+ parallel (bool, optional): Whether to run BFGS optimization over the initialization grid in parallel processing. To be passed to `fit`.
simulation (bool, optional): Indicates whether the scaling is part of a simulation. Defaults to False.
**scale_kwargs: Keyword arguments to be passed to `scale` (`scaling_factor` and `C`).
@@ -673,9 +679,9 @@ def predict_loss(cls, N: np.ndarray | float, D: np.ndarray | float, params: dict
return E + np.exp(log_term_2nd) + np.exp(log_term_3rd)
- def _evaluate_params(self, x) -> np.ndarray:
+ def _evaluate_params(self, x) -> float:
"""
- Internal method to compute the loss for the L-BFGS algorithm.
+ Internal method to compute the loss for the BFGS algorithm.
This method evaluates the loss function for a given set of parameters during the optimization process.
@@ -694,8 +700,8 @@ def _evaluate_params(self, x) -> np.ndarray:
f"This was possibly because `loss_fn` you specified is compatible with type `{DTYPE}`"
)
- # Scipy/Fortran implementation of LBFGS casts `x0` to float64 internally, so recover here.
- # Invert autoscaling & decompose
+ # Scipy/Fortran implementation of BFGS casts `x0` to float64 internally, so recover here.
+ # Unscale & decompose
E, a, b, alpha, beta = x.astype(DTYPE) * self._autoscale_range
# Ensure the log scale for `a` and `b` but `E`.
@@ -718,21 +724,34 @@ def _evaluate_params(self, x) -> np.ndarray:
if self.weight_fn:
losses = losses * self._const["weights"]
- return np.mean(losses)
+ return float(np.mean(losses))
- def get_params(self) -> dict:
+ @property
+ def params(self) -> dict:
+ """
+ A proxy to get scaling law parameters by internally calling Chinchilla.get_params()
"""
- Returns a dictionary of estimated parameters describing the scaling law / parametric loss estimator.
+ return self.get_params()
+
+ def get_params(self) -> dict[str, float]:
+ """
+ Returns a dictionary of the scaling law parameters.
Returns:
- float: The computed loss value.
+ dict: A dictionary of the optimized parameters
Raises:
ValueError: If the scaling law parameters have not been set as attributes.
"""
if not all(hasattr(self, param) for param in ["E", "A", "B", "alpha", "beta"]):
raise ValueError("You must call `fit` before training a model with scaled compute.")
- return {"E": self.E, "A": self.A, "B": self.B, "alpha": self.alpha, "beta": self.beta}
+ return {
+ "E": float(self.E),
+ "A": float(self.A),
+ "B": float(self.B),
+ "alpha": float(self.alpha),
+ "beta": float(self.beta),
+ }
def report(self, plot: bool = True) -> None:
"""
@@ -751,7 +770,7 @@ def report(self, plot: bool = True) -> None:
raise ValueError("You must call `fit` before generating a report.")
self.logger.info("Estimated scaling law parameters:")
- for param, value in self.get_params().items():
+ for param, value in self.params.items():
self.logger.info(f" - {param}: {value}")
if len(self.database.df):
self.logger.info("Goodness of fit:")
@@ -766,3 +785,98 @@ def report(self, plot: bool = True) -> None:
if plot:
self.logger.info("Landscape visualization:")
self.plot()
+
+ def sweep_param_grid(self, plot=True, img_name="sweep_param_grid"):
+ """Utility method to visualize the 1D landscape of minimum loss by each parameter value in grid"""
+
+ _df = self.database.df.copy()
+ if not len(_df):
+ raise ValueError("You do not have any training runs yet.")
+ # Pre-compute the series repeatedly accessed by `self._evaluate_params`
+ if self.weight_fn:
+ # raise NotImplementedError("When specifying `weight_fn`, you are expected to edit the source code by deleting this error and specify how to compute yourself.")
+ self.logger.warning(
+ "`weight_fn` receives `cc.dataframe.df.C` as its default argument. "
+ "If you want to weight BFGS losses by something else, please edit the source code."
+ )
+ weights = self.weight_fn(_df.C.values.astype(DTYPE))
+ weights /= weights.mean()
+ else:
+ weights = None
+
+ self._const = dict(
+ log_N=np.log(_df.N.values.astype(DTYPE)),
+ log_D=np.log(_df.D.values.astype(DTYPE)),
+ y_true=_df.loss.values.astype(DTYPE),
+ weights=weights,
+ )
+
+ # The absolute value range affects the differential optimization
+ self._autoscale_range = np.array(list(map(np.ptp, self.param_grid.values())))
+ # In case of any axis with a single initial value:
+ self._autoscale_range[self._autoscale_range == 0] = 1.0
+
+ initial_guesses = list(itertools.product(*self.param_grid.values()))
+ initial_guesses /= self._autoscale_range
+
+ global _eval_fn # for parallel
+
+ def _eval_fn(x0):
+ loss = self._evaluate_params(x0)
+ return (x0, loss) if np.isfinite(loss) else (x0, float("inf"))
+
+ with multiprocessing.Pool(os.cpu_count()) as pool:
+ with Progress(
+ SpinnerColumn(),
+ TextColumn("[progress.description]{task.description}"),
+ BarColumn(),
+ TimeElapsedColumn(),
+ "/",
+ TimeRemainingColumn(),
+ disable=self.logger.getEffectiveLevel() > 30,
+ ) as progress:
+ task = progress.add_task("Sweeping the parameter grid", total=len(initial_guesses))
+ results = []
+ for res in pool.imap_unordered(_eval_fn, initial_guesses):
+ results.append(res)
+ progress.update(task, advance=1.0)
+
+ best_fit, best_loss = min(results, key=lambda x: x[1])
+
+ if plot:
+ import matplotlib.pyplot as plt
+
+ # Create a subplot for each parameter
+ _, axes = plt.subplots(1, 5, figsize=(11, 3), sharey=True)
+ param_names = list(self.param_grid.keys())
+
+ # For each parameter:
+ ylim = [-float("inf"), float("inf")]
+ for param_idx, (param_name, ax) in enumerate(zip(param_names, axes)):
+ # Get unique values in the grid
+ v_unique = np.unique([x[param_idx] for x, _ in results])
+
+ # Minimizer for each unique value
+ min_losses = []
+ for val in v_unique:
+ losses = [loss for (x, loss) in results if x[param_idx] == val]
+ min_losses.append(min(losses))
+
+ # Plot parameter value vs minimum loss
+ ax.plot(v_unique * self._autoscale_range[param_idx], min_losses, "b-")
+ ax.set_xlabel(param_name)
+ if max(min_losses) < ylim[1]:
+ ylim[1] = max(min_losses)
+ if ylim[0] < min(min_losses):
+ ylim[0] = min(min_losses)
+ for x0 in self.param_grid[param_name]:
+ ax.axvline(x0, ls=":", c="tab:gray", lw=1)
+ ax.axhline(best_loss, ls="--", c="tab:red", lw=1 / 2)
+ ax.set_ylim(*[ylim[0] / 1.05, ylim[1] * 1.05])
+ plt.suptitle("1D loss landscape by initial parameter values")
+ plt.tight_layout()
+ plt.savefig(os.path.join(self.project_dir, img_name + ".png"))
+ plt.show()
+ plt.close()
+
+ return best_fit, best_loss
diff --git a/chinchilla/database.py b/chinchilla/database.py
index 73302f6..72a088d 100755
--- a/chinchilla/database.py
+++ b/chinchilla/database.py
@@ -55,6 +55,8 @@ def __init__(
)
# We define an empty database for when referencing the number of existing data points
self.df = pd.DataFrame([], columns=columns)
+ # numbers like FLOPS can be too large to be represented as int, which in turn gets interpreted as a string object
+ self.df.C = self.df.C.astype(float)
def append(self, **result: dict[str, float]) -> None:
"""
@@ -77,7 +79,6 @@ def append(self, **result: dict[str, float]) -> None:
result["C"] = 6 * result["N"] * result["D"]
for k in ["C", "N", "D"]:
result[k] = round(result[k]) # This helps prevent scientific notation of large values
-
# Collect all columns added by the user
cols_additional = [c for c in result.keys() if c not in self.df.columns]
record = pd.DataFrame([result], columns=self.df.columns.tolist() + cols_additional)
diff --git a/chinchilla/simulator.py b/chinchilla/simulator.py
index 7aa0603..3ddf31a 100755
--- a/chinchilla/simulator.py
+++ b/chinchilla/simulator.py
@@ -134,7 +134,6 @@ def __call__(
def _pseudo_training_run(self) -> None:
"""Perform a pseudo training run and record the results in the database."""
(N, D), _ = self.step(simulation=True)
-
# Get a *hypothetical* lowest loss you can achieve with N and D
pseudo_loss = sum(
[
diff --git a/chinchilla/visualizer.py b/chinchilla/visualizer.py
index f01a217..1cf6203 100755
--- a/chinchilla/visualizer.py
+++ b/chinchilla/visualizer.py
@@ -6,6 +6,7 @@
import numpy as np
import pandas as pd
import seaborn as sns
+import warnings
from ._logger import get_logger
@@ -17,7 +18,7 @@
class Visualizer:
"""
`Visualizer` includes methods for plotting the estimated loss gradient, the efficient frontier,
- and L-BFGS optimization results. It helps in understanding the distribution and relationships between
+ and BFGS optimization results. It helps in understanding the distribution and relationships between
compute resources, model parameters, and data samples, and highlights efficient allocation frontiers
and seed regimes.
@@ -55,7 +56,7 @@ def plot(
with the loss function and highlights efficient allocation frontiers and seed regimes.
**Example output**:
- ![](../examples/efficientcube-1e15_1e16/parametric_fit.png)
+ ![](https://github.com/kyo-takano/chinchilla/blob/master/docs/imgs/parametric_fit.png)
Args:
cc: A Chinchilla instance with a Database of training runs and scaling law parameters if estimated.
@@ -101,7 +102,9 @@ def plot(
loss_range = loss_max - loss_min
if self._next_point:
loss_min = min(loss_min, cc.L(self._next_point["N"], self._next_point["D"]))
- iso_losses = np.linspace(loss_min - loss_range * margin, loss_max + loss_range * margin, 32)
+ iso_losses = np.linspace(loss_min - loss_range * margin, loss_max + loss_range * margin, 32).astype(
+ np.float64
+ ) # cast: avoid potential error with float128
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
fig.tight_layout(pad=4.0, w_pad=3.0)
@@ -112,7 +115,7 @@ def plot(
# Get the highest value to include for each axis
x_max, y_max = [
max(
- self.cc.seed_ranges[k][1],
+ self.cc.seed_ranges[k][1] if hasattr(self.cc, "seed_ranges") else self.cc.database.df[k].max(),
self.cc.database.df[k].max(),
self._next_point[k] if self._next_point else -float("inf"),
)
@@ -152,20 +155,20 @@ def plot(
self.logger.info(f"Image saved to [u]{img_filepath}[/]")
- def LBFGS(
+ def optim(
self,
y_pred: np.ndarray,
y_true: np.ndarray,
C: np.ndarray | None = None,
simulation: bool = False,
- img_name: str = "LBFGS",
+ img_name: str = "optim",
) -> None:
"""
- Plots the results of L-BFGS optimization, including the loss history and prediction accuracy.
+ Plots the results of optimization, including the loss history and prediction accuracy.
This method visualizes the predicted values versus the true labels and the error distribution.
**Example output**:
- ![](../examples/efficientcube-1e15_1e16/LBFGS.png)
+ ![](https://github.com/kyo-takano/chinchilla/blob/master/docs/imgs/optim--asymmetric.jpg)
Args:
y_pred (np.ndarray): Predicted values by the model.
@@ -223,7 +226,7 @@ def LBFGS(
axs[1].set_yscale("log")
axs[1].set_title("Compute and absolute error")
- plt.suptitle("L-BFGS results")
+ plt.suptitle("Optimization result")
plt.savefig(os.path.join(self.project_dir, img_name + ".png"))
plt.show()
plt.close()
@@ -231,13 +234,15 @@ def LBFGS(
def _plot_loss_gradient(self, ax, x, y, iso_losses, y_max):
"""Helper method to plot the loss gradient."""
# / 1 for converting to float when int
- log_ymin = np.log10(self.cc.seed_ranges[y][0] / 1)
+ log_ymin = np.log10(
+ self.cc.seed_ranges[y][0] / 1 if hasattr(self.cc, "seed_ranges") else self.cc.database.df[y].min()
+ )
log_ymax = np.log10(y_max / 1)
log_ymin -= (log_ymax - log_ymin) * PADDING
log_ymax += (log_ymax - log_ymin) * PADDING
y_values = np.logspace(log_ymin, log_ymax, 1000, dtype=np.double)
- assert not np.isnan(y_values).sum(), (f"{100*np.isnan(y_values).mean()}% NaN:", y_values)
+ assert not np.isnan(y_values).sum(), (f"{100 * np.isnan(y_values).mean()}% NaN:", y_values)
for j, L in enumerate(iso_losses):
if y == "N":
N = y_values
@@ -260,23 +265,24 @@ def _plot_loss_gradient(self, ax, x, y, iso_losses, y_max):
ax.plot(x_values, y_values, c=self.cmap(j / len(iso_losses)), zorder=1)
def _shadow_seed_regime(self, ax, x, y, resolution: int = 100):
- """Helper method to fill the seed regime with gray."""
- if x == "C":
- c = np.logspace(*np.log10(self.cc.seed_ranges[x]), resolution)
- if y == "N":
- y_lower = np.sqrt(c / (6 * self.cc.seed_ranges.N_to_D[1]))
- y_upper = np.sqrt(c / (6 * self.cc.seed_ranges.N_to_D[0]))
- else: # y == "D"
- y_lower = np.sqrt(c * self.cc.seed_ranges.N_to_D[0] / 6)
- y_upper = np.sqrt(c * self.cc.seed_ranges.N_to_D[1] / 6)
- x_values = c
- else: # x in ["N", "D"]
- n = np.logspace(*np.log10(self.cc.seed_ranges[x]), resolution)
- y_lower = np.maximum(self.cc.seed_ranges["C"][0] / (6 * n), self.cc.seed_ranges.N_to_D[0] * n)
- y_upper = np.minimum(self.cc.seed_ranges["C"][1] / (6 * n), self.cc.seed_ranges.N_to_D[1] * n)
- x_values = n
-
- ax.fill_between(x_values, y_lower, y_upper, color="silver", alpha=0.5, zorder=0, label="Seed")
+ """Helper method to fill the seed regime with gray. Executed only if the Chinchilla instance has `seed_ranges` specified."""
+ if hasattr(self.cc, "seed_ranges"):
+ if x == "C":
+ c = np.logspace(*np.log10(self.cc.seed_ranges[x]), resolution)
+ if y == "N":
+ y_lower = np.sqrt(c / (6 * self.cc.seed_ranges.N_to_D[1]))
+ y_upper = np.sqrt(c / (6 * self.cc.seed_ranges.N_to_D[0]))
+ else: # y == "D"
+ y_lower = np.sqrt(c * self.cc.seed_ranges.N_to_D[0] / 6)
+ y_upper = np.sqrt(c * self.cc.seed_ranges.N_to_D[1] / 6)
+ x_values = c
+ else: # x in ["N", "D"]
+ n = np.logspace(*np.log10(self.cc.seed_ranges[x]), resolution)
+ y_lower = np.maximum(self.cc.seed_ranges["C"][0] / (6 * n), self.cc.seed_ranges.N_to_D[0] * n)
+ y_upper = np.minimum(self.cc.seed_ranges["C"][1] / (6 * n), self.cc.seed_ranges.N_to_D[1] * n)
+ x_values = n
+
+ ax.fill_between(x_values, y_lower, y_upper, color="silver", alpha=0.5, zorder=0, label="Seed")
def _adjust_subplot(self, ax, x, y, x_max, y_max):
"""Adjusts the subplot configurations and return the bound of values."""
@@ -284,12 +290,10 @@ def _adjust_subplot(self, ax, x, y, x_max, y_max):
limits_by_k = {}
for k in [x, y]:
# Converting to `float` in case of int dtype (`np.log` cannot intake astronomically large integers)
+ min_value = self.cc.seed_ranges[k][0] if hasattr(self.cc, "seed_ranges") else self.cc.database.df[k].min()
max_value = {x: x_max, y: y_max}[k] / 1
- log_range = np.log(max_value) - np.log(self.cc.seed_ranges[k][0])
- limits_by_k[k] = (
- self.cc.seed_ranges[k][0] * np.exp(-log_range * PADDING),
- max_value * np.exp(log_range * PADDING),
- )
+ log_range = np.log(max_value) - np.log(min_value)
+ limits_by_k[k] = (min_value * np.exp(-log_range * PADDING), max_value * np.exp(log_range * PADDING))
# Apply the bounds
xlim, ylim = limits_by_k[x], limits_by_k[y]
@@ -361,3 +365,12 @@ def _add_colorbar_to_plot(self, fig, axes, iso_losses):
cbar.set_array([iso_losses])
fig.colorbar(cbar, cax=cax)
cax.set_ylabel("Loss", rotation=270, labelpad=16)
+
+ def LBFGS(self, *args, **kwargs):
+ """Deprecated function. Please use optim() instead."""
+ warnings.warn(
+ "`Visualizer.LBFGS(...)` is deprecated and will be removed in a future version. Use `Visualizer.optim(...)` instead.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return self.optim(*args, **kwargs)
diff --git a/docs/TIPS.md b/docs/TIPS.md
index 1477fb6..d2ad703 100644
--- a/docs/TIPS.md
+++ b/docs/TIPS.md
@@ -1,81 +1,104 @@
# Tips / Best Practices
-Here are a few tips and best practices for both using `chinchilla` and training large-scale NNs in general.
+Here are a couple of tips and best practices for using `chinchilla`.
-## chinchilla-specific
+## 1. Be meticulous with `param_grid`
-### 1. Be specific on `param_grid`
+To fit a loss predictor $L(N, D | A, B, \alpha, \beta)$ (`Chinchilla.fit`) based on existing training runs, defining the `param_grid` of initial values is critical.
+The parametric model is sensitive to the initial distribution, and a well-chosen grid can significantly reduce the risk of underfitting or poor convergence.
-To fit a loss predictor $L(N, D | A, B, \alpha, \beta)$ (`Chinchilla.fit`) on existing training runs,
-it is crucial to define a `param_grid` of initial values carefully.
-The optimization of these values through L-BFGS aims to align predicted losses $\hat{L_{i}}$ closely with actual losses $L_{i}$, and given the sensitivity of the optimization algorithm, a tiny adjustment of a value in the initialization grid can significantly impact the result.
+### Example: Original Initialization Grid
-To mitigate estimation instability:
+
-- Utilize prior knowledge of expected losses for given $N$ and/or $D$
-- If no clue, inform your parameter grid from seed training runs
+The initialization grid used in the original Chinchilla study looked like this:
-Prior knowledge of expected losses for a given $N$ and/or $D$ can guide you in setting realistic upper and lower bounds for these parameters, enhancing the precision of your grid.
-For example, the cross-entropy loss can go below 1.5 for an LM with 32000 vocabularies.
-Narrowing down the search space like this will allow for more fine-grained exploration and better CPU time allocation.
+> ```python
+> """This grid matches the range used in the original paper."""
+> num_slices = 16 # Resolution increased from 1,500 -> 1,048,576 combinations for a finer sweep
+> cc = Chinchilla(
+> "./",
+> param_grid = dict(
+> e=np.linspace(-1, 1, num_slices),
+> a=np.linspace(0, 25, num_slices),
+> b=np.linspace(0, 25, num_slices),
+> alpha=np.linspace(0, 2, num_slices),
+> beta=np.linspace(0, 2, num_slices)
+> )
+> ...
+> )
+> cc.sweep_param_grid() # Visualizes the loss landscape for the grid.
+> ```
+>
+> ![Loss landscape of original initialization](imgs/sweep_param_grid.original.png)
-### 2. Keep `scaling_factor` moderate
+As seen, the loss landscape has sharp minima, making it difficult to converge to a good optimum unless any of the initial guesses happen to be very close to them. This is an example of ***poor initialization***.
-Scaling compute according to the loss predictor involves ***extrapolation*** beyond the FLOPs regime used in fitting the predictor.
-To avoid overstepping, it's advisable to:
+### Improving Initialization
-- Incrementally scale up compute,
-- Progressively update the scaling law, and
-- Aim for a scaling factor around 2.0, dedicating half of your total budget to estimate the scaling law and the other half for the final model.
+To address this, you can refine the parameter search space based on existing training data and prior observations. For example:
-### 3. Beware of "failure modes"
+- Reduce the range for other parameters based on the stability of their behavior.
-You may encounter different types of "failures" when fitting the loss predictor,
-and they often happen when you don't have a good configuration.
+- Narrow the range for $E$ to a region around the observed minimum loss.
-- **Insufficient compute for seed models**
+ This by definition sets the **upper bound** for the irreducible error. In the original, the $e=\log(E)$ range corresponds to [0.367879441, 2.71828183] in linear space, which is largely missing the point
+
+Here’s an improved grid based on this strategy:
- ![](./imgs/LBFGS--seeds-too-small.png)
+```python
+param_grid = dict(
+ E=np.linspace(1.4, 2.0774, num_slices), # 2.0774: Observed minimum irreducible error
+ a=np.linspace(1, 10, num_slices),
+ b=np.linspace(1, 10, num_slices),
+ alpha=np.linspace(0.1, 0.7, num_slices),
+ beta=np.linspace(0.1, 0.7, num_slices)
+)
+```
-- **Poor fit from L-BFGS optimization**
+And you get:
- ![](./imgs/LBFGS--underfit.png)
+![Improved loss landscape](imgs/sweep_param_grid.improved.png)
-## General Training Advice
+The minima are smoother and more stable, allowing for easier convergence during optimization.
-### Basics
+As a matter of fact, this technique is so effective that even a naive grid search can work almost as good as L-BFGS:
-- [Mixed Precision (bf16/fp16)](https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html)
-- [Gradient Accumulation](https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation) if a desired size of batches don't fit on device(s)
-- [Learning rate scheduling](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate)
-- A rule of thumb: larger networks often require smaller learning rates to prevent divergence during training
+
+
+
+
+ ➡️
+
+
+
+
-### Hyperparameter Optimization
+## 2. Keep `scaling_factor` moderate
+
+Scaling compute according to the loss predictor involves ***extrapolation*** beyond the FLOPs regime used for fitting the predictor.
+To avoid overstepping, it's advisable to:
-- [µP/µTransfer](https://github.com/microsoft/mup): Recommended
-- [Optuna](https://optuna.org/), [Hyperopt](https://hyperopt.github.io/hyperopt/), etc.
+- **Incrementally scale compute** rather than making large jumps.
+- ***Continuously update*** the scaling law as a new data point becomes available.
-### GPU
+As a rule of thumb, I would suggest using`scaling_factor=2.0` as a good starting point.
+This approach balances the compute budget by dedicating roughly half of it to scaling law estimation and the other half to final model training.
+
+## 3. Beware of "failure modes"
+
+When fitting the loss predictor, several common failure modes may arise. These are often tied to poor configurations, including;
+
+- **Insufficient compute for seed models**
-- [`torch.compile`](https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html)
-- [`triton`](https://github.com/openai/triton)
-- You might also want to learn to code custom CUDA kernels
+ ![Insufficient compute failure](imgs/optim--seeds-too-small.jpg)
-### Distributed Training
+- **Underfitting due to poor optimization**
-- [torch.distributed](https://pytorch.org/tutorials/beginner/dist_overview.html): Recommended if you need more than one GPU and are new to the concept of parallelism.
-- [DeepSpeed](https://www.deepspeed.ai/)
- - [3D Parallelism](https://www.deepspeed.ai/tutorials/pipeline/)
- - [ZeRO](https://www.deepspeed.ai/tutorials/zero/)
-- [Zero Bubble](https://github.com/sail-sg/zero-bubble-pipeline-parallelism): SOTA multi-GPU utilization rate
+ ![Underfitting failure](imgs/optim--underfit.jpg)
-### Transformers / LLM
+---
-- [Flash Attention](https://github.com/Dao-AILab/flash-attention)
-- [Megatron-LM](https://github.com/microsoft/Megatron-LM)
- - [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed)
-- [Mamba](https://github.com/state-spaces/mamba): State-Space Model for LM
-- Depth-to-Width ratio: As the number of parameters $N$ increases, model depth (number of layers) tends to increase as well, with studies such as [Limits to Depth Efficiencies of Self-Attention (Levine, et al., 2020)](https://proceedings.neurips.cc/paper/2020/hash/ff4dfdf5904e920ce52b48c1cef97829-Abstract.html) suggesting this trend continues up to 48 layers. However, shallower and wider models may be preferred in some cases due to their faster runtime achieved through more parallel operations.
-- Batch Size: When resources allow, batch sizes can be scaled up to a million tokens or more, which can lead to more efficient training for large models due to better GPU utilization and reduced communication overhead in distributed settings.
-- For enthusiasts interested in a more hands-on approach, [nanoGPT](https://github.com/karpathy/nanoGPT/)
- offers a hackable codebase for experimenting with GPT models.
+> [!NOTE]
+> *The section "General Training Advice" has been removed from this document. In case you still need it, you can find it [here](https://github.com/kyo-takano/chinchilla/blob/3db6ab51a0ceb82855cb66da41f0b8ab663b3857/docs/TIPS.md#general-training-advice)*
diff --git a/docs/api-reference.md b/docs/api-reference.md
index df798fc..1565386 100644
--- a/docs/api-reference.md
+++ b/docs/api-reference.md
@@ -1,4 +1,4 @@
-# API Reference
+# 🐭 API Reference
# `chinchilla.core.Chinchilla`
@@ -6,20 +6,26 @@
class Chinchilla()
```
-Estimates the scaling law for a deep learning task. Provides functionalities to:
+Estimates the scaling law for a deep learning task.
+Provides functionalities to:
1. Sample models from a specified "seed" regime.
2. Fit the loss predictor $L(N, D)$.
3. Suggest an allocation of scaled compute.
-This module includes the `Chinchilla` class, which provides methods for sampling model configurations, fitting the parametric loss predictor, suggesting allocations for scaled compute budgets, etc. It operates in a numerical precision of **128-bit** by default and integrates with [`chinchilla.Database`](#chinchilladatabaseDatabase) and [`chinchilla.Visualizer`](#chinchillavisualizerVisualizer) for storing and plotting data.
+This module includes the `Chinchilla` class, which provides methods for sampling model configurations,
+fitting the parametric loss predictor, suggesting allocations for scaled compute budgets, etc.
+It operates in a numerical precision of **128-bit** by default and integrates with
+[`chinchilla.Database`](#chinchilladatabaseDatabase) and
+[`chinchilla.Visualizer`](#chinchillavisualizerVisualizer)
+for storing and plotting data.
### `__init__`
```python
-def __init__(project_dir: str,
- param_grid: dict[str, np.ndarray | list | tuple],
- seed_ranges: dict[str, np.ndarray | list | tuple],
+def __init__(project_dir: str = "./",
+ param_grid: dict[str, np.ndarray | list | tuple] = {},
+ seed_ranges: dict[str, np.ndarray | list | tuple] = {},
model_search_config: dict[str, Callable | dict] | None = None,
loss_fn: Callable = asymmetric_mae,
weight_fn: Callable | None = None,
@@ -40,7 +46,8 @@ Initializes a Chinchilla instance with the given parameters and sets up the scal
- `weight_fn` _Callable | None, optional_ - A function to weight loss prediction errors. Defaults to None.
- `num_seeding_steps` _int | None, optional_ - The number of seeding steps to perform. Defaults to None.
- `scaling_factor` _float | None, optional_ - The scaling factor to be used when scaling up compute. Defaults to None.
-- `log_level` _int | str, optional_ - Specifies the threshold for logging messages. A value of 30 suppresses standard messages while any larger values hide all messages entirely. Defaults to 20 (`logging.INFO`).
+- `log_level` _int | str, optional_ - Specifies the threshold for logging messages.
+ A value of 30 suppresses standard messages while any larger values hide all messages entirely. Defaults to 20 (`logging.INFO`).
**Raises**:
@@ -76,7 +83,8 @@ Constructs a Chinchilla instance from a configuration file, with the option to o
def simulate(*args, **kwargs) -> None
```
-Simulates the scaling law estimation process using the provided arguments. This method is a wrapper around the Simulator class, allowing for quick setup and execution of simulations.
+Simulates the scaling law estimation process using the provided arguments.
+This method is a wrapper around the Simulator class, allowing for quick setup and execution of simulations.
**Arguments**:
@@ -93,7 +101,9 @@ Sample a random allocation and model configuration from the user-specified seed
**Returns**:
-`(N, D), model_config` - A tuple containing the allocation $(N, D)$ followed by a model configuration dictionary corresponding to $N$. If `model_search_config` is not specified, the latter will be `None`.
+ `(N, D), model_config` - A tuple containing the allocation $(N, D)$ followed by
+ a model configuration dictionary corresponding to $N$. If `model_search_config` is not specified,
+ the latter will be `None`.
**Raises**:
@@ -105,17 +115,20 @@ Sample a random allocation and model configuration from the user-specified seed
def fit(parallel: bool = True, simulation: bool = False) -> None
```
-Uses [L-BFGS optimization (SciPy implementation)](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html) to find the best-fitting parameters for the scaling law based on the collected data.
+Uses [BFGS optimization (SciPy implementation)](https://docs.scipy.org/doc/scipy/reference/optimize.minimize-bfgs.html)
+to find the best-fitting parameters for the scaling law based on the collected data.
+Note that this choice of optimizer is different from the original paper, which instead used L-BFGS (without explicit bounds).
+We choose BFGS for its absolute advantage in terms of accuracy and efficiency (see [this discussion](https://github.com/kyo-takano/chinchilla/blob/master/docs/changes.md#4-algorithm-l-bfgs-b--bfgs) for more details)
**Arguments**:
-- `parallel` _bool, optional_ - Whether to run L-BFGS optimization over the initialization grid in parallel processing.
+- `parallel` _bool, optional_ - Whether to run BFGS optimization over the initialization grid in parallel processing.
- `simulation` _bool, optional_ - Indicates whether the fitting is part of a simulation. Defaults to False.
**Raises**:
- `ValueError` - If there are not enough data points to perform the fitting.
-- `TypeError` - If the numerical precision is insufficient for the L-BFGS algorithm.
+- `TypeError` - If the numerical precision is insufficient for the BFGS algorithm.
### `scale`
@@ -160,7 +173,8 @@ Determines the compute-optimal allocation of a scaled FLOP budget for the next m
**Returns**:
-`(N, D), model_config` - A tuple containing the allocation $(N, D)$ and an optional dictionary with the model configuration corresponding to $N$.
+ `(N, D), model_config` - A tuple containing the allocation $(N, D)$ and
+ an optional dictionary with the model configuration corresponding to $N$.
**Raises**:
@@ -178,7 +192,8 @@ def step(num_seeding_steps: int | None = None,
**scale_kwargs) -> tuple[tuple[int, float], dict[str, int] | None]
```
-Shorthand method automatically routing to `seed` or `fit` & `scale` methods, depending on the existing number of training runs in the seed regime.
+Shorthand method automatically routing to `seed` or `fit` & `scale` methods,
+depending on the existing number of training runs in the seed regime.
> If you prefer to be explicit about the seeding and scaling steps, you can use the following approach:
>
@@ -194,13 +209,14 @@ Shorthand method automatically routing to `seed` or `fit` & `scale` methods, dep
**Arguments**:
- `num_seeding_steps` _int, optional_ - The threshold number of seed training runs before starting to scale the compute budget.
-- `parallel` _bool, optional_ - Whether to run L-BFGS optimization over the initialization grid in parallel processing. To be passed to `fit`.
+- `parallel` _bool, optional_ - Whether to run BFGS optimization over the initialization grid in parallel processing. To be passed to `fit`.
- `simulation` _bool, optional_ - Indicates whether the scaling is part of a simulation. Defaults to False.
- `**scale_kwargs` - Keyword arguments to be passed to `scale` (`scaling_factor` and `C`).
**Returns**:
-`(N, D), model_config` - A tuple containing the allocation $(N, D)$ and an optional dictionary with the model configuration corresponding to $N.
+ `(N, D), model_config` - A tuple containing the allocation $(N, D)$ and
+ an optional dictionary with the model configuration corresponding to $N.
**Raises**:
@@ -215,7 +231,8 @@ Shorthand method automatically routing to `seed` or `fit` & `scale` methods, dep
def adjust_D_to_N(N: float) -> float
```
-Adjusts $D$ (the number of data samples) to $N$ (the number of model parameters) based on the scaling law. Computes:
+Adjusts $D$ (the number of data samples) to $N$ (the number of model parameters) based on the scaling law.
+Computes:
$$D = G^{-(1 + b/a)} N^{b/a}$$
@@ -228,7 +245,8 @@ $$D = G^{-(1 + b/a)} N^{b/a}$$
> D = cc.adjust_D_to_N(N)
> ```
>
-> Once you get an estimate of the scaling law for your task, you may want to update $D$ to match the actual value of $N$ if your `estimate_model_size` is not strictly accurate.
+> Once you get an estimate of the scaling law for your task,
+> you may want to update $D$ to match the actual value of $N$ if your `estimate_model_size` is not strictly accurate.
**Arguments**:
@@ -250,11 +268,13 @@ def allocate_compute(cls, C: float | list | np.ndarray,
params: dict) -> tuple[float, float] | np.ndarray
```
-Allocates a given computational budget (C) to the optimal number of model parameters (N) and data samples (D), which wouls satisfy the following formula based on the scaling law parameters provided in the `params` dictionary.
+Allocates a given computational budget (C) to the optimal number of model parameters (N) and data samples (D),
+which wouls satisfy the following formula based on the scaling law parameters provided in the `params` dictionary.
$$\underset{N,\ D}{argmin}\ L(N,\ D\ |\ E,\ A,\ B,\ \alpha,\ \beta)$$
-Once instantiated, this class method gets overridden by `__allocate_compute` so that `params` are automatically specified from the instance attributes.
+Once instantiated, this class method gets overridden by `__allocate_compute` so that `params` are
+automatically specified from the instance attributes.
> **Example Usages**:
>
@@ -286,7 +306,8 @@ Once instantiated, this class method gets overridden by `__allocate_compute` so
**Returns**:
-tuple[float, float] | np.ndarray: A tuple containing the optimal number of model parameters (N) and data samples (D). If C is an array, the output will be a 2D array with shape (len(C), 2).
+ tuple[float, float] | np.ndarray: A tuple containing the optimal number of model parameters (N) and
+ data samples (D). If C is an array, the output will be a 2D array with shape (len(C), 2).
**Raises**:
@@ -300,11 +321,14 @@ def predict_loss(cls, N: np.ndarray | float, D: np.ndarray | float,
params: dict) -> np.ndarray | float
```
-Predicts the loss for given allocations of model parameters (N) and data samples (D) using the scaling law parameters provided in the `params` dictionary.
+Predicts the loss for given allocations of model parameters (N) and data samples (D) using the scaling law
+parameters provided in the `params` dictionary.
-The loss is calculated based on the following formula: $$L(N,\ D\ |\ E,\ A,\ B,\ \alpha,\ \beta) = E + A \cdot N^{-\alpha} + B \cdot D^{-\beta}$$
+The loss is calculated based on the following formula:
+$$L(N,\ D\ |\ E,\ A,\ B,\ \alpha,\ \beta) = E + A \cdot N^{-\alpha} + B \cdot D^{-\beta}$$
-Once instantiated, this class method gets overridden by `__predict_loss` so that `params` are automatically specified from the instance attributes.
+Once instantiated, this class method gets overridden by `__predict_loss` so that `params` are
+automatically specified from the instance attributes.
> **Example Usages**:
>
@@ -339,23 +363,33 @@ Once instantiated, this class method gets overridden by `__predict_loss` so that
**Returns**:
-np.ndarray | float: The predicted loss or an array of predicted losses.
+ np.ndarray | float: The predicted loss or an array of predicted losses.
**Raises**:
- `ValueError` - If `params` is missing any of the required parameters (E, A, B, alpha, beta).
+### `params`
+
+```python
+@property
+def params() -> dict
+```
+
+A proxy to get scaling law parameters by internally calling Chinchilla.get_params()
+
### `get_params`
```python
-def get_params() -> dict
+def get_params() -> dict[str, float]
```
-Returns a dictionary of estimated parameters describing the scaling law / parametric loss estimator.
+Returns a dictionary of the scaling law parameters.
**Returns**:
+`
-- `float` - The computed loss value.
+- `dict` - A dictionary of the optimized parameters
**Raises**:
@@ -380,15 +414,28 @@ The report includes:
- `ValueError` - If the scaling law parameters have not been estimated yet.
-# `chinchilla.database.Database`
+### `sweep_param_grid`
+
+```python
+def sweep_param_grid(plot=True, img_name="sweep_param_grid")
+```
+
+Utility method to visualize the 1D landscape of minimum loss by each parameter value in grid
+
+# `chinchilla.database`
+
+## `Database`
```python
class Database()
```
-Stores and manipulates scaling data in a Pandas DataFrame the default persistence to a CSV file. The Database class is used internally by a `Chinchilla` instance.
+Stores and manipulates scaling data in a Pandas DataFrame the default persistence to a CSV file.
+The Database class is used internally by a `Chinchilla` instance.
-If `project_dir` is provided, the DataFrame is initialized from the CSV file at that location. If the file does not exist or is empty, a new DataFrame is created. If `project_dir` is None, the DataFrame is kept in memory.
+If `project_dir` is provided, the DataFrame is initialized from the CSV file at that location.
+If the file does not exist or is empty, a new DataFrame is created. If `project_dir` is None,
+the DataFrame is kept in memory.
**Default columns**:
@@ -415,7 +462,8 @@ Initializes the Database instance.
**Arguments**:
-- `project_dir` _Optional[str]_ - The directory path to save the DataFrame as a CSV file. If None, the DataFrame will not be saved to disk.
+- `project_dir` _Optional[str]_ - The directory path to save the DataFrame as a CSV file.
+ If None, the DataFrame will not be saved to disk.
- `columns` _List[str]_ - A list of column names for the DataFrame.
- `log_level` _int_ - The logging level for the logger instance.
@@ -427,19 +475,29 @@ def append(**result: dict[str, float]) -> None
Appends a new row of results to the DataFrame and updates the CSV file if `project_dir` is set.
-If 'C' is not provided in `result`, it is automatically calculated as $6ND$. All numerical values are rounded to the nearest integer to prevent scientific notation in large values. Additional columns provided by the user are appended to the DataFrame.
+If 'C' is not provided in `result`, it is automatically calculated as $6ND$.
+All numerical values are rounded to the nearest integer to prevent scientific notation in large values.
+Additional columns provided by the user are appended to the DataFrame.
**Arguments**:
-- `result` _dict[str, float]_ - A dictionary containing the data to append. Must include 'N', 'D', and 'loss' keys. If 'C' is not provided in `result`, it is automatically calculated as $6ND$. All numerical values are rounded to the nearest integer to prevent losing precisions to scientific notation for large values. Additional columns provided by the user will be appended to the DataFrame without any conflicts.
+- `result` _dict[str, float]_ - A dictionary containing the data to append. Must include 'N', 'D', and 'loss' keys.
+ If 'C' is not provided in `result`, it is automatically calculated as $6ND$.
+ All numerical values are rounded to the nearest integer to prevent losing precisions to scientific notation for large values.
+ Additional columns provided by the user will be appended to the DataFrame without any conflicts.
+
+# `chinchilla.visualizer`
-# `chinchilla.visualizer.Visualizer`
+## `Visualizer`
```python
class Visualizer()
```
-`Visualizer` includes methods for plotting the estimated loss gradient, the efficient frontier, and L-BFGS optimization results. It helps in understanding the distribution and relationships between compute resources, model parameters, and data samples, and highlights efficient allocation frontiers and seed regimes.
+`Visualizer` includes methods for plotting the estimated loss gradient, the efficient frontier,
+and BFGS optimization results. It helps in understanding the distribution and relationships between
+compute resources, model parameters, and data samples, and highlights efficient allocation frontiers
+and seed regimes.
**Attributes**:
@@ -473,32 +531,38 @@ def plot(cc,
Plots the loss gradient and efficient frontier for resource allocation.
-This method visualizes the distribution and relationships between compute resources (FLOPs), model parameters, and data samples. It shows how these factors interact with the loss function and highlights efficient allocation frontiers and seed regimes.
+This method visualizes the distribution and relationships between compute resources
+(FLOPs), model parameters, and data samples. It shows how these factors interact
+with the loss function and highlights efficient allocation frontiers and seed regimes.
-**Example output**: ![](../examples/efficientcube-1e15_1e16/parametric_fit.png)
+**Example output**:
+![](https://github.com/kyo-takano/chinchilla/blob/master/docs/imgs/parametric_fit.png)
**Arguments**:
- `cc` - A Chinchilla instance with a Database of training runs and scaling law parameters if estimated.
- `next_point` _dict[str, float] | None_ - The next point to be plotted, if any.
-- `fitted` _bool_ - Whether to plot the scaling law gradient or only raw data points. If the loss predictor is not fitted, falls back to False.
+- `fitted` _bool_ - Whether to plot the scaling law gradient or only raw data points.
+ If the loss predictor is not fitted, falls back to False.
- `img_name` _str_ - The name of the image file to save the plot as.
- `cmap_name` _str_ - The name of the colormap to be used for plotting.
- `simulation` _bool_ - Whether the plot is for a simulation.
-### `LBFGS`
+### `optim`
```python
-def LBFGS(y_pred: np.ndarray,
+def optim(y_pred: np.ndarray,
y_true: np.ndarray,
C: np.ndarray | None = None,
simulation: bool = False,
- img_name: str = "LBFGS") -> None
+ img_name: str = "optim") -> None
```
-Plots the results of L-BFGS optimization, including the loss history and prediction accuracy. This method visualizes the predicted values versus the true labels and the error distribution.
+Plots the results of optimization, including the loss history and prediction accuracy.
+This method visualizes the predicted values versus the true labels and the error distribution.
-**Example output**: ![](../examples/efficientcube-1e15_1e16/LBFGS.png)
+**Example output**:
+![](https://github.com/kyo-takano/chinchilla/blob/master/docs/imgs/optim--asymmetric.jpg)
**Arguments**:
@@ -508,7 +572,17 @@ Plots the results of L-BFGS optimization, including the loss history and predict
- `simulation` _bool, optional_ - Whether the plot is for a simulation.
- `img_name` _str, optional_ - The name of the image file to save the plot as.
-# `chinchilla.simulator.Simulator`
+### `LBFGS`
+
+```python
+def LBFGS(*args, **kwargs)
+```
+
+Deprecated function. Please use optim() instead.
+
+# `chinchilla.simulator`
+
+## `Simulator`
```python
class Simulator(Chinchilla)
@@ -516,7 +590,9 @@ class Simulator(Chinchilla)
Simulates the scaling law estimation with `Chinchilla`, allowing you to understand its behaviors.
-Inheriting and extending the `Chinchilla` class with the capacity to simulate seeding and scaling in a hypothetical task, `Simulator` models how factors like `Chinchilla` configuration, number of seeds, scaling factor, the noisiness of losses, etc. would confound to affect the stability and the performance of scaling law estimation.
+Inheriting and extending the `Chinchilla` class with the capacity to simulate seeding and scaling in a hypothetical task,
+`Simulator` models how factors like `Chinchilla` configuration, number of seeds, scaling factor, the noisiness of losses, etc.
+would confound to affect the stability and the performance of scaling law estimation.
**Attributes**:
@@ -586,7 +662,8 @@ Simulate the compute-scaling on a hypothetical deep learning task with some nois
- `num_scaling_steps` _int_ - The number of scaling steps to simulate.
- `scaling_factor` _float | None, optional_ - The scaling factor to be used in the simulation.
- `target_params` _dict[str, float]_ - A dictionary of target parameters for the simulation.
-- `noise_generator` _Iterator | tuple[Callable, tuple[float, ...]] | None, optional_ - A callable or iterator that generates noise to be added to the loss. Defaults to `(random.expovariate(10) for _ in iter(int, 1))`, which generates an exponential distribution averaging at $0.100$.
+- `noise_generator` _Iterator | tuple[Callable, tuple[float, ...]] | None, optional_ - A callable or iterator that generates noise to be added to the loss.
+ Defaults to `(random.expovariate(10) for _ in iter(int, 1))`, which generates an exponential distribution averaging at $0.100$.
**Raises**:
@@ -606,7 +683,8 @@ def search_model_config(
use_cache: bool = False) -> tuple[dict[str, float], float]
```
-Finds the model configuration that is closest to a given target number of parameters, based on the provided hyperparameter grid and size estimator.
+Finds the model configuration that is closest to a given target number of parameters,
+based on the provided hyperparameter grid and size estimator.
> **Example Usage**:
>
@@ -634,11 +712,12 @@ Finds the model configuration that is closest to a given target number of parame
**Returns**:
-A tuple containing the closest model configuration and its estimated size.
+ A tuple containing the closest model configuration and its estimated size.
**Notes**:
-Although very efficient, you should set `use_cache` to True only when `hyperparam_grid` is guaranteed to be consistent; thus, it is disabled by default except for Simulator (x16 faster).
+ Although very efficient, you should set `use_cache` to True only when `hyperparam_grid` is guaranteed to be
+ consistent; thus, it is disabled by default except for Simulator (x16 faster).
### `is_between`
@@ -656,78 +735,7 @@ Checks if a value is within the given inclusive bounds.
**Returns**:
-bool | np.ndarray: NumPy array: A boolean or an NumPy array of booleans indicating whether the value is between the bounds.
-
-# `chinchilla._metrics`
-
-A few loss & weight functions you can use on demand.
-
-### `asymmetric_mae`
-
-```python
-def asymmetric_mae(y_true: np.ndarray,
- y_pred: np.ndarray,
- w: float = 1e1) -> np.ndarray
-```
-
-Asymmetric Mean Absolute Error loss function.
-
-### `huber`
-
-```python
-def huber(y_true: np.ndarray,
- y_pred: np.ndarray,
- delta: float = 1.0) -> np.ndarray
-```
-
-Huber loss function.
-
-### `log_huber`
-
-```python
-def log_huber(y_true: np.ndarray,
- y_pred: np.ndarray,
- delta: float = 1.0) -> np.ndarray
-```
-
-The original loss function used in the Chinchilla paper
-
-### `mae`
-
-```python
-def mae(y_true: np.ndarray, y_pred: np.ndarray) -> np.ndarray
-```
-
-Mean Absolute Error loss function.
-
-### `mse`
-
-```python
-def mse(y_true: np.ndarray, y_pred: np.ndarray) -> np.ndarray
-```
-
-Mean Squared Error loss function.
-
-# `chinchilla._logger`
-
-Contains a utility function `get_logger`. This module also filters out noisy debug messages from `matplotlib` and suppresses redundant warnings from `numpy` and `matplotlib`.
-
-### `get_logger`
-
-```python
-def get_logger(level: int | str, name: str) -> logging.Logger
-```
-
-Sets up a logger with the specified log level. This logger uses RichHandler for `rich` formatted logging output to the console.
-
-**Arguments**:
-
-- `level` _int | str_ - Logging level, e.g., 20 or logging.INFO, 30 or logging.WARNING.
-- `name` _str, optional_ - The name of the logger.
-
-**Returns**:
-
-- `logging.Logger` - Configured logger instance.
+ bool | np.ndarray: NumPy array: A boolean or an NumPy array of booleans indicating whether the value is between the bounds.
# `chinchilla._validator`
@@ -743,7 +751,9 @@ Validates a grid of initialization for scaling law (/loss predictor) parameters.
**Attributes**:
-E or e: Tuple of floats representing initial values for the E parameter or its log form. A or a: Tuple of floats representing initial values for the A parameter or its log form. B or b: Tuple of floats representing initial values for the B parameter or its log form.
+ E or e: Tuple of floats representing initial values for the E parameter or its log form.
+ A or a: Tuple of floats representing initial values for the A parameter or its log form.
+ B or b: Tuple of floats representing initial values for the B parameter or its log form.
- `alpha` - Tuple of floats representing initial values for the alpha parameter.
- `beta` - Tuple of floats representing initial values for the beta parameter.
@@ -852,3 +862,76 @@ def check_noise_generator(cls, v)
```
Validates the noise generator, ensuring it is an iterator or None.
+
+# `chinchilla._logger`
+
+Contains a utility function `get_logger`. This module also filters out noisy debug messages
+from `matplotlib` and suppresses redundant warnings from `numpy` and `matplotlib`.
+
+### `get_logger`
+
+```python
+def get_logger(level: int | str, name: str) -> logging.Logger
+```
+
+Sets up a logger with the specified log level.
+This logger uses RichHandler for `rich` formatted logging output to the console.
+
+**Arguments**:
+
+- `level` _int | str_ - Logging level, e.g., 20 or logging.INFO, 30 or logging.WARNING.
+- `name` _str, optional_ - The name of the logger.
+
+**Returns**:
+
+- `logging.Logger` - Configured logger instance.
+
+# `chinchilla._metrics`
+
+A few loss & weight functions you can use on demand.
+
+### `asymmetric_mae`
+
+```python
+def asymmetric_mae(y_true: np.ndarray,
+ y_pred: np.ndarray,
+ w: float = 1e1) -> np.ndarray
+```
+
+Asymmetric Mean Absolute Error loss function.
+
+### `huber`
+
+```python
+def huber(y_true: np.ndarray,
+ y_pred: np.ndarray,
+ delta: float = 1.0) -> np.ndarray
+```
+
+Huber loss function.
+
+### `log_huber`
+
+```python
+def log_huber(y_true: np.ndarray,
+ y_pred: np.ndarray,
+ delta: float = 1.0) -> np.ndarray
+```
+
+The original loss function used in the Chinchilla paper
+
+### `mae`
+
+```python
+def mae(y_true: np.ndarray, y_pred: np.ndarray) -> np.ndarray
+```
+
+Mean Absolute Error loss function.
+
+### `mse`
+
+```python
+def mse(y_true: np.ndarray, y_pred: np.ndarray) -> np.ndarray
+```
+
+Mean Squared Error loss function.
diff --git a/docs/changes.md b/docs/changes.md
index 53f473f..3d61b74 100644
--- a/docs/changes.md
+++ b/docs/changes.md
@@ -8,9 +8,9 @@ These changes aim to improve the theoretical consistency as well as the performa
The loss predictor $L(N,\ D\ |\ A,\ B,\ \alpha,\ \beta)$ aims to capture **_the lower bound of_** the loss achievable with a given allocation $(N, D)$.
However, the original approach utilizes a symmetric loss function (log-Huber) to predict the **_expected_** (mean) loss and does not adequately account for the distribution of errors.
-![L-BFGS optimization with log-Huber](./imgs/LBFGS--symmetric.png)
+![Optimization with log-Huber](./imgs/optim--symmetric.png)
-Modelling-wise, the additional loss attributed to the inherent incompleteness of a training setup---which we shall call **a noise term**---should be more exponentially distributed rather than normally.
+Modeling-wise, the additional loss attributed to the inherent incompleteness of a training setup---which we shall call **a noise term**---should be more exponentially distributed rather than normally.
If we comply with the modelling and assume the errors to be positive and asymmetrically biased to 0, losses in the right tail of such a distribution would have extensive effects on fitting the loss predictor when using a symmetric function like Huber.
Although you may find a symmetric distribution of errors, it's only _ad hoc_ so, and their choice of Huber to address outliers does *not* address it.
@@ -27,7 +27,7 @@ y - \hat{y},& \text{if } y - \hat{y} > 0\\
This modification more accurately fits the loss predictor to **_the lower bound of_** achievable losses.
-![L-BFGS optimization with asymmetric MAE](./imgs/LBFGS--asymmetric.png)
+![Optimization with asymmetric MAE](./imgs/optim--asymmetric.png)
Nonetheless, you are free to stick to the original log-Huber or use your own `loss_fn`.
@@ -48,3 +48,9 @@ To set any of them in log scale, simply lowercase the letters ($e$, $a$, $b$) fo
The original logarithmically scales these parameters (seemingly for numerical stability with the subsequent sum-exp operation), but there are no tangible reasons _to_ or _not to_ apply such a transformation.
I would personally suggest that you don't log-scale $E$, but it doesn't really matter fpr $A$ and $B$ as long as they are not too large in linear scale.
+
+## 4. Algorithm: L-BFGS-B → BFGS
+
+I have tested the temporal performance of fdifferent algorithms (including those not shown here) and **BFGS just works best**, regardness of how good the initial parameter grid is.
+
+![algorithmic performance](./imgs/algorithm.init-original.png)
diff --git a/docs/imgs/LBFGS--asymmetric.png b/docs/imgs/LBFGS--asymmetric.png
deleted file mode 100644
index a9af83556ee05fcac31fdd5a4ac8ceda0a3df290..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001
literal 42306
zcmd43cQ}{-|30i;l1j2mWMpKMt&p8vnMsta?7bQiN-{D^q>PX~GD2lnQX-p-$O_r}
zcV4}}_wP7<$9*67U-uvP=Q!T)k9T=puj_g~AJ50*e4OX`xPmm*759{C{f
z*Crv^T17#eMfq{7J6moF)EG%=xUIvyR<0XE#&Ft0d=4o$YPxoNX+x9CN+u=wxYk
zLy+eb4?p*@>(0*hPU5`0w*UJVcf)Zf5GXuI269q*R*kWpmAV+qtUSy{1?d4ym@pRJZ@VU)vk
z3~YUuY4}FK`=PYY%5)v0-*ubsZEbZQKayU&c+qoy?CfA|h)%hy)$r$7xm5LMLPa$-
zH4Z=DZ6Rlr{v1Mo%6)xBnDp*edU|@!^Xc*tZ0D}GWgSULNujG_Wnr2A#9Nu;Hg)Mn
zUqxfr8@t?`oafiSy(lg#8~pX_Z2q-Ifu+OshPWj){Jix0WoV3qTUJ)pLD?Zw;wh^o
zq*s0)pO~E768P}p@W%RToT$U?L+t9)^KB;Iam$pU^irPA!ChSE)01Xadds~Y9Fd4B
zbDO$iYI^3GxQo5L{b>`E1dFD4htX!%hexG8F828~Ezgd$61T=@)6S>oJQ@W`k1^BLOb0_q;n
z&c1h&@!N>Q&CRYYJFU!(RxQm8au^lc=eb&Zi8I|;Tc+#66`Rs@QeWxiGCw_KSKHlf
zWag)
zt0lWs>6>}ujd=qFu3K#1FPrBi>AOaQ*kRL%kGnRZ8pW-(cyp?^oF0EPUzxOYp7Ssk
zJ;iJB4{m4IsZ*!uPTlzY{LG!{+1c+a<1GI5n@JW3Ug@9RB~B#8rKQ)F7__vsCi*HF
zzh+UFXfid1T5xf3ksOtDf7DOYyExoPU+OX*ke0@A(y&N3)Ntq0@<>7~zuC@!fB<(7
z53RTE4o9WE>%Tmc5c6FVm2jPSZr%`n`bwzV$d_k9fq~DpUMdvWbcDym&>*lrBPvFF
z%1)gPrfkS7wrNp(m*p=#H{@PlHX
zRgvD_-p}z;hR2sH{5GWpExuIu_Qp3pJ!NKLk=pC_r(0|%IYa%MKHpDyS3ljXHf6Ev
zeQP?_k~TNqrZGqt78dq<^y;LXl7@x`P73j8+Q}*$8xFZXtF99h6YZ6LG6)p9Re`Q5
zUjKR%A{zv{P4t#abckTp#hrhj9&Jw6k-C9@Hx#vI85=$$t8|~)dir>-cUzWmnfu?N
zpxUD0#-|xx;#?QAKCdm$#!7jKV6(3x1jgE}e!Ol#&Ug?V9ZfYgEwgjyPPHdO9QwR-
zKQax2?%y9oKq^t$^_05I{QWzChd9p6yj4(8a9-NXY|y;Ty^>i$om2PKft8h&h9W~l
z!|`KuO!7B~4TLQuEj^|Gm1gv1-R-Qt|oQOGPoyxf9aT(x0D6P~VkJ
zY!#lIoNTRP?nN^3+d||$`^nBi^Hn)nStTW`c-p#$KIut@oX`l6`_4}
zLeG&ABY90OjqBwZ+#^5IUSN3$UzmLEsYpxu`TJI_>B*wK2D`O#h?IQdK)j?!_?0PJ
zgF&}_WMn4{i?~gEmsvO0X7@+)7=LJL3dQ+OU0taOKD>>ht}Vs$sXS3<}8n5
zbaCY-Zv8xeom6#)rD;C8eYUTfN5n``;({Z782UeL5iA6!m~?-#(9(Igb5D
zCF{Qy_ecI7AOButYe=N`;KN*^ej9IpG}v2}7b(jzV$omolQEzC_3`oGJx1SS($evM
z8_W0dEt>K?X0KJOFBqz+sr_zFzF#YRh=L;g*>OopqYP_HIb>`k)9;mjezLN%#oo&T
zyT}>d_4Oqn#WVD4&g*@ulP?ftlAd#cC|%Y*Cz<3o*odm?{cf->RLpT`_vySHV$FjQ
z8q)87{D>;|S`_eFu$kw(vv^iso~5hEZ`1cnoLIyafsKKhy{><{_ndBgA}q*Y5`L-N
z)AaJ?cX47)#rU1P_1xTCx}d-F<83+yD5|TeN!xtoQTf6Dz>A;ZrWCS)-p4a*=qlF!s%BC=cVo}1
zsjHhEYB==ph=c;-10mY#xm#9+ND4Wo+stYon4+>Hq7=&+^5f%=@|si(eSFL%={`+@
zVjdS4*HVW&J#1oPf~{4UA8U26w-y&;K-44sMyzWtOEvBwX=-Gi2bZs8ra+U
z#!jOuU>~v`_sq=9_FcQ2a0Y`ZIc`^0%8*oUtZ-o|Qx!t!B00~Ks;a6UmGZ2%t6UFA
zOiWZL5#GOlKMqh$&3$SlqrEO;E&abfMXxP%*$G|$daPzhg!S%T>e{NRJF3w<{H7na
zGaNbc?fdttPoM6Gv#K`dTWEZYibbioEPlT|-(o)^?d^Wns|>LuRz`R-QaNy8|FwBz`Xlj;f5=WJ}aFI>1l)KZ-6?}>8vxHEcx
zPEMY~=ay&c7)$K?w<6$bTUscQ>kJC5&OQ~fZL!6gXyB6cHO)iv%qcb@HQ7M
z$NcmE0ULoYcl>{bLH~b$G3Gj{|A3C`WM^`3(SHaiZ-S1R$LL*64T6uyyhH5l!pF~x
zHh+g2bTclTR2>`z@bX!iYiM>NWh$86m@-jW44KHYZsP%XLIKP5d>S21jT7-{!TL^j
zrE}t=tZntYxPDC}hmKH@zM-K`g}2Af{YTYs7W03V0vNqEE{p0cv>qHCjXl{`-&kbR
zaXTnz5AbZW%ZZX72z9+u=h0nVDAiAKq<@{3mkoR!)hzYfm^fNEKVsCHJYi3#4wi@&
z-Bt+i=N56D&|_d=&`i_y_xH~MehDBy@(~B!va`T4%7mAbGdNi_TEKJex8yUjP#XT*
zh}x9ZV!&(dB&9HOMz%t$R_0^JNVaXe^O}^u$qH~0zki|f;e!X}XbFf?i;DE>4fRO_
zUF!ZlOnI8P7oRELokM3A78bT|->&&ebEP?|<8v
zFS#X5E?vmIa+dSL>sP+(W361*zQh4+>;;fYJ4$`%(4AYiws?99JW&aJJFxEcNMvAe
zP>b_12b&+zQp}Sl*F@uAm)f1647S-+)9}m&_COu87%yyQUd)bTcPp{EhFlvKrj;kJ
zk*1qxTD23~aqsC1=m(;aK}y_$4dUS8
z;VCXI&H*&l9aN^<>oPNN_S)x|TdAq+tr-_bW}9R*Tql$Gwb4+$K($6klyBQ5Lh|9m
zhb81Y0;gwXWf5zO-+rmfDg5Oi6;P7BW53T;$X`XIRjeJ6
z8Ie-HWnAP%D$0B~g=Dsdq%*b{FI3B2$bWIy{-e5;)WC^s>T&f*T0)&HAu>J-6ST(_
zA12|GK=1cFW{1NbJ$n9a0kFY&tR)EXq*d3bfXJpWt9zKN9KiEXoB=gbOXWV)bp>b8Z2y4&v5Ol3Oar-KKi4e(>Kz
zz%zX1)}TGJ$e-7H--=;{(NE-Fe!umKnVImHgdaY~g^4qSHNbv+FLAOUu!B|i)I(a|
znSmWZNQvkzi2HheW}CtTn#2t?mGPrtbIbDpk99nAKhqQobuWf7$
zDYETy1QrWf@ASPi9IzU_6go#{ksBDe8^v|UX(grIC;`j@0^z9ARs~Pd<`BIURuNVI
zUR@|-{?$)ZH+tVT08aDs@f~Af@yF9^B`F=mDGo;O8pDHmppXvw_<6l(!@a;F-MkYUK66ri{!fa5Fx{7+02qz
z5rtxWwIQ4p*)Rl+<`n=Ngshm~h9to9Wwa^YvyG>4<^;T#?2k<;JAHlYFD6
zR4q>Q;LUgMPH(KuY5=)KUm-)=fo?7icQxI5(Zq7DnnB7lvb|lev&eSuJFIMsur*bt
zLE#I#zKZJVYI{`ctV`u8!0!bsbSZ5>Q&`)}U!R|;uB}a3NmW~Jw8~`#JzMD61l3d6%s6mM971_krmgtn)C0#=Y
z{5gjI4i4?NY!jK=I5GTJwns%pMV`=NZd??<_L=5SZ@DV^w%j*wMzBqQpaH-m13!O0
zZ85tQ?cmz6IkP)vOU0p9s_LN;TTSbE*T*N5mD39NIO}cxx(tIfs
z6BDO}g+=Q?&He9(M-&--l7d4*&fA&fUkmxz(V_1GyfR;e1}{DC*|YQP8u2ZnbvU=W
zc>MD{m3}5>tD75(GUt<39_u2u>1ML^^A&p7q@^#l0ti0N&(9x=Hu2T6GkNQtOI&V!
z1v`=sAe#>GnXICu^tJe*cAD;HC6+xM909vm!bnrXMKKA9j4uf?+4!q;gwJ{8lk<~Z
zZ!VryRLtmFM`r$a%P-E}Z|Pz?_aupUz>~(Mx`T1(E7L=zSCrO0f4tUD2Zxh^niF9x
za}o(&o4D#jMwzT!pZZ5k>DL;cq?w@5%A$goIX^R`=UD&D;#2-a*g7RRIJo&bo{LSn
zxk+17^N~ujb{aD?vo12eE^7U`mp%8Xd3A-YT9k^`eWv`H(B2f~ChM(sI8`B!W
zuF=vJUSZMaJsaZr6i>15h`25m>F{|ar9hSB<_$LuT+(y#Pv%ViW8=R4OnI_ai_eo*
zHN&4jKSr!tI@;T@zWXf7mE#G%D`~Y0m75!e#HOtb+~*Z4YI-Jdk@zohw?A>7hE|Iq
zkt{wupgc0JT)EPM+}W3fCHb+mt{N;cT5*t$E}dAROye?O<0O6lxN<=UGW+R3Eg9LE%o4-XHMtZl4Lfy}#V8HfG*{(H+f
z&gos!!#K)^=q3S$uD9nT){5W~RxPh?fdEukLuEgXQ;XHzYX5ggne;HHeCScBmbx+Y
zw>WhBYuylzkt(b&6+%=*uFdzvTiY5}`aXN~=m2n2B0p6I3YMst*v#rwCA-9MPw^I9
zIdP`_^M?<&U%fg(w1HWA$c1uY%xA%h>DcYvx9=A^udz%0Nwb^y5K>6e_*!f~Vx~XW
z2aF{vH}?aeM9Vd!-=m}b{rzEy0%xXbDQIaM(6+z2kV(h+I2c?an{^+`{ru)((jDcV
z^UORv51mH7Z0X$mlPph--EAK4@zQLrMEI~+x8+?`eVv)9BdLfSFMNr;hk7R&>Fz
zG%<9m5lPN{O3KY6j>O-^hx^QZQyg10v3d{^g2ebURA%F=RiKVlsq19={L*E#E+NWw
z*REY#T3WK79X9MaCZ!Nx*7DaA(I?Z
zKYW*_nNYPYzFSI43dnZxeNpFba&qrP*gMqjjYq!(v113wDPTo=et!ZZKSb11JE`EL4}U%vck!fn|HEGX(<7(fcq|4B+pUT;p?gY<(B
zsAY_&vj0dD?VEZ)G~m;H>pT+Xb?d6v!_7sXjv|gmM}xAndG|9)C&}I=J@HrtOvh5+
z=0@}Yzp_j<>D}A7;aRX@GhKF-4r47GDeBKYV(;tmpb@OG9ZboWWRta(*H`Az1KS7J
zvG|A%UG1EsY;SMRv1mF9(AkJQFXr)=1yJxqU7eyCg^>!Na~(cU{3$5Sy1qU}c6RoV
z^!?~gtHA2?`E5S?AGXSOWt1Jsxdu(H;QCRWaI8{c{L&)3*{cmdn||<)I&V3W++1B{
zWw&nINp?&~D3U0$Lqq;xim|0;XvrGbtgYn!_qw_&kMGg6dQ4W}!x??#@ICEja7qB`
zqM!j$^saq-arX74@+Y7ZwM!hYW_}54Jx^b-{`Rwa>1+S6i^>9V+roYKgzp>yRrds^
zih#kH(n9GMGF6amX=QvRR~IIU2776F`5QVSvVF8StB>Z1yNtausRYMM&3nl^=b|9T
zMt!{!>Ou513FJxCmL{Z0VT#4o#VKo-eaYGlsNk=3)3nnk3w$luLxJTPQ>L5}5Ef1bIZIG}3v79)e^m379)S|Js_o&k5RPU-|6;O8EH9Nad?X;o(s(W-ATF4H*XRN~rhjfhkOIzKTHj@-Fcrq(fx`^>YcNh(
zqdL4|C(v;Ud(lLxU(jZKOur2kt^z#E$H%AX7YJ59z~6s+6t7AAYaefKNmEnP(*vT~
ztzK6X!zrqg97RBIlZ=@a6oT>&ai}+ZbAuH?Aj7;EHIpE35MZob)KpXo`0OLSDFINC
zXm8UO*;!Zs+=lmGdWnA{&*s{6AW@o1BZa;QQ??QOXMMpv(eiEr22a!lnufBvk8ekj40+?2w`|Yh5od{qkU_jMan&F9=$s_y`{!K)lsE
z3FBZB0C%t3a2b~hfxh@58eO!w+ZMu-k#XZ5ihEX*pp#V|bEYK(Sgv}+#<%AUqVOs=
z*;7g2xZb*T>-0dQAM_GXf;Yhmq}Qqa0O?0Nf7I(+o5Q+!3GlyEjoqUXjAe*
zF%%(7&ki?o;ClhNuhu<05)&Jn7IDGM%Bmks$TAJ>s94D*HM!F_rl#TLnU4+t
z0%%gJ-UyoE9Lw))=mj3+=R4(@?c7Hvk7Sc}sa#w{L}dHUodf|6jqLS?KA`Y(EYO3a
zQcw8JYLix_uD=8W63ML}PAn!`oVoe=4uA|J27a}+>FE2FG*E8)B!swK$-oL+vnwJl;3i%{F^NEO6
z%L(k3ewph@Yk|>+&N2d6A3;Gv9*KTwk{LxVR))(oChRt
zh1@GO*T1Li0}J)dsdXxq(8%^JzD
zPf56m3{Oh?3>-{B}Afbwtjh=U8s&Ch>{E@jk!TzIq;
z+_5=s?zKT7DK@~M;QAQ?aT|$gK&XL6DgfStfac(qKmY#rk`^#ZLBiH|Q9Ila2`cH=
z`bw8FXu^noa~g^%K}f7b^t@CI)h@WM1zM;UpAqy{S6VAaPj(q_{d0m0Bice@U2&t$
zl35Vu3#{7+T$>9c0iwvk(QyER(sNY6G1t)&MQ!YVvQ&)z-^~~w&$|Sm+SNm+
zF7}+~ghu-whYnOrs>f$U<8RdI;8n;IGmDEF`Nj}^^^5II1I)7Y^VuNwjI~;TR&PSI
z@d*hH;i#69pFNiFNj6Zv(^q69-Fyke9SE|mkV7I>uI42N>VN?`j5C;(dy<*?Hj`XH
zQgOjFdUtzAN9{@;(6lL69tH*Bu+;tgX$p{DbP95^H7OGB&EijvGnr?H2L((m+q@a$@kB-cXn>IpJ>nJmPkFo
zJmn7E4+@r!AJy)jh^DJd+fPFRDzJ*qs3o(`_sD+pvxh2M3&Qqf^R!
zbj|lvjb92U~9ipMB2fye{nR$RYTPsZ$=h+?x4p8&XM;;;u)IPd3Oyh}(iUJ;tt_#7g
z24vcrJ^mx}erD!LBFk*qvc-}-P06}DwrEhx|A0pyQpj#n(ku8qcX#*S;tU8U{bLtH
zXurHx3_W`Az=6{!cZF7cXevlD*~TDJ-np|4O+~8<4emNmvto?47s-YI47Obt+7fp&
z{S88s2s_P#H*ZcMh}!UYfn+!j$9Qh4sy6%)#`liL6(sJ4{6ml*EKhNaC8
zrJIX__=l87GL7yE9779NJ@^QE)DtvBiuU$=iVqpSi7p_|cbB<^ZrQdo$^B2zA$IK)
zHCoP#S+Cww>S$z4Ds;lkSR1wEJNNS
zXeT82-nt(@M0DiX9M20(KmrH+YqrU^{GLMjM&pT45W4qd&+Z)4DjGoLlz~IClw{B@
z;3Cmx`xGR!4psfNQAs%1MI}2kp2tE8?JX_3bzy-MkLG+r?%)5UD-f5Ru8iouM(Fck
zNKI{QKBG*SpE(kif(BMAQFaIDLYE`KOC-x?tbQNUr!oj**O@C#A3*Twl{hA6$sP2N
zN9gP4Ub(yCfN-@V3}0H$({wp|D}3$)WPt)R$1y1-R~}*l0EW7G0kN_5C(J8^+XpoY
zE~jK}2|G14GDz1SK7HB?IQT@!GJL|x9!Q4p#1O6}^bIJHH6WVs4f1XAFE>88eT?eV9xVf=TphZZ!YuE|0BjS|Mq~SYB
zYb^x1UJE5dyZg@hMZw_eUoP<%LnGJSma6c
z+k|w2DBrVt_q(1RDV>oufHR^5AP-q)eapH2jX9(ZpMRmBe{Bt>5d+pLRLd(sf8HC*
zBWlmYg*#B87NHF9a=Dw0UU7P5-Wt{QC680=v!U-MyTj7+x4i${;jVT3;Lb*Qnqw?N
zPsHeX9Yv0b?t1yD7zF&^$v1s3bU6v&3C5?{MTt2SDi{rjj#6?t%449c-!(Uf<78Z0
zF_LimlLUK+0q=zWjuXNKXyDWqW
z*Bgh5X1ZSah(YN?{n>FBir%J3r$nrBp7W?SBrezoN&S1u+y(~+??4p8waAW2>b=}r
z1)h4eJ@+`YyaKD%V1g6A_bsLG*0x>0JMvSrst)X^#%=B1x38wDNyDsX2U3E!_u8&a
zR>CHPJVT(~CYjARB1kh0gdsQ8p+tropBi1uHmSS@Y#`~q^a$OD4_`0XZ^BeJlHeC_
z^%N=%;W5OX*P-xYYc7M(ah~dx>hijKCIIJB+d+_zML%@<-dpcOHI45Mbbq97OnD_R
z0=fgN$ph#E@cBG`%`R(Rewgb*&b3Cmfc=c>=<+?_ZA2{T^i}g3msSI+jCPj@0(t*J
zyG__}Kt9C)-j0iUPM)A71#${{_^=j@cUym;_YH`?F;Abu+wuWtjZpL+Eh&!RxJ$dQ
zK`4Pa@h2)VXv$oC9|+tU00EErvBP~ni$@4=22cZC9g%v}o{H>KdOcMJ7>T8c8ZHD6
zY2q_~7$nHM_wSin$pc~dJ4jFe>E@PgE#>cl7S$Y-&bmHA{s?<<(Udmh*^_UBd0hy*
zG=oA;6t3>ZI#(AiI@#%(5Nk)P2|89S8n$2G%%8ep`r4%O@fN
zbCCcZ)nT$z7=*;V|`(H-3tR2grlHA|Ws*Rt)yf7U`7`Y>-FA8?JNFkB%aNpKq!1lm)SHDG-C#e_&s{kEz_{wL@)F*JG%*IRyfPn);0vE7CGw5!bAtmUrOuqDh)U{`p65%
z6&?U{Jj}`A10o=S>Gs0rCW`Opr>8;@vdvI^@bzawwk<;qSQ<@D2tQv`Pu>iutQ5w)
zP0sJWL9Av+1KaqW27jC_9iUd!lUkrW=OgO39OuW*L(9>JmxYi#B_$b9pAZ!4
zvK{0#PsjEG$6p_}1WA3ux}yFgi>6#PJcU@cfBATgGt!xJ|V0{~+dR8^(_#HVEqD
zLS=7}Ej_{f1N#%EKR60WPJ_6x?8r+lokW|_B6lU2>f}c(cdN`Bw4cckU&o5LF1*$Z
z^*U)(%tshBG~WW7+w_z^#wM9rSp@@(p^grKbO6ttAR&+loNO@NBe7bHw
zA;|ynLy~|cvmaJxwI5{?P(_Ka0(k|H$Sfif1%k!0AkC;mfCNwZ;oG+vIXf;L*f3w*
z^r{T7*KYjzfXt0%VDd@E*ggW})
zU{US*^%IZU<>s@J+LTvkZ=;1C7#-zc{qgq^1*^*SwQrG;-dkws>2-(@u$GXJfbFX#
z+t|pcW%Cpt-`DGgul4vtp8QhYBg}HawEj1v71+$<;8bzcwrS>>D-#(BSPYDy@*>|J
zB?g23sVNOW3k8KBjcveF8lYjZ1Xu_{6$XKs_J?AaT1UrQ@16etnc5m@Ybh|`bu$M(^S=uYh=
zBiml@bYCAy3<8ufP;SJZqvA}>+Q{Un_m6GLKw|l=&tFXWVw|SU3VtneYy*iR*L(Tz
z{`fi+!5W%EOS5a&YLk>Bz;89*+xBHAvXcCkq;vcFzXs}YlcqjX8&28d1bOQv@Hvbq
zm+kB{Fs$HwMEh1-fu#<!2Ua{u38dDnmroo+QAl>UYMaXmS(2azKd=EPtp}Ko5G8uqU*s_JLj2p*))H|M
z=Z}Zwj%mNpPK)6)-HHsl^Eui(ZDH9VPUO*Ru)f02F)B_uvzgRwyb*c6?;`R2>;-8D
z4jw$5-C8nMJUpRYG@1xs#(%zaurq;_u+lmWRPTvZir&kzp2$E;n>MH<@2a#?aji&B
z9vCxB!`ys%C~_#p(Q=A(@809bj@<<9%Lfh?O{jc2TISC%y1X3c+aC0Z(NJH12E!sG
z;QFrKI3xY2p!65GtK7W2kIo}RpE7jl2URWOjikHXx;&G{IONv=XuHe3;A}gD%T@0x
zhe03x?cCfD(`z2qlM&A%58Pj?`uLGkHR?>Xxn7|6i@b<^B2l#l8jrjZ&YH?8xGdC9
z>-BF><}~C~2XdO5n_I#($!K3?C3+i;-C&{N{~|IR7_Bk!xUJ`a9l$Kw*xCY|e|HgO
zzE49#W8Pse_HV7`<|)bM9aM9l_>1gxb*|}dd34B6THQ_H=>@SjpVg|%9Dk95T{MFT
zLD*xVQe9gzk{wF=pT_SW`x3n52iu)M+vy}FoS?s9o_|(##hU&GZo!e7AABgzvcA~TJ{wcfO%huMPP(X5viqzX?
zWQTVC&s;@!sf#MX$}}GXwXv+(%uE}SA@Q@{=s}|N4O;1o1`V%qsY?|>=z{})w0Qxt
z-`TbQ8W6&3KH8
z36!GKJ@Wf^U}a_H$~^X(uqqKA;dk$DqK5EY{YVL6C~wYZcUV`Fi1&x)&yT(awhM)x
zQ@N&Na7(?*@kVEXs*=*_WNlMh+eV;cWx&}2@i+Bv@oj^XlOfNZJ6A$bcSyTkUjI`IajTJH%7-rr=R__wNvi&0Xf(EP%`n}R^bSX*05xLHAk
zSh-vZ6ZT!|-$nd`-aQED3Wlt3_1oNVKmlnI)}4wJ*&%J*+yb;5g9v7#h;dU|}&iMDRt3Id?1^c1r^m>Tb`6zmn5
zeT)5l{7O=>ttoL&i|)G~WZqF?_1XTyVYWVu))2g&l0Cc#I4AeT5AKBaD-EIz^ewY$7MJ{`
zFx@Q!>&i|X9;af@X{HyuF0!(3cz-S|d)tRyBxK7euCf*zS20_mtbHFHLL
zM+d+A^krXEyoEnKs-W=+iUCC<72ZF>Plu_l^KGWX%CNlwS!}nnv-=loXS2p=zmO6>
ze)Ojg;4m`t0(eqjCJPG91{;n2|J-ok?pF?5&eLzXIAYIbMA@qhh5@t_z^gIO3SEE&
zs0!yl-TEI>e^J1CPH>8l$m}sc2P{>Of>Dis&7AkPsyX25LhYHNac59Q<+>
zmLo#SN6-$!D+vd$4)_@|GBQ0h7~oWh(K{GuW!n;mX#w=#!Rmo&4kUi2YA@X6TeO$y
zx@osxzI=%q&3v)BXlwOz@}Tn4&N&v|9*@Iv+ZyBhd)WV^6ko4HFN02k6(#2dnAhg2
z7}Zz0oG|rN0Xh=5K!`bjBZMsjkmyyV2Jo>3ET@<>x)|Q?ED8?^-~mjF7(CJSI%VdGgE<*O&_R1wS66orG$;#LTx(Sj7qMXf$e9~K?RT=W{_w9)
z{(Ju4gCqY#sBcDbM7r{*?5WoKv(%nqay1v)RI#b{Xp;Mrpk9T=N9Ra>ist{STBV
zR;UE*XAnIN=U{GO;klx0NV^j~WzjGszp{>rmvDQCZ2T3}9m7N2-96vhbog58PS)_h
zQ~q$=Ow8?M%+8|=28%4I!@r)M;4$&5MnA_VEL_vkA*M5ONPh});V7r|$j_fQ@7$3S
zQHYe;2Oa?226j7{fGbn==aXAsym*0WKG;tZfZR~&6aU%9kLB|DsSLId{#xL*@N;ph
z&FSa4&^^8bP+kJl26oa6zAZpXc`LH}TkF0TZXZtBX<4*<<2$SR=Dd7QMBKqW4*b;$
zcMm<&3f{@6bh~Hbjamg6Knvl4g^F0q
z7v|?{;35YVi;c`PTb%5=2YnjJQDvqZ2b@Tt=!0j_ZEL^MWFk5ba02-o$iJU@$jF!^
z1t#gLR1Sacat`8HxAMth7Zbin&cnhpeDZAVoA4dNj=97W!_YZKI8h)!I`O?&OT*%n
zlAqZfb{kZ@1e#``dZ8{jggI&^fr?jt`jew&1RM@;{pF+XjFU-1M>6ZfR1?}d-1gn&d%mA
zLkU4y?P>e87vf|HGB*m|4@J>2ISLD}+C(ZfSuLEd47Up3Ecz4Va#xVQR4*
zz(1jMYDh0{S7K1H2ZD$?X;=#eMZmkUI~@-;H`c03fSz6^0-3xUt&GpP(suqs@5U)|
zQOmTU$SAoVDKDzaxEn)SHYuLdhSyDJ?S7D+yFse;^9y-@S_XvX5zBIdjGo`k7&
zabW=-^RVNJ=(BnlLYTn5<3vtlvdGfL4wN4-DZ7XWm7H7^LkLG1>pn$~x!(ujbo${S
zB)c2*VxZ(yoljfG{A#&*myY2avodyU?t1@+AIB|2zwg>zJiFEB?(MH{__@10Kzrv<
zFN&3TY%TOjF&)=4K7NqL>gtOueQw9pmf00*O}MxSuziaitK-o=nxUbs=HFL1
z+%}FdQgW#GFVBp<(AMEGK`s&6qwG@Y@hz|A`leXT#|!6*>;m6dr&kI4-!OU`vn2l<
zVhUZh3`<&h5x{FLa>4ZUH0U&Y45SkH1&t7#7J?uKb|{I6@s7GQZ>$lMD=;IRGoHP)
zJJAqhW?_n~C(gfKV*Pn%GMw#Qve2!Qf>qR!AGhCNFu7w~(%CM4Ays}yW1g1fzV8Q0
zTT?X{DM@!=BD@BD;kvjmoIiJE-xCIntZ^Zmc4C4e-Q?8uuO!gmiAoGsN{~V3(KX|t
z?n{R{>gxml6<{)^>P^fk2TG3N)R@|t6%=P*g%i^lNe`4%>0%;3E?(kiUrc116X!XjY&P%L
z$1G8Hv9MM?eoDqHEy7vDpK|oq{bc*3`blub2*BXl7xb6~F$E1uLMU%=*IqV@s7OEd
zKjT1n?!VVI
zctncGF^CIuDBek5V3cg@xOsVa`I3r@U;Q9`xAdEkCr45*8x`-Sy2tenKGu(H2^I
z4j?KhIQR-kb66tjK*fVBR{?X7H?jWf8hmHf-@b(*WsCzo{d>nmebo&q@5iG~EC}&l
zr<08$I~DHo^NFbD51dX-Ic)yH=;&bvhSyq6i;(2O$r3{uTL{q~khaY^!3~pV28H98
z@r0XJd2xC64q$XYh7iX2{7-xZHOd^l^7Ze>)Xf$9kbdLJzL&D2%tF&S8uQ)~4|`Qy
zUe)A#`DuR6-APsRQ9HjXNjKZ@xp3=I
zZ)YRsucq&(+-ViAc~HQ;2VEfQYt2CCWv`WB$>
z8Sp{_&}3R@Cbf!E;%_~V`GPN5lC{@2M&pj)`%DA6_g6`RXi|)tm
zn!imiB(a37tlca**ZC{cv&b=wl)|GMV=)Aufjwb{aE~beh9RW+4a_8J!lZRXA*K=r
zUyN>@hR5+PX@jkPU>sYei|p@r9XEecvR5SIYpqejN4rwAi>i}#+3X#f-cX*LR4;iV
z&8X&o-+=aeVTo!{HM65S&|*$r9u(t4XsJ-6G+eY{1EA+Vc@miahePD@MIcKEVfm2s
z{=JtZkRhefzpwz}TJroyr_top)KHg(U_DB%jPt8Zy*C6TnG=Y%2;_}MQP0PZ#JJ(u
zu@|@D(Ts-?s2RTWxjFMaButLsYqfjAs!(Lj-IoVhsICPV@P;Rb)PJ>*tn{P3N$OIn
z{aWq(y!0DK5j{g1%K~1js~;kL3(ei(zaVThgm(}nb&Lp72Do;Z?smS#G
zj1gDV9Qc3J(`c23cqae=~anE^77k}hB3J#TMnD11z~i9vIv?cc_W!hHpxhgS+G~W+Fs
z(w&IO=D%wj3ELU;70UlO@fGHrt$cLLGod7H%gZv|$EFZoQL+AKq#Z+~I}AWf1B+K-
z;Q8Qj1;W8X`hbkJig+^!BsnnY+mzHEmbr}A!ai~zlp#ocpwJ6u9vV!f=`Ea!<^hjm
z5AJ=l9$Vr>$r~#7D}{!-NWI8TJHfzRl=@T4^4Z>l7!WV_v}_Bf>L7e#5TKAT|I2XU
ziF=$0$pEC>SQ|0ANHj}mMu<^GVqy~P{_9|^KuNCr$H%p=m{~qhITPR+d!CbZk!O-g
zgp)z;&|XfF-y!i8!+q6YJMAAc9o(UUEMMY!(%Mh`Y^^s^%s*p|DeM#GCumWZg<;Vg
zYr`;IDh{b`romy*3Etk`OJGRzxRsdxa@uk!($J6j<(vx#$KxMRXMQ67ci_eDvf`)F
zXSSpj(GE;QaZ4@XFpR551(*DkHR
z>sKfYI~sP5b(}z&u`^0$Ez0I{AQynrDV$T-ieQg~{6{
z6+2FwXe7EkGrcPt>L~2PSE#nJEB&geoM%z`31#~&r`-M+8Q(kB3pW=zi@N$LLDdaP
z#ob|r0+D3V96CFpV%L7OG3x{d{GLXtaG
zosN4J*qbf}mC4t!u~@GT_m%leRuCDFlGlX2W_I6ZyD9?6u(GY|iP-J`8r>R#5I?iR7
zFAdZ@n0JKhfXjID|Iu7%?J|M%rU#uC%-t~o0SefWF@|~uS!4?_YF=AB%mU-im}i5m
zc~nvB@jM;#b|>1w33N
zFj5nNi!o{9g*0_xR1C!26l{}iFt1VWhMfho?#gXL!AZ$1o0zh4-T~F=h_anbqxG|8>avnc)@Zfv+p7DyDBJqP%%&F)By2;4;!CY{Y
z(o-ymEb9c>l(~8{oboliV>VA7(-rw4(NQO49S8n5)8XUtbkN&sYtuHF79mMu{>X`s
zIm{f*x%Ps3piR^$a{v~X@IXML6W|(9euMmT02Zl4*}K2-mLtRzh>aC*JTn?5R%;oU
zbBQ{G^oY4RIuP6q%g@hNz)lo%CZAl;rG}QuW?fM6kSWPkS?S-E+4!K@2D7zh6pS%hQ~LnYt#`jc#%sg!J74bv=zA)@c!l=)n8~`iZBhh|h8i@qR@E+mWz@nh=+|AA9FDfef94DsZV}!3u(@fk-
z7#N@9_{xDRf_q6g@-
z%*RTX@0OI5tQ?Kj&4ldcg38cW;-uKK_h~qkcQ}Z``=2+zgfMLRCp0LiAxu>M{mc6T
zR!v)Z4|fn`cZds1hhd?>;9&G;=tV*O(}r*u6MB`a
z>oDA)bANga%h&M(tQe`W+?yC7hRSXA!1$1?VJXMU6lNxAs$_vJT!LmqXMuI<0aYgG
zY}61wx~V_AxHv*QI?Mw&=hr6)om?r3B^vYh#l@l+?>>P!{os+4*PX<``pSc2&dJaJ
zPHd?J?hDfiz>WLC>Z%q^VL^fGj>|YIVDhR&Dr^8;QFj
zj0qSV({Olo7S@*OKKnN;%GnVM_vOJuwYlv6FHU6;5zr_V5W<%5sbXdk7I7z-#z4D0
zR7t)9O8
z#0weFtsgvkl>B*?A|nQpp&Fry*?vq?GVV&vhSx{TsqB2-%ez77<;1(FO5LV-Oc3tJa7_gE4!TSeFR1~OM0w12cyk0@icw=dEj&fQ*;Yo`NS3~0L{ubpozD#mz
zCnx#%mhHHclVh32Q{Q~Z(M+L!NGV$d`+aj9<&ndOAOC3;$D1nsv$L~ni_lp;6>}OU
zh>`x`VTCm9 $J01AjfA!tB&
zFW;U=dL4;K!DMtoa2;Khd|6+1W$-cg>ataliDKH;PahOu%Agq=tZms*rpZ=qYB}z&
z_~{#+t)lM)Q&3Jl6&A#%yEqJ`sef!NJ$DKyhEM={o*hLO^UoigjqL$Vw?l~`fFEE0
z8T)@#_9oC&x9{HgmXxVN6hcTMDr2EEu*sCnW0I*z%2=ULA!8*&DMRM5M5ZK)%q8~+kK#(t>z54;pM=zqsfJ|+ai)4BH8qikE
z@5S|2XC9SkP;aJs)y%)qWV<-No==e}CGfki=3t{Wx8Rdkkg5RgZAsHy54$sYPB#`}
zeV`@;Bmv)#SMiZ2xQatb$z<{3`{!qoDS@S1Y)bWkz`9G
zH^T7yLn|vZwGHxzW4PuCv6`iQcCBDZV|`1|%YtGFaMB5|AmpEUy?SnriFI-7C1a~r
zKo3oLSN%lLgkPKQHGdL~Lom3E8z#o3paO{++UBykswWiIa_VNnav7F5Ry=qwv;0sE
zyg7^m6FVSg#6}vqK|jAuH^Idv%^Sxe4{5zMwCVC*uab>~7d@I2D=aF;fx=AsSF65H
z3gIz=V~RMkQB}Z6(9(uVgd{NnvVQ@#t9zI};)IJR+NS7AB(uF`Y_jTVqR|FXGColhz~X5R+86p0K^Yl?_aEpZ7;t?65+v#f
zD7tM54uX78B+9S3>Hb}s%h&~Xry5ATdMPvRWduZ<@96xviurnO*_@Y1Y;pV82Xoc)
zw{PQ7rlQlKj~I7|W?q5@si8Rb&K&~xz&4@{m4e@khIM!{DLAi8H>9zciWIp#@7sPO
zf}_fE&rqU%MVzAft?PFUU*=CS3e2vDn*|zFq9sPHNHFL<>N&ROQTSwi!U<$q;Dlbl
z8vja8g6c~Q5xnk_+ZQapxU^2Jm#S)ez;pD9QEO(zn{uqs3>^`Y=!h#i24FdJ+g2`Z
z$5rPAW^dlTBp|P^1c1NaBp337ucb54}wAMb+7@D{>M{0@RrRV^aX&GrNN~NL0k)2N4_OEi@Awbt(78F
zqilFI!qZ%eS4Th0mw6w_*U7Nz`t!c3+e;UJiFI#W(N^qxSYw@R3ix(3UXL5k4PLAR
zy^VwgI_#wy=tQ70Kv}%$`ry|fELlRCff-E!m|pKD<$kaN(RM&{hY_(0t4$2zM99$&
z3l-8?P&L4IazFQ)VUSk)KkV+So$+^~s%xc273-BRzR5I8;*92Db`i6TlFc!4{q{mY
z^}@$XCCuS0k9EMhcjUaj2+o;uVZPItkRhO~0PZ6)FZWssyMwL>J3b>V`UpV3csPQG
z8(J(pL8jqoX9Qw^VtLgwczb#fgM^!66t^^-+6tPq9D+F=ZaW$PA>Q0v{*Q2O9j!uX
z!D%ziuOEA*(T}wsNmUJcLt=6ulqHc}xF=tNjDfb|FS}{q
z;)ia<4L7jG)!1!OjUP4nAANl@V~)p&l0VYlB*?wKMIsoN0`ZVS*+WPqoi&jlnC_tf
zA%Hz<{1XIvhpy5
z>+q^M#nf%5YD-8-RlRtz94rwUNmNHD%DS1x-%R82N
zqaVza-u8=APm@?7l+<-3xfg0?w0|(=M5A`fZaTY}RFtq*5d={o#OOkhUY0E$m_e>d
z1fJa#U{cqo%;8*6hf2bPhd@(4I4)K=Bisv~a`mQ|Z6uEvt#(t}{(zPMXhMA+9x_;9
zVjHoMEGkSfYBt^rO
zTR$ukP>hF#l4b;o9s)h@5l~jFhE*yR7Y!sYw4YlKZcNI^s6#c?+TDEumUpDKiITOQ
zLA*|2&1L#U){e-P*l?eHcTzvCfq!mJ);G=las5@6zP?hIO|b8nmEb0-gPm!^Cbn4U
z9_8F8SWp@L`|BDWUb=LkfRAkd@~2N935f`}qsk{Y)y7j98E5>=zeW()X2i!rd={Bj
zz^Ydw+SkB1i3!t_;XCwp`t`mdg$%MvXKQdJA^7!4~pfe!9adm;S4+_2%b+XG1uJ
z9k#zjyCzf8e46f_3D=r1M)S>%`W=n_joLd+xl34jtsvwDW`%2x*v)Y{$*;dr&qjiE
ziJ2MxB2vTUILv2QL)_JV(Hej$wpIJZIF{b&Q~M=yz8`5EaJjcvcOQcCQjFM|3%JFB1-sa*OYXSyL;Lw(|rw+Tnyen5|?dA*7ryg=C~!#0JB
zp83mJ!!2D(W2z8Gzr$TS*^NZ&AvBB)eMm-f1AKwpZa1kt5h^#?hH(vBAryMl-oHM+
zP$Li_1W|8Y=p&{JxmK2g=e~W1lLZh{+oS7e{-{2S;E>@h<noapqSjuk5v~JCbM_?e{Htz@8&+4WsB8+w*M>B2~`5Gn)g(
z9nF|Jy&JtfJuN5;pmc%^iIfXR4tfQu{QZrYx6jWn&CjhOpu0wzC#~cz*(eQG#QNe?
zhISC*%e^M1rY-*bIe(GVmiohpuPxC{5m87$+U^W;}eH{!f&s{bqHgFH(-i+l!%4t&-xa9df~+p8fV
zR#&8rZ~Vbu(&>2FgRef*p6Fb0=b*+d^HS=@w}5wNykq7F9s-(wfN1$b)uy@N0)eIg
zmGknNJ!TRl1OX)pO2XC~#XXIRY$XzR;Finl%qLYGV*Q{_ybLW6Pzo;O!=VT^C}CK$
z2FQ#&iOmDFz`gTSJ=nB|MMbY3lnA-9UR0#REv=U;{AGKz)Fn{xB6?MH*NtrSPOatR
zYJ#JVOH6z=6{;?r9vw`QLQk6+fP_$pXalALIO4&7C#&AYFQw
z1s=x4rX;?TpZ=aNU
z1Mjy%-uJl=`WjI;B!6f97+m^u5IT1}NB_Sv!g_-h3#b%`^&KDti5x_J0q#>4KE7*+
z8HT9kgF@o}X2Nf1TugocWTqkoKSnsVi?4ugoGgUQfRSVr;Wk0`F@*v~9#Mu(IOQEs
z!jxHacp5{nK&=t}T-~)#ws>&KZQqWK7HO(=4YxUdIENoIJ!wimjmtF(Ck9Om!exlN
zg+j8j0kXE(C(bE2RMwk;=xnHXuf%SPavvf7fb!AVZ+lT!2j{&7s0bmupktCA|
zVsYQH=DQJ+TI4bZwT(I|BlZG-BsSz!kp#Da_hVyJ7|`TAXH$i$>FUwlb{2x#e44xh!5v;M$P+iHCU|
zoc#rz54V{g%K2Ka*Ce>3seTzZV}M|FgERIh4ar_Z)o|Y1+ik3#6=61OZsc~DAsmH*
zoJFp(1huv)o>JdI^9$d8Q7>PUVz|3Tj&?C
z!Qo!f9YR^<1IGrE>j$nJx3&f>NDaKAaXod}2*@z~?`Qx2D7MS{+Ov(LrHjLFIVRXf
zMq75+?*B5IAoHP;U0-vz$L`Ro6@_Pps13=-O=7u{^!D4I4-+pEpIznsYt7Q|EeqT@
z8b^-sfWL!3368Y}1Pe`c=OvaAH6TcbNDa1LHBiQoBJo0`O?1u8|FpHkD6F$SRl1C-
zUR%Yhc4ztLKbFOCH{0P4?YXc37Gyo-ehcV7B6&A*b8o#q!Wfh~xu*4urQ6wG23vPc
z9>%-m4wiT}bnUdveN&C~0*n!P$baF63lV5G8q4=+Sgo+*z(Qwk(IipGXn;t*3BH+X
z9AadY0`wLgE_0^Ujc@4F^a76V{KFdG`gP0ZDj+wo_|3tz+#W>H>-U1XfEpXniQGYJjd^&^Y6bc0NUCi2V&!?ZhVFd`^G;0Z5J9Yri~C!
zFup4hi?Tu-$Vw-`58S?Hm(m19Ycqby!SNptHd#gbb}Ze0-WJyCNBzUPRI32Fh!{1S>IOh
z*b2-XM12t62H-~}9I4W?b}XLA-?aWy{`j^j$3fi+t?cPk4Sx}Og5AD%)O{Ew8o3aA
z9$h^gvifnSwo`p#hE;aKw84ekUDW$83ywh@%(?4yTy1e`H_kxdbBF;@N06yKX6cY2
z6@+C*3t5Bi8)UrDRK&vY)r}OQyRt^pgsHrflM#A^a
zh2}}c?vBV0`?IN-BUB6(BBJupkZA;J%rHdUdY3KUf?E-b
z?^h^SrFl~7&B;FlrC316*|~P})Z<$(Pz3LyRX)$LZkwT?|AKGj?{6DOkQ#vG{UZ{E
zq$Y>?j`yH&!Cio@P-xUuS1Xg+eH7o@;aI76SDe@x1BBw0#*qxIgq}WfRX_CMLq-*u
z*OJ);&oMAD*(YHML6UN{Lkz)PEj49mg7RlfDmv(TEgq-
zlyM$oD+~3f8*BwsKnIH>G`LMWB3fec$w@Q?l9B?DhibsYIzmf^8<12BtPn*0UiErF
zvP|~KHKVk(
zbRkEL5?%t@BvmoQ3l%@!-_6}q#Ili{hbK%UEg6^*t~U)qasL=_L;fi?jUFWe`Pxsk
zWVXv;tE!@C`GK?2^Vhfg1T*QkbC8NlMM-)D)a-8UZPXB$kkNpw1o%UD!%9IZRT&5l
zB=m=kvR0DS1W1=O=Dbv4_svk406Uh3wSxex#Z58$=22;iH3Ze`Vh1h4Bf2qHM9|-l
zkXYrPKtqvO9oQ8zFcN37>=6{#wm=(EEw&;T!@_^7`wQ$I@JqV*m8YI+pWSX4|F-E^
z=;9_Eta>y3zpARMAD{ms2%xLuWT$|Dlp{DUPA;xi$QfX=O(U(!+Px?w!tFi`S8}`u
z@0>pH{ZqCZ&dy;p{>_j11DZE)3+6>WZ30hXXMtnZmKn@qT)2jy%O(yw
zD88^F914v^*x@wU)kKRue_4-&e>7VxTsg^i+92-;5esTu_h;!H0=PmTM8F`}$teS;
z?d|I!Tt$!rAAaOGzP&Ogy)kYa7-PuD9zm(eE~-Z*h`WQ?mTG`n???rYzoTumva{R#
zNK>U)ZUf6poT%r&jmXw@t**&8+dWzQ>9x+rU9c*4c$^1C+uGXN)89`EeN3QI691mX
z%CbC&e+ZRd#&~#)h{rOpns3sFUQnL01$dvu^bi27X#X-cx_+VwK2?qzCHY1YLqh!V
zV@C_c1(I4391{T(h`N$Eb)6Vu+_4L3@1Ycip^WsIt-#(XD7No@{E#S8ikngkw?7TB
zCv7t;ypeHM@Nnd^`-0tSf74&MOI|yT&O~mYdARi`&2Oa@lRF)Ko@!s#i|!Kzgsy`g
z6bfgk-T^@6qb398M$Wt+pz%8zuGd@Sz6{qnK6=91Y`g-!3>u*k=p1^_UIZIamaSfI
z=_9zCxL1T^xM^*q=jK+S0&4Ji|BtPb{`9-hNf8Z!*+kseuDE59tb&jaThtUpX^YDr
zkk>&hdXNYqSRA4f(vmH`H*4|uMxg`k$P{Se3oYMf(eRm^#P}=SkVxim+DfxWiJTYQ
zv`K!#m*wHJe6wFN{S}O%&9V3VMfi#LpyB4ys(p~wjNnv%hreERc#=^^IAb5XjIIDS
zNb=ahMSTXF<+zyi1kyjm8)+h~2pcnO*&3DI
z9LCp)#KYcGj4JH5INC<;a#`x-y_RA=B;Iz)(a9=KS(icgGb%S6y>xzHMbi`LPxm~e
zfy@i3f`pAF0mQft#!(#GVZrXVW|O`6)18FXpwJ{&KnndaV
zaU)|6h&!SzO^oZ?moIp*t9v;@Mf{+cfNcH*!~un;I`7Q*_)yjCClxCJhi87Yjl9Cj
zs4GFbZ0s{P0E@{fYkgOShle}#JQN_}9-ZrhxO5*D7aLVRFfZ9oVlcpK+ajZu#HpVD
zVvW68w3`M2`pY)4sLHc4muJGaU@j7H9P&%4&%uVc@*Q$BNcg;@Wo6z(a#DdblutV?{
zqN9TF9a$f{;i|$f;JWF4uXA+i#DqPuVZz4P7MJxP
zFXP6ngVNXy45mq!7ZLy)jOzNtBsb)NI{?kqAwdDJdVKPT;5lhziDT)e;gga>9v2)5
zY-3%Zk&EcC?5!=;sUZmMU9Z%>M$y}>9r7i?EgHyxvf~fyBJSMT;o$%Wu%7O|VrwGJ@
z6dVgf-~yAe#TW6V6rM>uyXtckF-c>tBnp&X)SyVDnIslV3M#SerNp^Dlxv7`O#9^}
zcn)od^=R#jXezRU5c_19p17Sp!UC}2hns~ON6hrhHaFMRT?GczxF9LuN^(Rq%4V>a
z$M6@%P~z@}_qiGiy3HdV;1y9yEu*DH#j9DcMF}8GGR^^jyzTIr%%8<4fhsRFDaoqr
z!IlD}Hoezgv2l4s5=Mp{Jm3+cp5D~(P;I@Fu`ia}E2th3O7sq5&6=hrvAdhrD%p!=
zy!RvY8=1yMY}wdWe|}Ewn6JQ-w83@x-+5fxw0cyU>$B4mMnhzilzxY3nYe4%c+ZCv
z&pDaPKeA$saQL5~%BAO{=sqBEP%oH9{33308c0S1*ta-ADHnsj>iFJ;WlH^>gSp$n
zeaZ?u?98q(`ZAQ^LoAKC!n{&UC%CVBsZ6^dlAlIf#BVv2{x4;v#3hobgX
zF};W%X$eeM9OOUk^{GyAyqcfa6QmR8}Jq`kqxCFQ;<;vCeCV>9|2L~ZRaU#w9@sY71`Xw=)&
zGpjBfQ(TE9-tpooF!n`ebro2DrH|8|HpZ&OS0r4tUoVS<#5jY~2ggo3+-KXB
zb@<@+vYM96h{etw)T8+jGVVOpbI+$Lrf{+!T??5bhwOghtwyZf)J~m<@y(m7o|n9S
z`2MrH>c!l*?X&^~`gnxWe@*r<7T|wmVk8c4oPg?Gr_scN_zOhiLE>i64_?jBKRbG?
z#&^JS^QWQHmVL5LT351H%Va7X?Z$AaHY*4Q$j}}Vv^Na_3pqysqo5tdXuf)>+JMkw
z_KtZ8-I1LI*=(w21-}*r$Hzi#dys~R>W*CT_zqBArUDp6*LqN~>lJz`Tn+ljMGOlI
z!wADYc)nX8+9DE9Pao57(sno=kpCfVvncQ~EK25-j%BwM-pU6wu?`OyyKzI{`i-9S
z5ol}btB?xsFCkNBZylx$4&7PxOm0fJ!_IJXSchCstaQF{xcI)jmhcM^%$4T93eZ51
zIRd}{9)gfY_0J#_^Cj!Zk4m=fz;{R^j}zt&1S7z8+!Q|U*?}}ESSR$LLk8^}=j1!g
zBv5b*!s`jMV7ncwzGV`X>F*t?YwGjorEwJt-GK_Vz`+v`O@n)d
z^;q@JG{Ln{7QO-l)UFk1*ZVq(h`n&1&EmhR3yM?uy3pdGzk%TF9>yqv$6p2~aRZQ=
zADGGd0k9ni_#cS346xrfH2arsYavBkqz3k`4UPbD_NLV^j;RU_B+Qo!
z&=D|@?WfLR3?xOgC}uU8>_no8k3420Lj~@@;Z@#%kSt)puHbpY6{>Nk-Gf33E2cqj
zQm|ii?TGHywK^iHS5(Gwv8N8IsJK9-#_ytZ@EHGf|w72$GBcLnf-T0x49Xf<}3D`RUWAXt9-S^)T}nxz1eh)1`u!pER3M8k4b+
zcAPx>V}+mpg51%SOUkWnuD{fkz~{Fuxo!t`vk1z
z5YlyV#}I%P5AkH~9Rmc75_TJb63C;d#u!=jD$?!~EpeW1o=5iYrvSO`$T7Z-06)z`
z_5!mE2$X3?r(X12<-zPkpVc4vccI<)3@7_uDB0e^^ZDPM;p(FvsIMcPG#maSCsiyH
z!ayo&d*s%003y5vPsoLIAbTWqA6++bVo^lQi&pJi9WA-_RDebrHBB!mV@vfG>bD!Y
zsuy&CPxc}T7bnSI6Zk1IeV70(&?OOYf;cfm*u6~H-Ifo1dpy0E|I5*!(SHrkW@_zN
zc7j!eH+U}uat0yK4q=G*Kwcmnl|JGSGy!U`IN|Xdy?!hq(;@DW23GCDkAiE0>~yBB
zdZ#(L@21?q3mn3IE2<~LPFiYS-uomip!cuOWt-BN+0L{h`J(R__qT-GuuDlK`MW9d
z9^5>;7ADm|#99DfQ6Zlk!VGIDWhN{JUlfdx_fr|Uc#pkm`>
zrS(_HA>(}oS!8(b-lgat3hGj!mm(Mn(P}`lh-sP0+jT{$rwxndSM1g@XaDOFPs
zV_M(pVj!3oX|Z+A)0@cT>;R6UYJLo
zkKMk?X{x_z&i{HZ+US4&?TXE_;vcx0$6=>8nt2>^e8ZCF4(Lv*kokNynN4Y;4=_KB
zhQ~4CZqf<*Y%;qI^;4tXlwh}}S(tK`)=3ome;GAw2@@hil)gY-5nqt+Soq7B9
zMV-ppG^rCmKiUD(8+UeBqPHL;YmrUS(qjY>AcQg4TM8i>L38cx?HOiw55pj{YX2#N
zQm={YEPXd@#XErcFLW1gn?V&(e;m1M_{s9>F#?$sKE71gM1zRsT5h51Y)u
z5U!f0=kjhyDNO5rW}CZ2Kxh=@US>1{8U%NqEEd5G^zuP?Zkzl6KwRojuB{=AQ4KTh
z@k=-hh?oi6!WATHTwELo4c~&Z|E>Gs=Rg?CQr>yb!x}O92h;TO&%8C5d!;ebbt>~g
zHQVFcct3IflpBB?SOb&}w6^ha-m(9-06D`Uqccbztwdq8r9Y3*wf|uclB#iuLtA_d
zvASd+1IpSz0&;2S|Newjn``*Ve=2)mS$0cO~gQbcT0
zlGCTxeyWS2Em=1?{ZALB0VY%-Oq)K52~)~+i!I4OSlm#w5}vcR$Qw+^eT9a^0m(;#08>D$}x&J$hlB*e9y;zNc{9eEPt*nQ^n
z&|s#F@x8Th5hrKdI=kiw3JKB#JbJyiy$AV|=I>u85=ojkMW{flHygoO+B|R!YzWYc
z^-Ic+y_UlE_|hJ+`Z|)VNk{_Bhk5ZPR{~H+TWJ|B{zkCcmMlm)DSwR!|EcnKXN{@P
z$ek9mGJdd=Qu6xD_*@J_fzDM*>Iu0)4C3MefjmBB~nZE43+IUEEhZ#4j&P)T1PjQ0>M7!VLzP-NF!(Q;J
z+H2>%a8TCbOp$k=xIy@vpf#?puFL*x*c~S%A;E&`5jc5TD2f75)DH{9*6g>Pkmg(T
zxB-(er6AXhMnJLp$9~+is4h*+%t#$$bzJz7O#sVd%m36^PqLk&K*WsDaO17&K|IDi
z*~?$8$WtBUcgdO+^BJ@;tm4X;%Z%PAjo{QU3op7iJoAto~Y<4L$NTMPDJvJH5?(Lv!4P)>ykH2+BAbl3*@8-&ohZ@!CHRfLR~C?pzdw3#%)p;i}v
ze#jv8FBF3$2muUz+Yt-hv2s*d`w(l5dfx>o0Le3hZ5QE-OG7T8@4-h3^?pMKfYziV
zFA=#TXkz6si5^l0G94BfQez=~iXgORH^Y3Z5UuYSYpZNkwSjQ
zW%Ql*GHxyK``}jq?7ITD9)j0MMAJp3bLjal{Fj5;qKUs4?;R|`$y46v=98fb0u{yl
z3`N;tcXt`^0YsUAnIKyo2llTZK_>%lhiuGG8F0Etp1~@=+_&-2xK*4eg9Zmu9C=j$
z8Nc@#D*wJqXMvJ4yTlQofnlnGpJKpw+FO1o(HnOmzT^I
z%!ZR?w_aXOZfIzTve>O|3_^iIBI7}*HQbRc$ULasWO(sw?~lwO{BoiJSZn2nEo6fP
z)}z}Ej8VaD={HLIa4fs{2UT&mY|&XfTK{_Q+}zxS$(%fAclYJQJfaw5f+-qcqTv0#
z*TljUhA;R;Ef6qg4-euuWL~hJtZ>&r#2zds@ln76%Jcs^dat%TeSEM(AS6kZS-9
zBI#2eyKcs0Ra>wCL<+4I*BYz$bRR<0l$6^aJR;1w!FNT#mB#V4hH!hMkOrQ;_W{Y5l?lNmMGZ0TU|^iIl#j)f)+1LtUk
z4v)0=nCAHcgaBpsrG~`SH9w?FZ8%V1rpJ6|XXLsgBMzbjl}>zFg^Qjb^S!)`XZJ7YavPr`e2{Vu=@KF=A8!ls39-fy_
z;!}t$hVc66x@-SY{u2*QMa5q9WIBmYP}kv{2!SNwGq@o(%s>O%=78${S~n+&0Rmx;
zfx_Dl=82Ir;&KYEESUN&etUrkgGhmfh3ikq-O9YW7v&--wHLi-)+7w)e)Hs
zx*nQ-*$=D!~e&6Mo
zjsWdU164*&Rp>k4zr4FCzq}tL9K67yfSwzO3kB~GJnPZlOI7Q37V`a${8=O=z-Onl
zhf!tZ-X&@CEilvVMmCwyc7@9s=bClP%eMpyuaoCHceV+mcOZEz`1x!`Gn+jk6(72x
zIIR}A8I0_>H>Hn#IJtDo{lP)?aj(hl?ekv83`*%K!+Csb
zoo(g2zC}yO#1X4+_&QMN^b5bLw08R82&a^
zcA8D0inJle*>6qMajFOhg{W~+H!Gw>E<|ufwhpmvQdJ4aY+Y7oFXggHwQe(Q?U`iV
zu(0Jdt7Jv^H8^39+T`{BV&aM9Oge!X$!Y%c0M|^mO
zCrqKhTwS$m@Ip29c2H1|LgB49v@O-Gw3K7xb5d1oQdVCEUp;%dZTgs%`S(Y#2!CHW
zprSh&n5p>bvV{
zD4jQ@qTVzF95KqNPRO*8%C$ooNL&X*HZY@uHyk9>SYJ=SPLao|>+L!Ucy|A#U~}3_
zSzMH7(&vR&1T@o8>g{ADWTzg)CFr`2OI0ihGBA3+eRS;cmSXtOuSF*KwDSakstRuN+8^jHvTO5OEsulo3m6sOk7
zu-v8}-sQvKHGkjyXIJa!IZk=+P$`#B!xKqycXY)rhHTh(Avn8jH^tAAu~}f=cZ+C7
z!b!GBt8Jwq*KnD&er)!jQVqs&I1&~do@@Sfp%HO$%$R39^Ds!QX}NIv!-v0})*oA+
zr|bMByT7-)BX@Ug`44%Xl^u#5-R*WPwFmQbJ{4xD^PZFqZq^ozz(5Bc3|b>O^eEx7
zW^RygH6(-)R7Z+srH`7r`T-XgNwVc9Cr=7(-uz+r9>F&0O(|us>dH)eQ+zjmU)Owk
zqi{91U}S>?2A~DiSQ8Ivumr2FnC_Q4o%m@Br_v3sQw10Xkor;01XM5OVtyF;A)W!t`E>}`IlS*X&CWn?bg0Jshr*9W4=kwY&trVx7>C)G{&Fj*O5?{qhFJ6Z4Co1xZ
zCu6+yZ4s;=dRxQ3bTV6Wuom1FVy$Fb)_l)17F?IeR7|ccN_}N
z%+A&m%gfC415zA0uTb?XaAx@1aA^^5oz@e%6WFG6m{Whq>9n(5*0ElWBD=~2w?L8b)#3L0~OqU&Yo(^zd+$-W&L0X
zm^B}8*u|>{s4GIw1%kK@sz2t^z6$-app;3uG=3YEx+1he0vm>vOPstim1_HtyiHct
zi09T)xwN>ERrU1PAkABjBLnFYNrv;#JPWV|cN|3#)b9GiLfN~Hml6^hnvYpFK$WWc
z`!|5qCvLzZPN(kH1eblje=elVctoiA6I`1Aq?GNh({G-kRvH=|rOh`9!>3bAYg;!Z
z%Fc*Lrd61j;fG8B6gz>SS+n9HB@M%aFCxOA|m#kS2>E#ym?w)PDZIj
zO9fLeBR~pYsH2#Hpn-t_zk&icgj32I8jR$Dz{hG}Y`jTSG^@7^GN3vfA^B*f@NOWW
zSx-preSf~JL0wFtoIH6_0{k}R8Co_S3yW|_n&`n9s-8TV-SsPDTOVJ^%a>YsG2%5q
zUHk%_WW>#zmm&W;28^(5BH75^`%S{F*x2Ba5o;%pG)LVVT@AKU?_T2(8bU=;VK|TY
z0hpky1B$CAXZv=4(u@(;#T=YNy5{CN_gl&ezg_cI29l(MF%XD^t;o%43F^S;4nqvk
zPC1V#oC0xaX?;ly%BVqRQ=9Em98Cdxyy~LEVcddnB#1LwWps+Ci{roLT1{R6w12bs0R6)26O;6vC_!h)XTs>g_pLcZ`
znc`Agf$tsDIfSPy%Fna11uw^fY?MmU34#QX1Mk()!BU9rId-V2@VZbdN5@bSZdWX=
zkUH%GfuerSZhTBMQ?Ns+NyoR)I$bw^igFYQ^b}C1+~Jiu)-herVT+=NI`uLT1=NTU
ziHUhLH_8f8XeXzquRz~^el&a+25Qh@g+n>%9{l)Zyo$Oy18f7ocE0*5d=eRuO3_pf
z5)fns6(u^#<5M&`IsHHo-&Fc;?Ck7Bh^DRWa%1D}0EH^(AE?KVmeYDAPrF1my(7L?
z&~tQXde}KQ9Lm>NutpBAJ_S2qW1^+!I}Y4pC{6{oZCf?-^n1deWm8pSuJV*)P$+>B
z;G*yYtOCw^z{^VkN8X0t_k>?5z*PrQdk9v0R2piph1eKSoeq8c7>2I#;??$~>A}j4
zn>TMJH%^7}2Lri7$!o#8Jww||kOACDcAZOge4Lz_dVK8c%d-qZuuG>liC-aL?ie0z
zQTNgIb0W6nE7CPIOfx#Ft4p(C!`S=mKY1_r{cx-J`a~++W-cCrJ+*UN$>VP$k
z{}YFI3LXw*G0L#j$+pudw!SY_^qgG|BD44NqGICtA#u$M07+%Mv{+aXKG}$4q`)rb)k22G3sL_nq+{MTBb($=xB~|a>LO?
zgrV6TMjTsX0h}^P<>fq}5$;Tft;S=d9D_Fmfx_%So|q%lQh#VqT-sSjMrUJj8F}8f
ztmLEIxN*ZTDvEvEeD>GTI4j=sKj*Ss8R9XGX-lNPlMx!0E~K7jfWUNZx~r9fBN|G<0mz69UY^F0z%1oh=0{#JBfCQ04FGGbmp!M@Mr&-9j{M4}@AZrn%1DJIz7HLfQa{
zs;{p{@;5Es*TQ}!v7&QtE|rP0DJ9*o9FJ&50>UwPHVG1feTakg68hat#mvqgZxki#
z02v}RQ$$Rx8Y9XhVq(%~xtuWh34r?0l{I^xnV)$|2Zpp9$I*FwA0BP{oLaw?cIG|4
z8xs>)SSS(|{6*R&rIGq(jZJ_z6fAuOLE6d^TBzajZU?&-_X8b4IcLqUt@#i>G
zpiWyytXmk{eGQBLZ(kt(&+EfY0iOpaEmB=EibGKD6yTay07MYp7jO~KuBqKJS8C;}JF-yEEu-M%e|V}_3I*5*
zyd_AT3&52JALla4jiz_RVh!oqb(|A6II7h&G?Y|S0yMm_4xo=#1-BW<%yk)<2Rl1^
zrbS7Xd^O}2s+f+1#%%>gNFm{$6=rOlKjy{Ec$aiKyY`j-G0e-CulDrxBwhhI!E;VTKTp^;!B968o7DwHeGYehvzXT^D=*T9b9$5jm<4Uzy1V|`%1Dw9US
z5~>*vN(*qrIK<-zJFk4U8=sKipA_WYF;Oz_-&cWuJ#aNIiBqjYk4Cu>73Gg=hYhqM
z@~D(_by?v?4aMHaCKk!qP5hEQ;Ms7Rut1c6tDg*j%f$C68T0YuDX8y})IbBT$bPID
z=h50#aeFUvL!*2f!Et0O;%9`mY$@qEdf))fpSg^E5P&ZOE^>)XMMGpg9vbvCD^aHE
z$}eH_K))&rM(jGCk`;E~`K90A^O1T=+3(;Wio>ZT<=`5OM_3Qxw#e44Rfsw2?;92t
z6r7yB&8&Y_I6_Uw$Y=%bI9&4tnL*^2JW94;6o?15Mq2~GChpgQX1G56fD+muTj4e=
zrUiTODDkEa;5a7>q@X|wKT3-t{Ta?}Tz|_cWM*FU(&Jq!!jS@^q6`pH67=>M3Wq)5
z5hxT`YhXv)Sn%i(#F(k;ewE=(qekc7w22l2>L`9_S8d3ze{_5m9$$)U1>nfG_V#F(
zrgyGn#^=AM6(J5ujlFH#A^FbbNKad99u^RkDMqe0(j?((tm-MX@#RX0h`5AlvON2I
zcu40r%o%m@L=-@X4#e~X3q*b2^5oo^ck<9{8fQgDN7La(M;U^ZBx(uV
zcVgh_6W45ITz4qXKi6(JBeGV5jPN@x`2cGGMGQ%s+!`!Fi7qzN#E0L#djsBtk#H4-
zTrSvrWRmO+HKDB@2${)ty>EQflTIp@85IF71-f<`3KIL@X77GWrLO8c+;noX)0x}&
zpkRa=X<`tNmMv5Tv0|W_P!%vickaWQ-K*BDIe6&MDk*7c5AO9}nk77()z#I3g5#VZ
zON-G%Xt=uzb!h!NSAr4|)G=naynI%iablFQk9i0s&L6&^~c
z!5@6{g7lbX@$89tEc>nVO^uBmM)CHUK3l7C{lEF>dH?t80iez(eybrY?YzdZZXKzq
zp<%y&GDiRh9s1Ofq32gHIAI$jBO}R>Ia{Pc?YNDj92XJ5bzW@DS_pw42kRY?yyuZK
zQP`oX3_+HPkI!E67S4+A+qV(ajDqF(Q`v4_OcX=jnhO-2ysD~Nfmuy(Ze7t5t_D%x
zd9AanYikB8Bmq3>>FGW2I?bNmN_p32{|dvTFQXDTJ2Pr_M9XZJ*-;0VKSz
zmLzvZa*^-4F;Ngp%d-D%)eMz--*8Fd>(@M}&2!>Q
zFsKtEZFX#{^N>@U?=h+jX$GoII$8|6T%)3*g5#5nLF@1M^OfvlieE-Xh6X0#)nKrb
z`(BmgZQb*8b1#5cda!VGj><4hx{@UZZ(VA7`pCYeA8YXLkfz{*X?*D9hvUx;JV#52
zZ`%Zevi=c{CDaUKsJQ;0OoOdCe4zT%zZOH-**ENBRz8K&h^{O{DCRJqAiFp*kzX}a70|eyG&eK9N&4p`j(|na99{K
zvABX=tATar^Y_XnUVsDit5=_P^QPLCLUNd81Q@{u7g2Np{s0%aVaLMo+G|A~9P#h_
ze2?NSP%>Y+Z5I*o(-Lm?lzJ00RaROm!PO-xT-~wxDcVC$e6`r)U*L#P=4ZKw();{$
zQ8$Dg8GFt^ql$%F|MKMvat6>$hoe=^yf68No49Y`
z7`Vo#7;^I_8&0d|z;#fq8=B@{4$9m-zeajrUjFzxAW3&lr3~lC7`66-v;Z>GSYrLE
zXsLX5))k}SGZip*@oi^a&j_e12I!j&L^X~YveUl|MeG8i)eOm
zy5Y#7%Yf?KwKwEUbbOtkr=u*+
zwI6OQApCM}u9nT+%|%A$d3SH!%9uUH-|3SKOYu5P4
z$TD;sD0fyPE1c$YzZ>_tPj8+>+1oLxbw`E)BEi$&W$(7dO#Q(xz+qPh{j?0I=6SV4ak*2_GY|@EFo~0AfD1*9px~97-Twz(`9D|w{{ou-
ek3W#Rw9NiLSZ#AHZZ49
zXv8QKsx$^Vys}?WYd%Wk_`0W8V_x=1g
ztcfQgidbCbPRextqPPDp;1(gZI`V6v(DdwvPAz?@p{EZtSz1)ZqS^Hr#G~0a4OR$8
z{P#!m<9n(8y{5;IE-0@{{uck~r%(Goe=;~)MEUQv=*O(w|N9GS@~Hpkt0XpBkZ&sP
zUjFmP(b+j_eB8b28oQoxa&Bt?1+Rbc^75ikj(w;vp3#V96^sr4`^U;lFtt7E$`uhI
zAt6iuhFZktGB)w`LU?!JG|<}m90tlkA0Mol%(0bd9&)eb*ImsU8A*iYj?SGJJZT4
zt{wjV>YD>5D4YCo#qk4griQshSwUOJ!7GyUA~;WT2y=R&25|N6JL4G
zE88VT($Mpq{P5|?*`)VJ4H+<%vU>`9<~d`@oP)B1qWP_6ALLJ12CK`eqB
zSk}8ei@1K>Eq&zM`?@#{&o}YjTf$egz75pS*x1-ierynZ_wJq3(%p)Rnc=pib?eCu
zbSLKkym!4wx_q7_h
zqk}_3kul7?!(V&v7?%E8{Bc;(x{QL8CpYC_rF-O|2O*9_%`pq}Qy0gplLa?6KYB!m
zE3fF6jHT1p*Vp**V?)oR;Krj|_vS8bKEZ?^&i~8!V?9e0E`C#QFRzkKRrIxMl3H3?
zs&3EtZ0m!}hOe#>kM1$(p~i_S)`kDorZhG+XcW&IE
zr^F{FmiWXrKMenyf;+6IuTOJ%OCV3OK2-$np)-zbubElojT>umMp3DvI>UYC&edUm
zHm8VeW!krIAB9$Fi_V|9H7{Si>Mph0;{R=cf77N-@hK^f-rhZ|=0Co7{Oi{pN`Xl{
zN~)m?KC9^Hn%}(PIdtey{;6i=V_)CgZqfO>OmqDBasRJxSaFw(jg57!t@%!$KHc@;
zWMJ_qR_2%3|5hf>Hi7Qi*r!io^72d@k9{y`iDfMhnjPls?63B9PWN_mbIYe1SFwL+
zaP+7$IoOb&Uoy8?jlH;xWzQz09m>bg&p+N%yhdQ-QAL3}<{Za9K8(44f8+jJ*5dnT
zLKkMJl6X~Ob8`6Sr$4TCc6RQ5h=uJuH{NaiFm%&~V;@*@bCcd-wW=Napml~%O>a^p
zCnsmnudqLR`txVH_3PJ@lY8-^wD0t{$LW%)8#X+`rOqoYRj#^Ip>!ac-Tb{7DJ*{e
zI5vs?xw*Nnn^r2;2M?YP`fz>M#Y8X9?);b{iixo?g;K8bIo$vICzICJ)_Ct9-@kL7
zx_0c?u~e*Vhqgp+&6D$r*y0LL-_!>mpZnUo`pT6noLVh^&GzqqbZOJ^#L6n6&HfC*
z!NJ)&;bH0Z-@bjDnVW0H%1XO(g-uCGsasn^J`srU}VH=W6QNNucfNGSDj(2ippv=HMPFFKoxA5xGjO-96mlg)fRTmK1VKs
zYW6&@IL^QPKxJ*#4;)1?^Vm#Ckmb`Y)6){aszNV(8nc3N<63d&7Lq1Zq
zH*oEWI$oz<(FtdxEd2Ty&cVa8-zD7KoO2;mJ3l{vaBz^aXV0Enc6B^EvXtB2*9CT8
z-zBK-{f>)OaKkyxq{^RPUNbMvPwD^q_P+bJ&Bp55+I*9S9ydc*CFIXN0y57%JQP-so(0GPkfQuAfuPDGv?Ia6=6!*@v#@?ot@8~
zt=hZyv5-!f5H@NnHjvNgQ*KH`LIUIVRFPy{jveQWE_{A*nHfv{l2$M$6%`c&14Bec
z2Isqbp7i12;XSu)tiI(ih#Q#wYd_|LpFi&}-p_RS@L{b}zq#`A^0?)7QA~34^G{b-
zYZ@3DTDl7v`vz%=kV}qY?}WV+G&304N^5L#drk9E%RN`LIKDbrUZ7(Y7{obm4-3;?
zS(?^~j*fQtyYh#u>7LiGyBos8%TIn^T$boH-m&BC_@Ru!*&x16eo~~k78FQn1%F?~
zt8A;gcQ5;ghVWE0Azed5LyupQ*p|`Phoqy^?>W`4>hVcL*ET;pHQZLUCVw0#i5O-WQV
zw-$MIV~^3)yL-O#lLu~Dm+SS!6qjLtN%X(Cs*`&1*Een(inNr}APN^H;`8&1R+?A4
z(!}+1u_!VXtcF|SRvXTbeg2%ty%xnovFfLUni@yM!l3`!wQDJ~QQv<2_+i*TGCcg$
zGJ9ftJfdZY%jWT?&!2~%oLy^JH8wh$!1Sbfy)%a%!=mVy(>Ov-d2gyZ!z3P)Iis=E
zs3v@zW6hcYBQGz-D_X(Sr%s((MN3OXd2wk|l%!Eg?t!~*
zWo2a%s8VSB(OA3W{(5;F!n(tj8t%SkH2$$c=U|mrLTjtRK(#N8_q$5Ds$AKW>(^y2
zUA{bvo%HO|rZqpm_R?o{#Gp&;+_g(BXvP7{nHLo&wzyc%&(DvYlk;?ZJVQ`ONdC4c
zy3CbL9D2ql3`w^)E~-9I8jYK>)|X16b;u|8_HEWJTebw9JUMdXYFNqKt=+5w>W#QO
zzC+Elwhu%3aba5U#Rf?f($gpV%Ev!EsOvMGxMq~P{Xxi2@=NFqg8%}LpFfWQrXnr&
z%GQ(lR(B4WUp(=&zhAJbs;ck7$&Dh~q49Nr)1=|u^X%Q8rEJ%QLegE1sz5pjJ`;=2
zh;P{<_~_9i0?2JwR0
zVbz!8T|52cCUAp#m`2_62yy7
z#5u$b-X$)v{O(SdG#VTpmT1tmG_YJbQ}pyLLmst36G6G3KYw=Mqcbrv(LC{mYk7IO
z%)5Vs|9IE7FTXkf-T)L-gTK4PDC^4X%%QmoRMZ%xw+}UA@-;$ccfKH73J+
zaYa#nVtu~)hYugfg14}+cq(|;cKk<@g{wt`Y<9rZyQc1Lv$+LqO+B*OSC)fwxWvA<
zs;Eh#B6ue%~~_ujpj+qZ9z`}+B<<54-pA}A<`?VS~L
zX!mY59Bb34`JTOcT!urPVkV!)#`MM5xt8by-P~lrXyh*$9lRBwy3
z$gE4qotIr>yv0o8XYHF%kaaCB?>!dnrH-VWoS)p5DzbHOd^~>t?SlkU8=IJLY*AKJ
zWO06UCi3D%!AqAe4NgqRdAOyfq{P->tQ-gW<&z2>G(Lo-7yl!%@l3q|{fLLwp}K0ZFWH+ElUXJ@BG
z>WT#<<8Kz0mad_;x__wO^!Zs%RAptQM2)Ecjo;^I{L6E%_V@$@#NNEA5X-`U8YL+4
z`-79e*#538-5IZ8USi4WiJN~}xFdgT`19vje8~`)gHt<0`*)w
z#jQgR1VXj#C;QNra{*@EUluT4*x;Q=2*vb=2huCcfR0`HhDlC!rUf@{v~+iK+_bvG
z
z-zc$&Xz`(&li$FqoI!nM8Vy-qP`keSaw33JOH&gSu7q{D6K%!ayGdJDevEbUt#@Ow
zwYBX6aKk=eWNC>PmYkSA5=4;F{(83x!I+%Cw-n}q!<0^37
z2FRCDtM_a89i-}E3HCSVqjpPnbkzbhkgp_
z7c=wov7k`|Poa3nPi4*hsgMs1<1iQSpfXdPn3yo$y_=3AB`xg;===EDGXq@XwiKIc
zA8~A}7?5(OzA{Hbe88gEMMP5c7fUS
zn~r}rp1V6xd1MXNd%Vo93xufd1#tlPnHg-NsXX$E?#JinEkKNHtgNid%Rf3PrqkP{
zCMBn^t?a`VX>mS4BCUX2zQ5;=jN{dm(I1QC6tTNzuv>;#{w(#~^W;@jR5TEM41C_A
z(y&a8YyIqk1~Y}7pFaV3&Z#w?qsT$xcp)WCWH$j;X>6QD3OJ)KH!_?ue?
z2;I%ok5dI60dX$rD-|zfWMa|_2-pCev})_ttPs2TB~8
zoIC^eI`ZQ3mcD9VNiY)BU75|hGd?~TE}IQuH$S8kM#t!@DTKP4>MyA;ZwszW2C
zsHjNS%F3$p7F??l_o`ZvzQ)Ez3Mzvhs(JCRq{PJLGyLlZaex4JgRKc`vNw1O1Wfitg1=>j
z9Ch>x$I6z8m5oM0QGN5~jp54J;9w)DiTPw7Dxex(dWLJzR)xiGynJL^uND{eDyT*E_9lmM~{}eW3l%Cdyl53b>(dw=$U)>G>$LLdzX7+3+MRy`Z{81
zi&g#U)cNZ&IXRi#*JoY6jAFzqeD>u*Q{A1
z`A#PEdD!1Sf!!M-bsHBJ7uA4n-7+#V_+rzS15+LO+_1}?{#Ln=qRA?KijGkyE?PT$
z<(ajO&H7yzH*o(s#K0Kd8@@QoTRwHizQOI*?b|sJ9+JH6?|E>O^B8Cdx39ci(3w2U
z@GkeC{0AzbjMjY*hS9c4EEW!Vdv;_r2-AW}2tH
zb+qq*04G%y78a`Awz=;X8X9WQFLkOxQR$+Vb_0VP^NA*NRWzYiRMGO2zm6`?wu_>D
zwm?Obd9zcHi))p)w|C{y_X1CzJjws7add!BKtLeh%?htl8lPX>n4Xoj_VpzcH?F1Y
zfnB-V_L`eF<9Aw9)^FTs1TNlIMN3PYhfdgq7S{2x#8LrEK`UgAf#QSZB`YWAh!5o-
z57rWiWfMN=_Hb~}0n23P1@+>`e9@vd?ovO$kM1Ca8*g(u<%uP?T0uc!0GsdeCY{f8
z5pK^fBxpSOIzBF@pumiRrC@a)Vy2$Gy(pACF7F*CCfkgSZGQWCVvCuL|?l3pJ`9
zJ+kFfAUQn!Di#A{+kK;_J5bi0Za1C!J;!lsCDUNnF4m_{pAsoW`iu5~3@RErW33O@
z0NI?@1iBwN($wBgKQuIi?oi|tTk&}KM^akFUS37okx!qbHc-mYJcv94*wKvxKph)|
z_y^ZE1pr$o35z*5rI&we#v#_atvp$}sImP-G5ufQgFhhBWta$$XuRNaf$SElBF4`&YG&H2W>-M}E?Y_JAgeOqdH9#(0
z=xz9jRBd>01=De!6>0#v5X#E0;I+&Z3(FD(ion{M%2~`CH*SQaIE)+N-c!V}ikb>)
z!0g!7aGsMXGs#gcLk?gxK`YCfjt*2u5=HJqeK6s*DMFjfKdnA_K=xa|Po!=H>b)~)
za@!TjQ2qTb8KIJH?(XsE-b5>}KDD&r=*Nf2XV>m&Zf`g85)1alLXJdj7(_9j`TkuG
z!q2T+w_4q9dH_H`mG$}Yc^A>}tx9d1nxg2{j=YotYq|Hbvs|<7!3XJkn%p8?kC(&k
zXQ5ScEp4;?aKIJq6U;dV73Pfa79QN?uCA`e-aGba)b=jTG$zKs>#T$}99pgCQdoKZ
zMTVP0M&-`~JnN--4u5)_4&~Co$4B+shX+Z}7$n8SD3D8HP{j;-)8@VQWjwTh%6V~p
zQ2LiBrZf-Tg-4P?gKkHTBtW^ncI4G{&Ie|7)zyt{ZM1L*8d2C)J>Mu2n|-mU3|adsfC$s`6p3U08<;G
z^0tmhom#nm{d)dM-j(?!Xh`{gMho-vsUccI=Fk&kcbFY+)AakqAhY|D^(UJLqBT$`
zK})KGW+eznr(Z1+PbaZr4D|srff1@s`o)V$a{I13py2W8gw+pRKf=X!6mkl;;JEDa
z{@un4$%T1&jyZofNXZ;+-UJI9Qa`>Hbt
z;MLKw&!;Z1xqmchev%*N%{~`8`przsJ
z2#yNftPKs<2g)JHXVQrV`ES4F<>d)n=8NTc5OCnYf%y1%*>zu&)fwyT7~1D4v~6=h
znQbYa{Z&e!bu&OSTwC0V>-l4x!^1)!@W*a*aBxT%6Vmk2o$Mou0KQ@_Pl*s8so<>)
z#VOIRUh!OU3dJ3FTI%QkZ=$24GyIUX=kjLD&r2cu_wxWuwpJZDcu)`a%H+X}u{em``pNi+K$J5!TT9
zDzCSvQE0A#&cTP;25-otw=@|Hk(vS@=L`zA`$A&-ig68Jtgg>L9ND`9uYdY@o<;w=nR%jKmsLzh5t7TPH%6>1h
z)tnt%T%ytRn!CE#)6>)Y8p3tpvl;w757|%+hStt2TKxO2@09y>Jk0=C*3rWw5#UR5
z-8xSAplf$uia?X@yB~7!;6b9ng#GzxH8<7Iiya>q8A;{j?Cc9_O;c%gNTU*S04e|3R})CG9Tv_D4Pt%4F%R#DLmp)={3e1-pnIh=q&a60SzM}_h2
z&z?Q&stY_uu7hrq=nh9mM^Ys@*R08d8VnVFd2#Hj)3F!0&xDtzK(v~^FCUv)Or-P_x
zHXOGD^eFH8_1LNY`y;3XAe40V_4TSJ=e^3W`RXMSty
zK@Yj*Lq*LM`U1|wNKf%T1~#_X^mL9#siLV^F;)u~bepERL7s6Z%H5s|fSWjDH&{cb
zC?0!VypNvT7W7}Iok)3~eF_jICpU-qj&L
z<$wSDN)NLpbZMRyOUn>y22o}C`1mYjcR&rIqM(1@ocSRhEh`i22~@>|@1onzs|ZZnw(
z*Li`0n2$n9_7sqUQr0Bwz#;5z>$(6XIMd8KcI=SXHk|j5CqjIRe5h|yHt7{BY
zLEcT@&>-*+J<6XMRtOpe4Qw$7aW$9^5p;RHX11O&moEz=QIL+SHMt48m{S`)1zgof!wA@i&(bDx9KH*=Uhiv9w4e$M4G*8j
zLLEVE{>}<>#UPrAosmdmuy(p$zRXy-gv5oIoLq71YC{-#94)FOM+3-hYiCyui>K)3
z&AY-kO@F{g2c&?DK&KAW-RBI|C;~;%0hQMMZG|*aGECqBkUI{%IKn6%&0xlRYn@iu
zG9P$kE3A1b?Knq1qG=Js2~xG`)~ZVFXecA9YHH~V52M-Jel9N_XW>(&pkk4zLccGU
zp0Ax#_#9#*1o6#JzIE!ywM+rL7TdUy7f_|*j4^B@K8uKR!)UDNSabse10-vaKyne1
zB3eNa5Gr(0rDGWx#S@ivbVN-}O$jXp>!3wGrb^OM5Uq_8fu)q>eGJIw`}glN_(6|C
z8Nar%A7JP00W1I;X?h418
zBi{>EQXg7>1XG%m%i+V$%L}t0qb+DkFbI{pgJI0IVZEuH_+mli^mV&U7|i@VJUpP4
z(7={_n}R5lxZ|uD>Y}^5dzWaXrBI71G+VA+QU>;$jYYA40r#A7+mJhIg<$Idf3!Oo
zi=Y)^Z_iutq{t389G |