Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update readme #145

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@

Optimisers.jl defines many standard gradient-based optimisation rules, and tools for applying them to deeply nested models.

This is the future of training for [Flux.jl](https://github.com/FluxML/Flux.jl) neural networks,
and the present for [Lux.jl](https://github.com/avik-pal/Lux.jl).
This was written as a new training back-end for [Flux.jl](https://github.com/FluxML/Flux.jl) neural networks,
and is also used by [Lux.jl](https://github.com/avik-pal/Lux.jl).
But it can be used separately on any array, or anything else understood by [Functors.jl](https://github.com/FluxML/Functors.jl).

## Installation
Expand All @@ -38,17 +38,23 @@ It is initialised by `setup`, and then at each step, `update` returns both the n
state, and the model with its trainable parameters adjusted:

```julia
state = Optimisers.setup(Optimisers.Adam(), model) # just once
state_tree = Optimisers.setup(Optimisers.Adam(), model) # just once

grad = Zygote.gradient(m -> loss(m(x), y), model)[1]

state, model = Optimisers.update(state, model, grad) # at every step
state_tree, model = Optimisers.update(opt_state, model, grad) # at every step
```

For models with deeply nested layers containing the parameters (like [Flux.jl](https://github.com/FluxML/Flux.jl) models),
this state is a similarly nested tree. As is the gradient: if using Zygote, you must use the "explicit" style as shown,
this `state_tree` is a similarly nested object. As is the gradient: if using Zygote, you must use the "explicit" style as shown,
not the "implicit" one with `Params`.

You can change the learning rate during training by mutating all the states:

```julia
Optimisers.adjust!(state_tree, 0.01)
```

The function `destructure` collects all the trainable parameters into one vector,
and returns this along with a function to re-build a similar model:

Expand Down