diff --git a/.gitignore b/.gitignore index ac67aaf..85c9867 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ /_build /cover +/doc /deps erl_crash.dump *.ez diff --git a/README.md b/README.md index de50586..62e5f7a 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,137 @@ # Exstreme -**TODO: Add description** +Exstreme is an implementation of a Stream Push data structure in the way of a runnable graph where all the nodes must be connected and process a message and pass the result to next node(s) ## Installation -If [available in Hex](https://hex.pm/docs/publish), the package can be installed as: +The package can be installed as: 1. Add exstreme to your list of dependencies in `mix.exs`: def deps do - [{:exstreme, "~> 0.0.1"}] + [{:exstreme, "~> 0.0.2"}] end - 2. Ensure exstreme is started before your application: + 2. Check the documentation: [available in Hex](https://hexdocs.pm/exstreme/doc/Exstreme.html) - def application do - [applications: [:exstreme]] - end +## Usage + +A graph is a data structure that contains nodes connected between them, this graphs must start with only one node and can finish in many nodes, all the nodes in the graph must be connected, for example: + +``` + n3 + | +n1 - n2 - b1 + | + n4 +``` + +The information of a graph is: + +* `:name` - Name assigned for the graph, if you don't assign a name this will be generated +* `:nodes` - The nodes with their parameters as a keyword list +* `:connections` - The nodes and their connections + +The nodes can be of three types: + +* `Common` - it represents a node that is connected to can be connected to another node and can receive a message from another node, represented by the n letter. + +* `Broadcast` - it is a node that can broadcast a message to multiple nodes, represented by the b letter. + +* `Funnel` - it receives messages from a group of nodes and sends it to the next, represented by the f letter. + +A graph could look like this: + +``` + n3 + | | +n1 - n2 - b1 f1 - n5 + | | + n4 +``` + +It works this way: + +- **n1** passes the message to **n2** +- **n2** passes the message to **b1** +- **b1** broadcasts the message to **n3** and **n4** +- **f1** receives the message from **n3** and **n4** and packages them as one and sends that to **n5** +- **n5** process the message received from **f1** + +How to create a graph: + +```elixir + graph = GraphCreator.create_graph("name") + {graph, n1} = GraphCreator.create_node(graph, params) + {graph, n2} = GraphCreator.create_node(graph, params) + GraphCreator.add_connection(graph, n1, n2) +``` + +A complex one(this is the one for graph above): + +```elixir + graph = GraphCreator.create_graph("name") + {graph, n1} = GraphCreator.create_node(graph, params) + {graph, n2} = GraphCreator.create_node(graph, params) + {graph, b1} = GraphCreator.create_broadcast(graph, params_broadcast) + {graph, n3} = GraphCreator.create_node(graph, params) + {graph, n4} = GraphCreator.create_node(graph, params) + {graph, f1} = GraphCreator.create_funnel(graph, params_funnel) + {graph, n5} = GraphCreator.create_node(graph, params) + + graph + |> GraphCreator.add_connection(n1, n2) + |> GraphCreator.add_connection(n2, b1) + |> GraphCreator.add_connection(b1, n3) + |> GraphCreator.add_connection(b1, n4) + |> GraphCreator.add_connection(n3, f1) + |> GraphCreator.add_connection(n4, f1) + |> GraphCreator.add_connection(f1, n5) + ``` + +The nodes in the graph are named like this if the name of the graph is "demo": + +* `n1` - :n_demo_1 +* `n2` - :n_demo_2 +* `b1` - :b_demo_1 +* `f1` - :f_demo_1 + +The node params must have a function that is the one called every time a message arrives to the node. The function receives a tuple where the first parameter is the message and the second one the node data, it must return a tuple with :ok and the new message. + +```elixir + params = [func: fn({msg, node_data}) -> {:ok, new_msg} end] +``` + +We build a graph after we create it, like this: + +```elixir + graph_built = GraphBuilder.build(graph) +``` + +The name of the supervisor is the name of the graph so you can get the pid for the supervisor: + +```elixir + pid = + graph_built.name + |> String.to_atom + |> Process.whereis +``` + +Also we can get the pid for the nodes: + +```elixir + Enum.each(graph_built.nodes, fn({nid, params}) -> + pid = Process.whereis(nid) + end) +``` + +And we can connect a process to the graph and receive the output of the processing: + +```elixir + [start_node] = Graph.find_start_node(graph_built) + [last_node] = Graph.find_last_node(graph_built) + :ok = GenServer.call(last_node, {:connect, self}) + GenServer.cast(start_node, {:next, self, {:sum, 0}}) +``` +If I try to build another graph with the same I'll get an error because there can't be two process with the same name. diff --git a/TODO.md b/TODO.md index 8f585d8..05d2498 100644 --- a/TODO.md +++ b/TODO.md @@ -1,13 +1,2 @@ -x- Validate before build -x- Named nodes with graph name -x- Add documentation -x- Use nid to connect -X- Get nid for a node -X- A node must have a function -X- The function in Broadcast and Funnel nodes can be optional -X- Add Supervisors functionality -X- Use counters for stats -x- Improve test - -Future +- Allow to extend the behaviour - Improve API diff --git a/lib/exstreme.ex b/lib/exstreme.ex index 8b39fc4..d8b3d2a 100644 --- a/lib/exstreme.ex +++ b/lib/exstreme.ex @@ -1,2 +1,123 @@ defmodule Exstreme do + @moduledoc """ + A graph is a data structure that contains nodes connected between them, this graphs must start with only one node and can finish in many nodes, all the nodes in the graph must be connected, for example: + + ``` + n3 + | + n1 - n2 - b1 + | + n4 + ``` + + The information of a graph is: + + * `:name` - Name assigned for the graph, if you don't assign a name this will be generated + * `:nodes` - The nodes with their parameters as a keyword list + * `:connections` - The nodes and their connections + + The nodes can be of three types: + + * `Common` - it represents a node that is connected to can be connected to another node and can receive a message from another node, represented by the n letter. + + * `Broadcast` - it is a node that can broadcast a message to multiple nodes, represented by the b letter. + + * `Funnel` - it receives messages from a group of nodes and sends it to the next, represented by the f letter. + + A graph could look like this: + + ``` + n3 + | | + n1 - n2 - b1 f1 - n5 + | | + n4 + ``` + + It works this way: + + - **n1** passes the message to **n2** + - **n2** passes the message to **b1** + - **b1** broadcasts the message to **n3** and **n4** + - **f1** receives the message from **n3** and **n4** and packages them as one and sends that to **n5** + - **n5** process the message received from **f1** + + How to create a graph: + + ```elixir + graph = GraphCreator.create_graph("name") + {graph, n1} = GraphCreator.create_node(graph, params) + {graph, n2} = GraphCreator.create_node(graph, params) + GraphCreator.add_connection(graph, n1, n2) + ``` + + A complex one(this is the one for graph above): + + ```elixir + graph = GraphCreator.create_graph("name") + {graph, n1} = GraphCreator.create_node(graph, params) + {graph, n2} = GraphCreator.create_node(graph, params) + {graph, b1} = GraphCreator.create_broadcast(graph, params_broadcast) + {graph, n3} = GraphCreator.create_node(graph, params) + {graph, n4} = GraphCreator.create_node(graph, params) + {graph, f1} = GraphCreator.create_funnel(graph, params_funnel) + {graph, n5} = GraphCreator.create_node(graph, params) + + graph + |> GraphCreator.add_connection(n1, n2) + |> GraphCreator.add_connection(n2, b1) + |> GraphCreator.add_connection(b1, n3) + |> GraphCreator.add_connection(b1, n4) + |> GraphCreator.add_connection(n3, f1) + |> GraphCreator.add_connection(n4, f1) + |> GraphCreator.add_connection(f1, n5) + ``` + + The nodes in the graph are named like this if the name of the graph is "demo": + + * `n1` - :n_demo_1 + * `n2` - :n_demo_2 + * `b1` - :b_demo_1 + * `f1` - :f_demo_1 + + The node params must have a function that is the one called every time a message arrives to the node. The function receives a tuple where the first parameter is the message and the second one the node data, it must return a tuple with :ok and the new message. + + ```elixir + params = [func: fn({msg, node_data}) -> {:ok, new_msg} end] + ``` + + We build a graph after we create it, like this: + + ```elixir + graph_built = GraphBuilder.build(graph) + ``` + + The name of the supervisor is the name of the graph so you can get the pid for the supervisor: + + ```elixir + pid = + graph_built.name + |> String.to_atom + |> Process.whereis + ``` + + Also we can get the pid for the nodes: + + ```elixir + Enum.each(graph_built.nodes, fn({nid, params}) -> + pid = Process.whereis(nid) + end) + ``` + + And we can connect a process to the graph and receive the output of the processing: + + ```elixir + [start_node] = Graph.find_start_node(graph_built) + [last_node] = Graph.find_last_node(graph_built) + :ok = GenServer.call(last_node, {:connect, self}) + GenServer.cast(start_node, {:next, self, {:sum, 0}}) + ``` + + If I try to build another graph with the same I'll get an error because there can't be two process with the same name. + """ end diff --git a/lib/exstreme/graph.ex b/lib/exstreme/graph.ex index 0d9f70c..bf5dd50 100644 --- a/lib/exstreme/graph.ex +++ b/lib/exstreme/graph.ex @@ -1,6 +1,6 @@ defmodule Exstreme.Graph do @moduledoc """ - Provides information for a Graph + This module contains useful functions to get information about a graph. """ alias __MODULE__ diff --git a/lib/exstreme/graph_builder.ex b/lib/exstreme/graph_builder.ex index e206bce..721e745 100644 --- a/lib/exstreme/graph_builder.ex +++ b/lib/exstreme/graph_builder.ex @@ -1,6 +1,6 @@ defmodule Exstreme.GraphBuilder do @moduledoc """ - Builds the Graph into a Supervision tree of process + Builds the graph generating a Supervision tree of process for the graph that supervises each node. """ alias Exstreme.GNode.Broadcast alias Exstreme.GNode.Funnel diff --git a/lib/exstreme/graph_creator.ex b/lib/exstreme/graph_creator.ex index 3ae7683..08d62fd 100644 --- a/lib/exstreme/graph_creator.ex +++ b/lib/exstreme/graph_creator.ex @@ -1,6 +1,6 @@ defmodule Exstreme.GraphCreator do @moduledoc """ - Creates a Graph representation + Contains the functions to create graph, create nodes and to connect them. """ alias Exstreme.Graph @@ -36,7 +36,7 @@ defmodule Exstreme.GraphCreator do @spec create_node(Graph.t, [key: term]) :: {Graph.t, atom} def create_node(graph = %Graph{nodes: nodes}, params \\ []) do key = next_node_key(graph, nodes) - + params = Keyword.put_new(params, :type, :common) new_graph = update_in(graph.nodes, &(Map.put(&1, key, params))) {new_graph, key} end @@ -47,7 +47,7 @@ defmodule Exstreme.GraphCreator do @spec create_broadcast(Graph.t, [key: term]) :: {Graph.t, atom} def create_broadcast(graph = %Graph{nodes: nodes}, params \\ []) do key = next_broadcast_key(graph, nodes) - + params = Keyword.put_new(params, :type, :broadcast) new_graph = update_in(graph.nodes, &(Map.put(&1, key, params))) {new_graph, key} end @@ -58,7 +58,7 @@ defmodule Exstreme.GraphCreator do @spec create_funnel(Graph.t, [key: term]) :: {Graph.t, atom} def create_funnel(graph = %Graph{nodes: nodes}, params \\ []) do key = next_funnel_key(graph, nodes) - + params = Keyword.put_new(params, :type, :funnel) new_graph = update_in(graph.nodes, &(Map.put(&1, key, params))) {new_graph, key} end diff --git a/lib/exstreme/graph_validator.ex b/lib/exstreme/graph_validator.ex index 590b493..b01f8be 100644 --- a/lib/exstreme/graph_validator.ex +++ b/lib/exstreme/graph_validator.ex @@ -1,7 +1,7 @@ defmodule Exstreme.GraphValidator do alias Exstreme.Graph @moduledoc """ - Validates the Graph + Contains the functions to validate a graph """ @typedoc """ diff --git a/mix.exs b/mix.exs index e7935c7..01134cb 100644 --- a/mix.exs +++ b/mix.exs @@ -2,13 +2,17 @@ defmodule Exstreme.Mixfile do use Mix.Project def project do - [app: :exstreme, - version: "0.0.1", - elixir: "~> 1.3", - elixirc_paths: elixirc_paths(Mix.env), - build_embedded: Mix.env == :prod, - start_permanent: Mix.env == :prod, - deps: deps] + [ + app: :exstreme, + version: "0.0.2", + description: description, + package: package, + elixir: "~> 1.3", + elixirc_paths: elixirc_paths(Mix.env), + build_embedded: Mix.env == :prod, + start_permanent: Mix.env == :prod, + deps: deps + ] end # Configuration for the OTP application @@ -21,6 +25,12 @@ defmodule Exstreme.Mixfile do defp elixirc_paths(:test), do: ["lib", "test/support"] defp elixirc_paths(_), do: ["lib"] + defp description do + """ + Exstreme is an implementation of a Stream Push data structure in the way of a runnable graph where all the nodes must be connected and process a message and pass the result to next node(s) + """ + end + # Dependencies can be Hex packages: # # {:mydep, "~> 0.3.0"} @@ -32,8 +42,19 @@ defmodule Exstreme.Mixfile do # Type "mix help deps" for more examples and options defp deps do [ + {:ex_doc, "~> 0.12", only: :dev}, {:dialyxir, "~> 0.3", only: [:dev]}, {:credo, "~> 0.4", only: [:dev, :test]} ] end + + defp package do + [# These are the default files included in the package + name: :exstreme, + files: ["lib", "priv", "mix.exs", "README*", "readme*", "LICENSE*", "license*"], + maintainers: ["Michel Perez"], + licenses: ["Apache 2.0"], + links: %{"GitHub" => "https://github.com/mrkaspa/exstreme"} + ] + end end diff --git a/mix.lock b/mix.lock index 6eca57c..e4902a9 100644 --- a/mix.lock +++ b/mix.lock @@ -1,3 +1,5 @@ %{"bunt": {:hex, :bunt, "0.1.6", "5d95a6882f73f3b9969fdfd1953798046664e6f77ec4e486e6fafc7caad97c6f", [:mix], []}, "credo": {:hex, :credo, "0.4.5", "5c5daaf50a2a96068c0f21b6fbd382d206702efa8836a946eeab0b8ac25f5f22", [:mix], [{:bunt, "~> 0.1.6", [hex: :bunt, optional: false]}]}, - "dialyxir": {:hex, :dialyxir, "0.3.3", "2f8bb8ab4e17acf4086cae847bd385c0f89296d3e3448dc304c26bfbe4b46cb4", [:mix], []}} + "dialyxir": {:hex, :dialyxir, "0.3.3", "2f8bb8ab4e17acf4086cae847bd385c0f89296d3e3448dc304c26bfbe4b46cb4", [:mix], []}, + "earmark": {:hex, :earmark, "1.0.1", "2c2cd903bfdc3de3f189bd9a8d4569a075b88a8981ded9a0d95672f6e2b63141", [:mix], []}, + "ex_doc": {:hex, :ex_doc, "0.13.0", "aa2f8fe4c6136a2f7cfc0a7e06805f82530e91df00e2bff4b4362002b43ada65", [:mix], [{:earmark, "~> 1.0", [hex: :earmark, optional: false]}]}}