Yard doc improvements (#152)

* Add yardopts to include README, license, and other markdown

* include examples too

* copy over tons of the wiki into the specific classes. move class comments to the appropriate place. try to document base classes when possible

* standardrb

* move these two to the base class

* Update replicate.rb

* Update few_shot_prompt_template.rb

* Update few_shot_prompt_template.rb

* Update prompt_template.rb

* fix ai21 formating

* pull in vectorsearch

* Add agents

* Add tools

* add Langchain

* standardrb

* go back to ...

* run yardoc directly?

* only fix I've found when there is removing .yardopts, but that means no LICENSE or extra files

---------

Co-authored-by: Andrei Bondarev <[email protected]>

.github/workflows/ci.yml

@@ -61,4 +61,4 @@ jobs:
           bundler: default
           bundler-cache: true
       - name: Build docs
-        run: bundle exec rake yard
+        run: bundle exec yardoc

Rakefile

@@ -14,5 +14,4 @@ Rake::Task["spec"].enhance do
 end
 
 YARD::Rake::YardocTask.new do |t|
-  t.options = ["--fail-on-warning"]
 end

lib/langchain.rb

@@ -6,10 +6,53 @@
 
 require_relative "./langchain/version"
 
+# Langchain.rb a is library for building LLM-backed Ruby applications. It is an abstraction layer that sits on top of the emerging AI-related tools that makes it easy for developers to consume and string those services together.
+#
+# = Installation
+# Install the gem and add to the application's Gemfile by executing:
+#
+#     $ bundle add langchainrb
+#
+# If bundler is not being used to manage dependencies, install the gem by executing:
+#
+#     $ gem install langchainrb
+#
+# Require the gem to start using it:
+#
+#     require "langchain"
+#
+# = Concepts
+#
+# == Processors
+# Processors load and parse/process various data types such as CSVs, PDFs, Word documents, HTML pages, and others.
+#
+# == Chunkers
+# Chunkers split data based on various available options such as delimeters, chunk sizes or custom-defined functions. Chunkers are used when data needs to be split up before being imported in vector databases.
+#
+# == Prompts
+# Prompts are structured inputs to the LLMs. Prompts provide instructions, context and other user input that LLMs use to generate responses.
+#
+# == Large Language Models (LLMs)
+# LLM is a language model consisting of a neural network with many parameters (typically billions of weights or more), trained on large quantities of unlabeled text using self-supervised learning or semi-supervised learning.
+#
+# == Vectorsearch Databases
+# Vector database is a type of database that stores data as high-dimensional vectors, which are mathematical representations of features or attributes. Each vector has a certain number of dimensions, which can range from tens to thousands, depending on the complexity and granularity of the data.
+#
+# == Embedding
+# Word embedding or word vector is an approach with which we represent documents and words. It is defined as a numeric vector input that allows words with similar meanings to have the same representation. It can approximate meaning and represent a word in a lower dimensional space.
+#
+#
+# = Logging
+#
+# LangChain.rb uses standard logging mechanisms and defaults to :debug level. Most messages are at info level, but we will add debug or warn statements as needed. To show all log messages:
+#
+#     Langchain.logger.level = :info
 module Langchain
   class << self
+    # @return [Logger]
     attr_accessor :logger
 
+    # @return [Pathname]
     attr_reader :root
   end
 

lib/langchain/agent/base.rb

@@ -1,6 +1,14 @@
 # frozen_string_literal: true
 
 module Langchain::Agent
+  # = Agents
+  #
+  # Agents are semi-autonomous bots that can respond to user questions and use available to them Tools to provide informed replies. They break down problems into series of steps and define Actions (and Action Inputs) along the way that are executed and fed back to them as additional information. Once an Agent decides that it has the Final Answer it responds with it.
+  #
+  # Available:
+  # - {Langchain::Agent::ChainOfThoughtAgent}
+  #
+  # @abstract
   class Base
   end
 end

lib/langchain/agent/chain_of_thought_agent/chain_of_thought_agent.rb

@@ -1,6 +1,20 @@
 # frozen_string_literal: true
 
 module Langchain::Agent
+  # = Chain of Thought Agent
+  #
+  #     llm = Langchain::LLM::OpenAI.new(api_key: ENV["OPENAI_API_KEY"]) # or your choice of Langchain::LLM::Base implementation
+  #
+  #     agent = Langchain::Agent::ChainOfThoughtAgent.new(
+  #       llm: llm,
+  #       tools: ["search", "calculator", "wikipedia"]
+  #     )
+  #
+  #     agent.tools
+  #     # => ["search", "calculator", "wikipedia"]
+  #
+  #     agent.run(question: "How many full soccer fields would be needed to cover the distance between NYC and DC in a straight line?")
+  #     #=> "Approximately 2,945 soccer fields would be needed to cover the distance between NYC and DC in a straight line."
   class ChainOfThoughtAgent < Base
     attr_reader :llm, :tools
 

lib/langchain/llm/ai21.rb

@@ -1,16 +1,16 @@
 # frozen_string_literal: true
 
 module Langchain::LLM
+  #
+  # Wrapper around AI21 Studio APIs.
+  #
+  # Gem requirements:
+  #   gem "ai21", "~> 0.2.0"
+  #
+  # Usage:
+  #     ai21 = Langchain::LLM::AI21.new(api_key:)
+  #
   class AI21 < Base
-    #
-    # Wrapper around AI21 Studio APIs.
-    #
-    # Gem requirements: gem "ai21", "~> 0.2.0"
-    #
-    # Usage:
-    # ai21 = Langchain::LLM::AI21.new(api_key:)
-    #
-
     def initialize(api_key:)
       depends_on "ai21"
       require "ai21"

lib/langchain/llm/base.rb

@@ -1,31 +1,58 @@
 # frozen_string_literal: true
 
 module Langchain::LLM
+  # A LLM is a language model consisting of a neural network with many parameters (typically billions of weights or more), trained on large quantities of unlabeled text using self-supervised learning or semi-supervised learning.
+  #
+  # Langchain.rb provides a common interface to interact with all supported LLMs:
+  #
+  # - {Langchain::LLM::AI21}
+  # - {Langchain::LLM::Cohere}
+  # - {Langchain::LLM::GooglePalm}
+  # - {Langchain::LLM::HuggingFace}
+  # - {Langchain::LLM::OpenAI}
+  # - {Langchain::LLM::Replicate}
+  #
+  # @abstract
   class Base
     include Langchain::DependencyHelper
 
+    # A client for communicating with the LLM
     attr_reader :client
 
     def default_dimension
       self.class.const_get(:DEFAULTS).dig(:dimension)
     end
 
-    # Method supported by an LLM that generates a response for a given chat-style prompt
+    #
+    # Generate a chat completion for a given prompt. Parameters will depend on the LLM
+    #
+    # @raise NotImplementedError if not supported by the LLM
     def chat(...)
       raise NotImplementedError, "#{self.class.name} does not support chat"
     end
 
-    # Method supported by an LLM that completes a given prompt
+    #
+    # Generate a completion for a given prompt. Parameters will depend on the LLM.
+    #
+    # @raise NotImplementedError if not supported by the LLM
     def complete(...)
       raise NotImplementedError, "#{self.class.name} does not support completion"
     end
 
-    # Method supported by an LLM that generates an embedding for a given text or array of texts
+    #
+    # Generate an embedding for a given text. Parameters depends on the LLM.
+    #
+    # @raise NotImplementedError if not supported by the LLM
+    #
     def embed(...)
       raise NotImplementedError, "#{self.class.name} does not support generating embeddings"
     end
 
-    # Method supported by an LLM that summarizes a given text
+    #
+    # Generate a summary for a given text. Parameters depends on the LLM.
+    #
+    # @raise NotImplementedError if not supported by the LLM
+    #
     def summarize(...)
       raise NotImplementedError, "#{self.class.name} does not support summarization"
     end

lib/langchain/llm/cohere.rb

@@ -1,16 +1,16 @@
 # frozen_string_literal: true
 
 module Langchain::LLM
+  #
+  # Wrapper around the Cohere API.
+  #
+  # Gem requirements:
+  #     gem "cohere-ruby", "~> 0.9.4"
+  #
+  # Usage:
+  #     cohere = Langchain::LLM::Cohere.new(api_key: "YOUR_API_KEY")
+  #
   class Cohere < Base
-    #
-    # Wrapper around the Cohere API.
-    #
-    # Gem requirements: gem "cohere-ruby", "~> 0.9.4"
-    #
-    # Usage:
-    # cohere = Langchain::LLM::Cohere.new(api_key: "YOUR_API_KEY")
-    #
-
     DEFAULTS = {
       temperature: 0.0,
       completion_model_name: "base",
@@ -43,6 +43,7 @@ def embed(text:)
     # Generate a completion for a given prompt
     #
     # @param prompt [String] The prompt to generate a completion for
+    # @param params[:stop_sequences]
     # @return [Hash] The completion
     #
     def complete(prompt:, **params)

lib/langchain/llm/google_palm.rb

@@ -1,6 +1,15 @@
 # frozen_string_literal: true
 
 module Langchain::LLM
+  #
+  # Wrapper around the Google PaLM (Pathways Language Model) APIs: https://ai.google/build/machine-learning/
+  #
+  # Gem requirements:
+  #     gem "google_palm_api", "~> 0.1.0"
+  #
+  # Usage:
+  #     google_palm = Langchain::LLM::GooglePalm.new(api_key: "YOUR_API_KEY")
+  #
   class GooglePalm < Base
     #
     # Wrapper around the Google PaLM (Pathways Language Model) APIs.
@@ -40,6 +49,7 @@ def embed(text:)
     # Generate a completion for a given prompt
     #
     # @param prompt [String] The prompt to generate a completion for
+    # @param params extra parameters passed to GooglePalmAPI::Client#generate_text
     # @return [String] The completion
     #
     def complete(prompt:, **params)
@@ -67,6 +77,7 @@ def complete(prompt:, **params)
     #
     # @param prompt [String] The prompt to generate a chat completion for
     # @param messages [Array] The messages that have been sent in the conversation
+    # @param params extra parameters passed to GooglePalmAPI::Client#generate_chat_message
     # @return [String] The chat completion
     #
     def chat(prompt: "", messages: [], **params)

lib/langchain/llm/hugging_face.rb

@@ -1,16 +1,16 @@
 # frozen_string_literal: true
 
 module Langchain::LLM
+  #
+  # Wrapper around the HuggingFace Inference API: https://huggingface.co/inference-api
+  #
+  # Gem requirements:
+  #     gem "hugging-face", "~> 0.3.4"
+  #
+  # Usage:
+  #     hf = Langchain::LLM::HuggingFace.new(api_key: "YOUR_API_KEY")
+  #
   class HuggingFace < Base
-    #
-    # Wrapper around the HuggingFace Inference API.
-    #
-    # Gem requirements: gem "hugging-face", "~> 0.3.4"
-    #
-    # Usage:
-    # hf = Langchain::LLM::HuggingFace.new(api_key: "YOUR_API_KEY")
-    #
-
     # The gem does not currently accept other models:
     # https://github.com/alchaplinsky/hugging-face/blob/main/lib/hugging_face/inference_api.rb#L32-L34
     DEFAULTS = {

lib/langchain/llm/openai.rb

@@ -1,16 +1,15 @@
 # frozen_string_literal: true
 
 module Langchain::LLM
+  # LLM interface for OpenAI APIs: https://platform.openai.com/overview
+  #
+  # Gem requirements:
+  #    gem "ruby-openai", "~> 4.0.0"
+  #
+  # Usage:
+  #    openai = Langchain::LLM::OpenAI.new(api_key:, llm_options: {})
+  #
   class OpenAI < Base
-    #
-    # Wrapper around OpenAI APIs.
-    #
-    # Gem requirements: gem "ruby-openai", "~> 4.0.0"
-    #
-    # Usage:
-    # openai = Langchain::LLM::OpenAI.new(api_key:, llm_options: {})
-    #
-
     DEFAULTS = {
       temperature: 0.0,
       completion_model_name: "text-davinci-003",
@@ -30,6 +29,7 @@ def initialize(api_key:, llm_options: {})
     # Generate an embedding for a given text
     #
     # @param text [String] The text to generate an embedding for
+    # @param params extra parameters passed to OpenAI::Client#embeddings
     # @return [Array] The embedding
     #
     def embed(text:, **params)
@@ -45,6 +45,7 @@ def embed(text:, **params)
     # Generate a completion for a given prompt
     #
     # @param prompt [String] The prompt to generate a completion for
+    # @param params  extra parameters passed to OpenAI::Client#complete
     # @return [String] The completion
     #
     def complete(prompt:, **params)
@@ -62,6 +63,7 @@ def complete(prompt:, **params)
     #
     # @param prompt [String] The prompt to generate a chat completion for
     # @param messages [Array] The messages that have been sent in the conversation
+    # @param params extra parameters passed to OpenAI::Client#chat
     # @return [String] The chat completion
     #
     def chat(prompt: "", messages: [], **params)

lib/langchain/llm/replicate.rb

@@ -1,22 +1,23 @@
 # frozen_string_literal: true
 
 module Langchain::LLM
+  #
+  # Wrapper around Replicate.com LLM provider
+  #
+  # Gem requirements:
+  #     gem "replicate-ruby", "~> 0.2.2"
+  #
+  # Use it directly:
+  #     replicate = Langchain::LLM::Replicate.new(api_key: ENV["REPLICATE_API_KEY"])
+  #
+  # Or pass it to be used by a vector search DB:
+  #     chroma = Langchain::Vectorsearch::Chroma.new(
+  #       url: ENV["CHROMA_URL"],
+  #       index_name: "...",
+  #       llm: replicate
+  #     )
+  #
   class Replicate < Base
-    #
-    # Wrapper around Replicate.com LLM provider
-    #
-    # Gem requirements: gem "replicate-ruby", "~> 0.2.2"
-    #
-    # Use it directly:
-    # replicate = LLM::Replicate.new(api_key: ENV["REPLICATE_API_KEY"])
-    #
-    # Or pass it to be instantiated by a vector search DB:
-    # chroma = Vectorsearch::Chroma.new(
-    #   url: ENV["CHROMA_URL"],
-    #   index_name: "...",
-    #   llm: Langchain::LLM::Replicate(api_key: ENV["REPLICATE_API_KEY"])
-    # )
-
     DEFAULTS = {
       # TODO: Figure out how to send the temperature to the API
       temperature: 0.01, # Minimum accepted value

lib/langchain/processors/base.rb

@@ -2,6 +2,7 @@
 
 module Langchain
   module Processors
+    # Processors load and parse/process various data types such as CSVs, PDFs, Word documents, HTML pages, and others.
     class Base
       include Langchain::DependencyHelper
 

lib/langchain/prompt/base.rb

@@ -5,15 +5,20 @@
 require "yaml"
 
 module Langchain::Prompt
+  # Prompts are structured inputs to the LLMs. Prompts provide instructions, context and other user input that LLMs use to generate responses.
+  #
+  # @abstract
   class Base
     def format(**kwargs)
       raise NotImplementedError
     end
 
+    # @return [String] the type of the prompt
     def prompt_type
       raise NotImplementedError
     end
 
+    # @return [Hash] a hash representation of the prompt
     def to_h
       raise NotImplementedError
     end