Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added remove_texts on base, elasticsearch, pgvector vectorsearch #525

Merged
merged 8 commits into from
Mar 25, 2024
Merged

Added remove_texts on base, elasticsearch, pgvector vectorsearch #525

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7479a67
Added remove_texts on base vectorsearch
gregogalante Mar 16, 2024
42e26c3
Added remove_texts on elasticsearch vectorsearch
gregogalante Mar 16, 2024
a43ec75
Added remove_texts on pgvector vectorsearch
gregogalante Mar 16, 2024
8536cc5
Update tests
gregogalante Mar 16, 2024
0ac3ded
Merge branch 'main' into main
andreibondarev Mar 17, 2024
8591cf2
Merge branch 'andreibondarev:main' into main
gregogalante Mar 22, 2024
0abad4f
Merge branch 'main' into main
andreibondarev Mar 24, 2024
eac3a68
Update pgvector_spec.rb
andreibondarev Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions lib/langchain/vectorsearch/pgvector.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,13 @@ def update_texts(texts:, ids:)
upsert_texts(texts: texts, ids: ids)
end

# Remove a list of texts from the index
# @param ids [Array<Integer>] The ids of the texts to remove from the index
# @return [Integer] The number of texts removed from the index
def remove_texts(ids:)
@db[table_name.to_sym].where(id: ids).delete
end

# Create default schema
def create_default_schema
db.run "CREATE EXTENSION IF NOT EXISTS vector"
Expand Down
30 changes: 30 additions & 0 deletions spec/langchain/vectorsearch/pgvector_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,36 @@
end
end

describe "#remove_texts" do
before do
allow_any_instance_of(
OpenAI::Client
).to receive(:embeddings)
.with(
parameters: {
dimensions: 1536,
model: "text-embedding-3-small",
input: "Hello World"
}
)
.and_return({
"object" => "list",
"data" => [
{"embedding" => 1536.times.map { rand }}
]
})
end

it "removes texts" do
values = subject.add_texts(texts: ["Hello World", "Hello World"])
ids = values.flatten
expect(ids.length).to eq(2)

result = subject.remove_texts(ids: ids)
expect(result.size).to eq(2)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

  1) Langchain::Vectorsearch::Pgvector#remove_texts removes texts
     Failure/Error: expect(result.size).to eq(2)

       expected: 2
            got: 8

       (compared using ==)
     # ./spec/langchain/vectorsearch/pgvector_spec.rb:127:in `block (3 levels) in <top (required)>'
     # ./spec/support/vcr.rb:22:in `block (3 levels) in <top (required)>'
     # ./vendor/bundle/ruby/3.2.0/gems/vcr-6.2.0/lib/vcr.rb:274:in `turned_off'
     # ./spec/support/vcr.rb:22:in `block (2 levels) in <top (required)>'

Finished in 4.16 seconds (files took 2.4 seconds to load)
426 examples, 1 failure, 3 pending

Failed examples:

rspec ./spec/langchain/vectorsearch/pgvector_spec.rb:121 # Langchain::Vectorsearch::Pgvector#remove_texts removes texts

Perhaps the result object is stale somehow? Can we try using:

subject.db.count

to see if it returns accurate count?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreibondarev I don't have a local pg vector database to test it. Can you try to run only remove_texts test to check length?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gregogalante Turns out that result is the actual number of elements deleted:

        result = subject.remove_texts(ids: ids)
        expect(result.size).to eq(2)

But when you call result.size it's the same as calling 2.size which is the number of bytes that Integer represent.

end
end

describe "#similarity_search" do
before do
allow_any_instance_of(
Expand Down
Loading