v0.21.0-beta.3

What's Changed

New Features 🎉

• feat(java): support drop columns for dataset by @yanghua in #3237
• feat(java): expose uri method for Dataset instance by @yanghua in #3231

Bug Fixes 🐛

• fix: remove overzealous warning by @westonpace in #3239
• fix: correctly copy null buffer when making deep copy by @westonpace in #3238

Full Changelog: v0.21.0-beta.2...v0.21.0-beta.3

v0.21.0-beta.2

What's Changed

Breaking Changes 🛠

• feat!: support hamming distance & binary vector by @BubbleCal in #3198

New Features 🎉

• feat: support blob api in pytorch loader by @eddyxu in #3217
• feat(python): add experimental parameter enable_move_stable_row_ids for pylance by @SaintBacchus in #3216
• feat: add the repetition index to the miniblock write path by @westonpace in #3208
• feat: packed struct encoding by @broccoliSpicy in #3186
• feat: support between sql clauses by @connellPortrait in #3225

Bug Fixes 🐛

• fix: test failure in test_fsl_packed_struct by @broccoliSpicy in #3227

Documentation 📚

• docs: add doc and test for 4bit PQ by @BubbleCal in #3212

New Contributors

• @connellPortrait made their first contribution in #3225

Full Changelog: v0.21.0-beta.1...v0.21.0-beta.2

v0.21.0-beta.1

What's Changed

Breaking Changes 🛠

• fix!: correctly handle nulls in btree and bitmap indices by @westonpace in #3211

New Features 🎉

• feat: enhance repdef utilities to handle empty / null lists by @westonpace in #3200
• feat: support _rowid meta column for spark connector in java by @SaintBacchus in #3194

Documentation 📚

• docs: add the documentation about how to install packages for tests by @yanghua in #3213

Performance Improvements 🚀

• perf: in-register lookup table & SIMD for 4bit PQ by @BubbleCal in #3178

Full Changelog: v0.20.0...v0.21.0-beta.1

v0.20.0

What's Changed

Breaking Changes 🛠

• feat!: allow passing down existing dataset for write by @wjones127 in #3119
• fix!: low recall with cosine/dot on v3 index types by @BubbleCal in #3141

New Features 🎉

• feat: start recording index details in the mainifest, cache index type lookup by @westonpace in #3131
• feat: make dataset version serializable by @albertlockett in #3143
• feat: support 4bit PQ on new IVF_PQ by @BubbleCal in #3144
• feat: add commit_batch API by @wjones127 in #3142
• feat: allow async stream for writing and appending to a dataset by @HoKim98 in #3146
• feat: add dictionary encoding by @broccoliSpicy in #3134
• feat(rust): make JSON serialization of DataType and Field public by @wjones127 in #3161
• feat: expose the table provider by @westonpace in #3162
• feat: support write multi fragments or empty fragment in one spark task by @SaintBacchus in #3183
• feat: add drop to dataset by @chenkovsky in #3184
• feat: upgrade arrow (to 53) & datafusion (to 42) by @westonpace in #3201

Bug Fixes 🐛

• fix: fix error about schema is not writable pd to pa by @Jay-ju in #3109
• fix: handle filter on empty partition by @eddyxu in #3151
• fix: fix dynamodb drop table by @LuQQiu in #3152
• fix: full text search index broken after optimize_indices() by @BubbleCal in #3145
• fix: fix performance regression introduced during reader refactor by @westonpace in #3170
• fix: panic if all docs are deleted in a posting list by @BubbleCal in #3163
• fix: full text search may produce dup results when search over multiple columns by @BubbleCal in #3189
• fix: fix typing for _write_fragment by @chenkovsky in #3171
• fix: fix storage options for dataset builder by @chenkovsky in #3156
• fix: fix storage options for ray by @chenkovsky in #3164

Performance Improvements 🚀

• perf: optimize reading transactions in commit loop by @wjones127 in #3117
• perf: improve PQ computing distances by @BubbleCal in #3150
• perf: improve constructing dist table by @BubbleCal in #3155
• perf: improve dot distance computing by @BubbleCal in #3169

Other Changes

• refactor: remove the queue in LanceArrowWriter to reduce memory usage for spark sink by @SaintBacchus in #3110

New Contributors

• @Jay-ju made their first contribution in #3109
• @chenkovsky made their first contribution in #3171
• @imotai made their first contribution in #3078
• @yanghua made their first contribution in #3193

Full Changelog: v0.19.2...v0.20.0

v0.20.0-beta.3

What's Changed

New Features 🎉

• feat: add commit_batch API by @wjones127 in #3142
• feat: allow async stream for writing and appending to a dataset by @HoKim98 in #3146
• feat: add dictionary encoding by @broccoliSpicy in #3134
• feat(rust): make JSON serialization of DataType and Field public by @wjones127 in #3161
• feat: expose the table provider by @westonpace in #3162

Bug Fixes 🐛

• fix: fix dynamodb drop table by @LuQQiu in #3152
• fix: full text search index broken after optimize_indices() by @BubbleCal in #3145
• fix: fix performance regression introduced during reader refactor by @westonpace in #3170
• fix: panic if all docs are deleted in a posting list by @BubbleCal in #3163

Performance Improvements 🚀

• perf: improve PQ computing distances by @BubbleCal in #3150
• perf: improve constructing dist table by @BubbleCal in #3155
• perf: improve dot distance computing by @BubbleCal in #3169

Full Changelog: v0.20.0-beta.2...v0.20.0-beta.3

v0.20.0-beta.2

What's Changed

New Features 🎉

• feat: support 4bit PQ on new IVF_PQ by @BubbleCal in #3144

Bug Fixes 🐛

• fix: handle filter on empty partition by @eddyxu in #3151

Performance Improvements 🚀

• perf: optimize reading transactions in commit loop by @wjones127 in #3117

Full Changelog: v0.20.0-beta.1...v0.20.0-beta.2

v0.20.0-beta.1

What's Changed

Breaking Changes 🛠

• feat!: allow passing down existing dataset for write by @wjones127 in #3119
• fix!: low recall with cosine/dot on v3 index types by @BubbleCal in #3141

New Features 🎉

• feat: make dataset version serializable by @albertlockett in #3143

Full Changelog: v0.19.3-beta.1...v0.20.0-beta.1

v0.19.3-beta.1

What's Changed

New Features 🎉

• feat: start recording index details in the mainifest, cache index type lookup by @westonpace in #3131

Bug Fixes 🐛

• fix: fix error about schema is not writable pd to pa by @Jay-ju in #3109

Other Changes

• refactor: remove the queue in LanceArrowWriter to reduce memory usage for spark sink by @SaintBacchus in #3110

New Contributors

• @Jay-ju made their first contribution in #3109

Full Changelog: v0.19.2...v0.19.3-beta.1

v0.19.2

What's Changed

New Features 🎉

• feat: support FTS on multiple fields by @BubbleCal in #3025
• feat: adds support for detached commits by @westonpace in #3028
• feat: add 2.1 read path by @westonpace in #2968
• feat: expose compression-level configuration for general compression by @niyue in #3034
• feat: make it possible to use rowid and rowaddr in filters by @westonpace in #2973
• feat: more flexible and sophisticated handling of non-null constraints by @westonpace in #2467
• feat: support build lance java with rust release mode by @SaintBacchus in #3051
• feat: do brute force search on unindexed data by @BubbleCal in #3036
• feat: bitpack with miniblock by @broccoliSpicy in #3067
• feat: introduce concept of "storage class" wtih separate dataset for "blob" storage data by @westonpace in #3064
• feat: add max length statistics to VariableWidth Datablock by @broccoliSpicy in #3082
• feat: add support for "take" operation to balanced storage by @westonpace in #3079
• feat: enable write variable width data in 2.1 by @broccoliSpicy in #3090
• feat: add compaction support to balanced datasets by @westonpace in #3088
• feat: align buffers in 2.1 files by @westonpace in #3101
• feat: allow inserting subschemas by @wjones127 in #3041
• feat: support miniblock with binary data by @broccoliSpicy in #3099
• feat: add full zip encoding for wide data types by @westonpace in #3114
• feat: fsst compression with mini-block by @broccoliSpicy in #3121

Bug Fixes 🐛

• fix: fix the incorrect assertion for fixed size binary decoder by @niyue in #3050
• fix: verify num_sub_vectors is valid before creating index by @westonpace in #3056
• fix: always return correct batch size by @westonpace in #3066
• fix: fp16 GPU training by @chebbyChefNEQ in #3105

Performance Improvements 🚀

• perf: use total cmp for ordered float by @chebbyChefNEQ in #3059
• perf: parallel search partitions in subindex by @chebbyChefNEQ in #3060
• perf: implement fast path for union many row id tree map by @chebbyChefNEQ in #3087
• perf: transpose the PQ codes to improve search performance by @BubbleCal in #3120

Other Changes

• refactor: fix confusing impl of indexed_fragments by @wjones127 in #3047

Full Changelog: v0.19.1...v0.19.2

v0.19.2-beta.5

What's Changed

New Features 🎉

• feat: align buffers in 2.1 files by @westonpace in #3101
• feat: allow inserting subschemas by @wjones127 in #3041

Bug Fixes 🐛

• fix: fp16 GPU training by @chebbyChefNEQ in #3105

Full Changelog: v0.19.2-beta.4...v0.19.2-beta.5