Code

Rewriting the queue#78

Code

Merged

equals215 merged 107 commits intomaininternetarchive/Zeno:mainfrom queueinternetarchive/Zeno:queueCopy head branch name to clipboardAug 4, 2024

+5,400-989

Commits

Commits on Jul 12, 2024

• Initial commit for frontier rewrite

  CorentinB

  committed
  fd106db

  Copy full SHA for fd106db

• Replace mmap files with "regular" files

  CorentinB

  committed
  96a4e38

  Copy full SHA for 96a4e38

• Remove useless stat

  CorentinB

  committed
  1090690

  Copy full SHA for 1090690

• Remove dequeue recursion

  CorentinB

  committed
  1c41124

  Copy full SHA for 1c41124

• Fix: dequeue timeout mechanism

  CorentinB

  committed
  a0045d3

  Copy full SHA for a0045d3

• Big brain modifications of queue & enqueue with signal

  CorentinB

  committed
  487319d

  Copy full SHA for 487319d

• Add: enqueue tests

  CorentinB

  committed
  3a70670

  Copy full SHA for 3a70670

• Increment Prometheus counter only when --api is specified

  CorentinB

  committed
  a2fb09d

  Copy full SHA for a2fb09d

• WIP: Dequeue test

  CorentinB

  committed
  52d751a

  Copy full SHA for 52d751a

• Remove timeout for Dequeue

  CorentinB

  committed
  5bc73f1

  Copy full SHA for 5bc73f1

• Remove any mention of 'frontier'

  CorentinB

  committed
  307cc9c

  Copy full SHA for 307cc9c

• Move stats updates to separate functions

  CorentinB

  committed
  2abaf8e

  Copy full SHA for 2abaf8e

• Remove a test that doesn't make sense in a root env (like in a Docker container)

  CorentinB

  committed
  ea748ea

  Copy full SHA for ea748ea

• Add: large scale test (doesn't pass yet)

  CorentinB

  committed
  2da5d78

  Copy full SHA for 2da5d78

• Restore previous version of Dequeue

  CorentinB

  committed
  61b7b9c

  Copy full SHA for 61b7b9c

Commits on Jul 15, 2024

• chore: align structures to save space

  

  equals215

  authored and

  CorentinB

  committed
  d81a5ff

  Copy full SHA for d81a5ff

• feat: remove the dequeue recusion

  

  equals215

  authored and

  CorentinB

  committed
  a1f0efd

  Copy full SHA for a1f0efd

• fix: crash when --api is not set (#79)

  Show description for de41b1d
  

  NGTmeaty

  authored and

  CorentinB

  committed
  de41b1d

  Copy full SHA for de41b1d

• Increment Prometheus counter only when --api is specified

  CorentinB

  committed
  7423728

  Copy full SHA for 7423728

Commits on Jul 16, 2024

• WIP - Switching to Protobuf

  CorentinB

  committed
  cc9eed1

  Copy full SHA for cc9eed1

• Fix: dequeue properly

  CorentinB

  committed
  2786d62

  Copy full SHA for 2786d62

• Use new NewItem function

  CorentinB

  committed
  bbdfa78

  Copy full SHA for bbdfa78

• Remove Gob queue encoder/decoder

  CorentinB

  committed
  cea09f0

  Copy full SHA for cea09f0

• Move item stuff to a separate file

  CorentinB

  committed
  7cb11da

  Copy full SHA for 7cb11da

• chore: moved protobuf to a new package to respect conventions

  equals215

  committed
  9427e39

  Copy full SHA for 9427e39

• Fix: cond wait lock problem

  CorentinB

  committed
  9c1a579

  Copy full SHA for 9c1a579

• WIP fix parallel test

  CorentinB

  committed
  25602e7

  Copy full SHA for 25602e7

• item encoding revamped

  equals215

  committed
  42386e5

  Copy full SHA for 42386e5

• reflect changes on crawl.go

  equals215

  committed
  1ea1232

  Copy full SHA for 1ea1232

Commits on Jul 19, 2024

• Removing useless boolean comparison

  CorentinB

  committed
  c07731a

  Copy full SHA for c07731a

• Merge branch 'main' into queue

  equals215

  committed
  90bc8bd

  Copy full SHA for 90bc8bd

• chore: second pass for merge

  equals215

  committed
  53b329c

  Copy full SHA for 53b329c

• Merge branch 'queue' of github.com:internetarchive/Zeno into queue

  equals215

  committed
  708fd86

  Copy full SHA for 708fd86

• chore: removed relicate

  equals215

  committed
  5ccc9e1

  Copy full SHA for 5ccc9e1

• Fix TestParallelEnqueueDequeue

  CorentinB

  committed
  19cb96d

  Copy full SHA for 19cb96d

• Remove the very useless LoggingChan

  CorentinB

  committed
  ab683ee

  Copy full SHA for ab683ee

Commits on Jul 20, 2024

• Remove sync.Cond & rewrite parallel test

  CorentinB

  committed
  c0ad1c9

  Copy full SHA for c0ad1c9

• Fix: TestNewItem

  CorentinB

  committed
  a7ac043

  Copy full SHA for a7ac043

• Typo: TestNewItem

  CorentinB

  committed
  2081aa3

  Copy full SHA for 2081aa3

Commits on Jul 21, 2024

• Merge branch 'main' into queue

  equals215

  committed
  56ab6ef

  Copy full SHA for 56ab6ef

• fix: some leftovers unseen errors from main merge

  equals215

  committed
  e50b979

  Copy full SHA for e50b979

Commits on Jul 23, 2024

• WIP on index

  equals215

  committed
  48a2817

  Copy full SHA for 48a2817

• index MVP - to be tested __thoroughly__

  equals215

  committed
  ac520c4

  Copy full SHA for ac520c4

• WIP changes

  equals215

  committed
  2998afc

  Copy full SHA for 2998afc

• enhancements for WAL + start of tests

  equals215

  committed
  1e0db1a

  Copy full SHA for 1e0db1a

• fix: corrected WAL fd inconsistencies thanks to new tests

  equals215

  committed
  2a3075b

  Copy full SHA for 2a3075b

• add more tests and corrected truncate accordingly

  equals215

  committed
  a39c0d5

  Copy full SHA for a39c0d5

• feat: index can recover after non-graceful exit (still need to check index vs. queue) ; tests: tested manager struct and functions + unit tested recovery

  equals215

  committed
  7c6b106

  Copy full SHA for 7c6b106

• prepare to merge and apply Corentin live review

  equals215

  committed
  0812bed

  Copy full SHA for 0812bed

• feat: remove index/file_io.go:performDump() printf

  equals215

  committed
  d0092e4

  Copy full SHA for d0092e4

• implemented requested changes

  equals215

  committed
  ed7f6cf

  Copy full SHA for ed7f6cf

• feat: enhance periodicDump routine

  equals215

  committed
  f11170c

  Copy full SHA for f11170c

• fix: periodicDump() missing a stat check

  equals215

  committed
  3221141

  Copy full SHA for 3221141

• Persist & load queue stats (#93)

  Show description for c602dcf
  

  CorentinB

  and

  equals215

  authored
  c602dcf

  Copy full SHA for c602dcf

Commits on Jul 24, 2024

• fix: queue-stats merge typo

  equals215

  committed
  7ce49a5

  Copy full SHA for 7ce49a5

Commits on Jul 25, 2024

• Merge remote-tracking branch 'origin/main' into queue

  CorentinB

  committed
  85682e7

  Copy full SHA for 85682e7

• tests: add some benchmarking for queue and index

  equals215

  committed
  3000dfa

  Copy full SHA for 3000dfa

• fix: temp files not contained in the same directory as Zeno & index file+WAL file gob codecs not being rotated ; tests: test the case when WAL is not empty, crash then recover

  equals215

  committed
  11d67e9

  Copy full SHA for 11d67e9

Commits on Jul 26, 2024

• fix: edge-case of queue fatigue: queue depletes over time when not enough seeds are fetched (from HQ f.ex.) to keep 100% of the workers running thus creating a "fatigue" where the less workers are …

  Show description for 2d56d81
  equals215

  committed
  2d56d81

  Copy full SHA for 2d56d81

• add queue empty bool state to live stats

  equals215

  committed
  8e724c4

  Copy full SHA for 8e724c4

• Implement host rotation and Enqueue/Dequeue access regulation via atomic booleans (#99)

  equals215

  authored
  67c3384

  Copy full SHA for 67c3384

Commits on Jul 27, 2024

• fix: open the WAL file as O_SYNC and get rid of file.Sync() calls consuming 1-5ms

  equals215

  committed
  98ec9fa

  Copy full SHA for 98ec9fa

• fix: reduce sleeps in worker

  equals215

  committed
  ccb6ac4

  Copy full SHA for ccb6ac4

• feat: make Dequeue non-blocking and manage the wait on the worker side for better control

  equals215

  committed
  65c0cf3

  Copy full SHA for 65c0cf3

• fix: TestDequeue expected return of a deprecated error

  equals215

  committed
  73ac185

  Copy full SHA for 73ac185

• fix: TestPersistentGroupedQueue_Close expected return of a deprecated error

  equals215

  committed
  1f4d8a6

  Copy full SHA for 1f4d8a6

Commits on Jul 29, 2024

• feat: add TEMP_DIR env variable support for queue benchmarks

  equals215

  committed
  49f4989

  Copy full SHA for 49f4989

• add: TestBatchEnqueue

  CorentinB

  committed
  2a8aa48

  Copy full SHA for 2a8aa48

• modify HQConsumer to batch enqueue

  CorentinB

  committed
  d2e0503

  Copy full SHA for d2e0503

• change HQ consumption logic

  CorentinB

  committed
  97c64ef

  Copy full SHA for 97c64ef

• fix: TestEnqueue & TestBatchEnqueue

  CorentinB

  committed
  cb68078

  Copy full SHA for cb68078

• rework HQConsumer pause logic a bit

  CorentinB

  committed
  d78a187

  Copy full SHA for d78a187

• fix: many tests

  CorentinB

  committed
  63bafd7

  Copy full SHA for 63bafd7

• fix: resuming queue couldn't start due to broken logic: queue.Empty was set to true so no worker would try to dequeue even though the que wasn't really empty

  equals215

  committed
  2ed7b9c

  Copy full SHA for 2ed7b9c

• enhancement: close all workers in parallel

  CorentinB

  committed
  6c0006d

  Copy full SHA for 6c0006d

• fix: TestQueueEmptyBool

  CorentinB

  committed
  ae2056d

  Copy full SHA for ae2056d

• fix: index.Pop logic error: WAL writing unnamed goroutine couldn't return on hostIndex.pop() error due to a lack of channel synchronisation

  equals215

  committed
  fae5403

  Copy full SHA for fae5403

Commits on Jul 30, 2024

• feat: implemented a handover process so that batch enqueue process can skip the queue by giving items directly to workers, also optimized workers with runtime.Goshed() (#101)

  equals215

  authored
  ca10b08

  Copy full SHA for ca10b08

• fix: infinite for loop on BatchEnqueue()

  equals215

  committed
  d6480c9

  Copy full SHA for d6480c9

• make workers yield more frequently

  equals215

  committed
  12e9aea

  Copy full SHA for 12e9aea

• feat: make outlinks batch enqueueable

  equals215

  committed
  01a0e22

  Copy full SHA for 01a0e22

• feat: add handover stats

  equals215

  committed
  e8ed999

  Copy full SHA for e8ed999

Commits on Aug 1, 2024

• optimize get list loading performance (#104)

  yzqzss

  authored
  c9c2a2c

  Copy full SHA for c9c2a2c

• Remove runtime.Gosched() in polling (#105)

  Show description for 4ce3cde
  yzqzss

  authored
  4ce3cde

  Copy full SHA for 4ce3cde

• Queue handover v2 (#106)

  Show description for 09ddbc7
  equals215

  authored
  09ddbc7

  Copy full SHA for 09ddbc7

Commits on Aug 2, 2024

• slight enhancements of handover and adjustments of workers sleeps

  equals215

  committed
  7029d7e

  Copy full SHA for 7029d7e

• fix: infinite for loop when finishing

  equals215

  committed
  6a34cac

  Copy full SHA for 6a34cac

Commits on Aug 3, 2024

• chore: Set Content-Type header for json response

  Show description for beac4a4
  yzqzss

  committed
  beac4a4

  Copy full SHA for beac4a4

• Improve WAL concurrency performance through the commit mechanism

  Show description for bc6f381
  yzqzss

  committed
  bc6f381

  Copy full SHA for bc6f381

• tests: update

  yzqzss

  committed
  538e2ab

  Copy full SHA for 538e2ab

• fix: QueueStats data race

  yzqzss

  committed
  37744a7

  Copy full SHA for 37744a7

• fix: deadlock :(

  yzqzss

  committed
  7a64613

  Copy full SHA for 7a64613

• push the seed list to the queue in batches

  yzqzss

  committed
  1c80c19

  Copy full SHA for 1c80c19

• typo: committed

  yzqzss

  committed
  be8b9b4

  Copy full SHA for be8b9b4

• Gracefully shutdown walCommitsSyncer()

  yzqzss

  committed
  04e02e6

  Copy full SHA for 04e02e6

• chore

  yzqzss

  committed
  5f81a4d

  Copy full SHA for 5f81a4d

• Merge pull request #107 from saveweb/queue-commit-await

  Show description for f56c463
  equals215

  authored
  f56c463

  Copy full SHA for f56c463

• feat: made @yzqzss commit feature optional by flag

  equals215

  committed
  9d179ab

  Copy full SHA for 9d179ab

• fix: forgot to separate Dequeue() function

  equals215

  committed
  f633a8f

  Copy full SHA for f633a8f

• chore: rm uint64 init to 0

  equals215

  committed
  5eaefe1

  Copy full SHA for 5eaefe1

• feat: check if items have been written before trying to commit WAL

  equals215

  committed
  48dc14b

  Copy full SHA for 48dc14b

• forgot to save before commit

  equals215

  committed
  c1ff8f0

  Copy full SHA for c1ff8f0

• fix: changes asked by yzqzss

  equals215

  committed
  03b2877

  Copy full SHA for 03b2877

• Merge pull request #109 from internetarchive/queue-WAL-write-batches

  Show description for 15df42d
  equals215

  authored
  15df42d

  Copy full SHA for 15df42d

• fix: hanging on index.Close()

  yzqzss

  authored
  2c68252

  Copy full SHA for 2c68252

Commits on Aug 4, 2024

• Merge pull request #110 from yzqzss/patch-3

  Show description for e09bf35
  equals215

  authored
  e09bf35

  Copy full SHA for e09bf35

• Merge branch 'main' into queue

  equals215

  authored
  b404d69

  Copy full SHA for b404d69