Add missing indexes and a new index to jobs

[iwiznia]

Held on adding the new index to all prod dbs

Details

Added existing indexes PLUS a new one for staging jobs

Fixed Issues

https://github.com/Expensify/Expensify/issues/415244

Tests

Start webrock, check all indexes are correctly created

---

Internal Testing Reminder: when changing bedrock, please compile auth against your new changes

[iwiznia]

This is the new index, the rest of them I got them from prod.

[iwiznia]

Dry run mode of the index on virt1.lax:

(saltfab3) ➜  saltfab git:(main) ✗ fab -g bastion1.sjc -H "virt1.lax" authrock.deployReadinessCheck:quiet=True bedrock.addIndex:dryRun=True


/Users/ionatan/venv/lib/python3.7/site-packages/paramiko/transport.py:178: CryptographyDeprecationWarning: Blowfish has been deprecated
  'class': algorithms.Blowfish,
Checking installed python packages match the versions in requirements.txt...
Packages are installed and up to date.
Checking for latest saltfab version...
[localhost] local: git pull origin main
From github.com:Expensify/Ops-Configs
 * branch              main       -> FETCH_HEAD
Updating 96861e3b..8c66d7ba
Fast-forward
 saltfab/libs/admin.py   | 18 ++++++++++++++++++
 saltfab/libs/grafana.py | 12 ++++++++++++
 saltfab/tasks/admin.py  | 20 ++++++++++----------
 3 files changed, 40 insertions(+), 10 deletions(-)
I updated for you AND I am re-running the command because I am great
/Users/ionatan/venv/lib/python3.7/site-packages/paramiko/transport.py:178: CryptographyDeprecationWarning: Blowfish has been deprecated
  'class': algorithms.Blowfish,
[localhost] local: /usr/bin/uname -n
Checking installed python packages match the versions in requirements.txt...
Packages are installed and up to date.
Checking for latest saltfab version...
[INFO] Matched hosts: ["virt1.lax"]
[virt1.lax] Executing task 'authrock.deployReadinessCheck'
[virt1.lax] Executing task 'bedrock.addIndex'
Enter the new index to add: CREATE INDEX jobsPriorityNextRunWWWStag ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'www-stag/*';
[INFO] [virt1.lax] Table: jobs
[INFO] [virt1.lax] Schema of jobs to be updated:
[virt1.lax] run: sudo sqlite3 /var/lib/bedrock/bedrock.db ".schema jobs"
[virt1.lax] out: CREATE TABLE jobs ( created TIMESTAMP NOT NULL, jobID INTEGER NOT NULL PRIMARY KEY, state TEXT NOT NULL, name TEXT NOT NULL, nextRun TIMESTAMP NOT NULL, lastRun TIMESTAMP, repeat TEXT NOT NULL, data TEXT NOT NULL, priority INTEGER NOT NULL DEFAULT 500, parentJobID INTEGER NOT NULL DEFAULT 0, retryAfter TEXT NOT NULL DEFAULT "");
[virt1.lax] out: CREATE INDEX jobsName    ON jobs ( name    );
[virt1.lax] out: CREATE INDEX jobsStatePriorityNextRunName ON jobs ( state, priority, nextRun, name );
[virt1.lax] out: CREATE INDEX jobsParentJobIDState ON jobs ( parentJobID, state ) WHERE parentJobID != 0;
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanMerchantAndCategory ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanMerchantAndCategory*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanAmountAndCurrency ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanAmountAndCurrency*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanCreated ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanCreated*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanIsCash ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanIsCash*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScan ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScan*';
[virt1.lax] out: CREATE INDEX jobsManualSmartscanReceiptID ON jobs ( JSON_EXTRACT(data, '$.receiptID') ) WHERE JSON_VALID(data) AND name GLOB 'manual/SmartScan*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunWWWProd ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'www-prod/*';
[virt1.lax] out:

[INFO] [virt1.lax] Command to run: PRAGMA threads=50; CREATE INDEX jobsPriorityNextRunWWWStag ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'www-stag/*';
[INFO] [virt1.lax] Execute?
[localhost] local: read

[INFO] [virt1.lax] Not stopping bedrock because dryRun=True
[INFO] [virt1.lax] [DRYRUN] Not applying schema change.
[INFO] [virt1.lax] Schema of jobs after update:
[virt1.lax] run: sudo sqlite3 /var/lib/bedrock/bedrock.db ".schema jobs"
[virt1.lax] out: CREATE TABLE jobs ( created TIMESTAMP NOT NULL, jobID INTEGER NOT NULL PRIMARY KEY, state TEXT NOT NULL, name TEXT NOT NULL, nextRun TIMESTAMP NOT NULL, lastRun TIMESTAMP, repeat TEXT NOT NULL, data TEXT NOT NULL, priority INTEGER NOT NULL DEFAULT 500, parentJobID INTEGER NOT NULL DEFAULT 0, retryAfter TEXT NOT NULL DEFAULT "");
[virt1.lax] out: CREATE INDEX jobsName    ON jobs ( name    );
[virt1.lax] out: CREATE INDEX jobsStatePriorityNextRunName ON jobs ( state, priority, nextRun, name );
[virt1.lax] out: CREATE INDEX jobsParentJobIDState ON jobs ( parentJobID, state ) WHERE parentJobID != 0;
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanMerchantAndCategory ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanMerchantAndCategory*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanAmountAndCurrency ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanAmountAndCurrency*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanCreated ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanCreated*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScanIsCash ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScanIsCash*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunManualSmartScan ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'manual/SmartScan*';
[virt1.lax] out: CREATE INDEX jobsManualSmartscanReceiptID ON jobs ( JSON_EXTRACT(data, '$.receiptID') ) WHERE JSON_VALID(data) AND name GLOB 'manual/SmartScan*';
[virt1.lax] out: CREATE INDEX jobsPriorityNextRunWWWProd ON jobs (priority, nextRun) WHERE state IN ('QUEUED', 'RUNQUEUED') AND name GLOB 'www-prod/*';
[virt1.lax] out:

[INFO] [virt1.lax] [DRYRUN] restarting bedrock
[INFO] [virt1.lax] Success? If you want to run this for real, run without dryRun!

Done.
Disconnecting from bastion1.sjc... done.
Disconnecting from virt1.lax... done.

[flodnv]

👍

[iwiznia]

virt1.lax index is added and all seems good

[iwiznia]

virt2.lax is done

[iwiznia]

virt1.sjc was skipped since it is offline for klara

[iwiznia]

virt2.sjc done

[iwiznia]

virt1.rno done

[iwiznia]

virt2.rno is done