GTFS Updates: Smart Updates & Storage

[nateschickler0]

GTFS data is not updated automatically and will cause routing to fail if updated too early or not at all. We want to

1. automatically download and verify the correct GTFS data on setup/install/run
2. store non-current (old and future) GTFS data both locally and on S3
3. detect and switch out-of-date GTFS data files with new GTFS data automatically to ensure 0 downtime
4. ensure the server/graph-hopper replaces and rebuild all caches and graphs when new GTFS data is provided (bonus: without needing to restart the server or redeploy)

[nateschickler0]

#52 The install/deploy script includes functionality to download and unzip GTFS data, so these issues are somewhat related.

[nateschickler0]

Data scripts removed and functionality placed into TCATUtils.js, also added date validation and data expiration and invalid notifications #89

[mattbarker016]

Summary

GTFS data is not updated automatically. This will cause all routing to fail if updated too early or not at all.

Implementation / Scope:

• Updating graphhopper to latest release
• GTFS diff-ing, versioning, tagging, and downloading
• Creating a database
• Pulling and pushing GTFS data to/from our own database
• DB Management
  • Redundant backups
  • Delete old and unusable data
  • Maintain data for current period
• Updating the GTFS data for the Graphhopper routing services
  • Rebuilding the graph cache
  • Restarting the service automatically

Key Issues:

• Graphhopper services run on a separate servers and Docker containers but need to update the GTFS data in sync right as the old version becomes invalidated
• Future-proofing for possible changes in how TCAT formats or serves the GTFS data