-
Notifications
You must be signed in to change notification settings - Fork 18
Background Tasks
The Play! Framework Application has a number of background tasks that run at repeated intervals.
This task runs every 3 seconds, and polls the NationStates API for the latest happenings activity in the world. Every type of happening is categorized into one of over 50 types, and actions occur based on the latest happenings. New nation foundations add new rows to the nation table. New regions add new rows to the region table. Endorsements update the endorsements table. World assembly applications, and resignations, refoundations, etc are all monitored. Happenings are inserted into the global happenings table. Because the happenings feed only lists happenings for nations, not regions, happenings that would affect a region also are generated based on the type of national happening that occurred. For example, when a nation is ejected from a region, it will show up in the happening feed for that nation. To ensure it is also recorded for that region, the proper happening text is transformed based on that event and inserted into the regional happenings table. This allows users to show past happenings in the region happenings area without having to request or record regional happenings separately from the NationStates API.
This task runs every 30 seconds. Because nation shards are not included in the daily nation dump of information, they must be gathered individually by the shard API. The shard data for World Assembly member nations is prioritized because their shard data changes the most, due to influence and endorsement changes. 12 WA nations have all their shard data updated every 30 seconds. Roughly, this translates into a shard update every 12 hours for WA nations. Non-WA nations also have 12 updates, but due to the much larger number, it results in an update every 4 to 5 days. Updates occur in order for the oldest data first. All national shard data and the timestamp it was taken is inserted into the shard table. In addition, for performance reasons, it is also updated in the latest_shard table. Originally, a SQL view was used to view the latest shards for every nation, but the size of the shard table caused this to be extremely slow.
This task runs every 30 seconds. It updates the region update order for the nations in 2 regions every 30 seconds. This ensures the ordering of the nations stays fresh (as the order that the names of nations that appear in the NATIONS request for a region is the order the nations will be updated during a NS major or minor update).
This task runs every 30 seconds. It monitors happenings for regions and nations who change their flag. It queues these nations or regions and updates 2 of each type every 30 seconds, requesting their latest flag data. This allows the flag redirect feature of the application to stay up to date and serve the correct flag for any nation or region. (I wish NationStates used permalinks for nation and regions, but they don't, and so I have to use this instead). Only the URL link is stored, not the actual flag. A 301 redirect is served to the user to the correct URL, making it a seamless redirect. If the nation or region is dead, it is redirected to the exnation or exregion flag.
This task runs every 60 seconds. It updates or creates bot-written pages for 3 NSWiki regions every 60 seconds. Most of the stats required for the update come from the shards table. It uses a mediawiki library to log into the mediawiki API for NSWiki.org and make changes with the bot-user (NS-BOT). The way the bot works is that it looks for comments in the markup and replaces the values inside of the comments. If the comments are removed (if say, the page is rewritten), then the page is left unaltered.
There are two of these tasks, one for each council. They run once an hour. They record the vote tallies for any active resolution. The reason they run once an hour is that to count votes, they must scan the national happenings table for votes, and this searches millions of rows. It takes 15 seconds to execute a single one of the vote tally queries. The vote tally is then stored in a table for the world assembly votes.
This is not really a repeating task, but it does run in a separate thread. It checks that the SQL Connection Pool is still connected, monitors each of the repeating tasks above (there are rare occasions where a thread may die and then the task would stop running) and if they die, restarts the application. The restart process actually invokes a shell command, which is part of the configuration.
##Daily Dump Thread Also not really a repeating task, as it polls the NationStates daily dump urls every 5 minutes and checks to see if the timestamp or size has changed since the previous values. If they do, it downloads the dump, and starts a Dump Task to convert the XML into a H2 (in memory, local) database. Then the data is copied from the H2 database to the MySQL database. Yes, it could read the data directly into the MySQL database, but I have latency concerns with the huge volume of data, considering not all of it is used. The process to download a dump, convert it, and read the data takes roughly an hour. It is slowed down a bit intentionally (with sleep calls) to prevent the process from consuming all the CPU cycles and preventing normal web requests from being served.
If the configuration contains AWS secret and rest keys, it will also put the latest dumps in a AWS S3 bucket for offsite storage.