Single post in Server performance and desyncs
Forum Index > Core > Announcements > News Archive > Server performance and desyncs >
![Niet [Adam]'s Avatar](https://pokefarm.com/upload/:J/avatar/Niet_Icon150.png)
As I'm sure you've noticed, if you've tried playing PFQ around Reset, it's... laggy. Here's what I'm currently doing to try and fix it.
Update(s) at end of post The underlying cause is, of course, "my ten-year-old code was never designed to handle this." But short of rewriting the whole thing, I can make some tweaks to squeeze some performance out of things. The database is definitely the biggest bottleneck right now. The major lag spikes occur when, for whatever reason, the "sessions" table falls out of active cache. When this happens, every single user gets lag because their requests to the server are trying to query the "sessions" table for authentication and updating online status. Imagine a door that's supposed to stay open, but closes for some reason as thousands of people are trying to get through it at once... Even when the door opens again, it's a mess for some time afterwards. That's the kind of thing going on here. To address this, configuration settings are being tweaked, that should hopefully increase the "open table cache" and either prevent this issue outright, or at least significantly reduce its impact. A consequence of the database "misbehaving" is that this leads to desyncing of data between it and the server's Memcache. I've been using Memcache to store copies of data in memory rather than having to query the database for them. However, this can lead to issues if the Memcache and database disagree on what the value should be. In the best case, things will just look a bit weird for a while. Most commonly, Pokémon may be "stuck" in a Scour mission that they aren't actually on, for instance. But in the worst case, if you buy ZC and it gets desynced, and then you do something else with ZC, the desynced value would then be saved back to the database. Thankfully I can fix ZC in particular, but it's not good! To try and reduce the impact of this, I have changed the default time that Memcached holds on to data, reducing it from one hour all the way down to one minute. The downside is that the database may need to be queried more often for the data, the upside is that any desyncs will fix themselves in a minute rather than an hour! I'm going to run the game like this for a while, maybe a few days, and see what impact this has on the game's performance. Hopefully, it won't affect much, and we'll just get the benefits of "less desync." We'll see!
Update Feb 27th A number of factors have been successfully identified as contributing to lag. • Pokémon Platform images each caused the full session handler to load. This is almost certainly a major culprit in why the session table was getting so hammered. • Additionally, forum posts with very large numbers of images would cause an excessive number of connections to the server.In the near future I will be implementing "limit per post" limits on certain BBCodes to reduce lag caused by "expensive" codes. This will probably be around 50 for [img] but we'll see what works. This will affect Journals that have "Live Dexes" - I'm very sorry but these were causing unmanageable amounts of lag... Please consider breaking down your Live Dex posts into smaller pieces, or instead just using your Fields to present such a Live Dex.
On further consideration, I can do something better than just nuking forum posts. I will investigate better options.
• 42 accounts have been locked for (strongly) suspected botting.
Update: Thankfully, the vast majority of these were not as strong as previously thought. The mod team is working on resolving the affected users' tickets and most should be unlocked very soon.
The underlying cause was a combination of server lag and some QoL/CSS codes making certain activity look very suspicious. The UI has been updated to alleviate this issue, but in general please try to avoid hammering the server. If it doesn't load the first time, clicking again isn't gonna make it load any faster!
If you know someone who was locked, encourage them to send a ticket to the Support Centre for discussion. This thread is not the location to discuss locks, especially locks that are not your own (doing so is in violation of Site Rule 2H).
• It was possible to create shortlinks of shortlinks. We heard you like shortlinks, but it was a bit silly to have a system potentially enter an endless loop of redirects. This has been patched.
I will be continuing to monitor the health of the game going forward, and will report back if there are any further updates to this situation. Thank you for your continued patience as we deal with this!
I forgot to mention possibly the biggest adjustment: The Shelter updates its contents every 15 minutes, including at 00:04. This is right after the Shelter "opens" for the day at 00:02, causing a lot of people to be trying to use the Shelter while it performs its first update of the day. To address this, I have hopefully simplified the Shelter update code, and also temporarily the Shelter will remain closed until 00:10. I will be looking to bring this back to opening as early as possible, but for now this should give the script plenty of time to run. Also, the Shelter page will be much more forward with its "Shelter is closed for Reset" text. Further adjustments made today: • The "Clickback" and "Open 10" links will now disable themselves and show a "Loading..." text when clicked. This will prevent issues involving certain QoL/CSS codes that pull it in to your "no mouse move clicking" script area, as no matter how many times you click them, it will only cause one load. • Images on the forum have had the loading="lazy" attribute added to them. This instructs the browser to only load them when they become visible on screen rather than all at once. This doesn't always play nice with Accordions when they are opened for the first time (because the images haven't been loaded yet, it can't animate to the correct height) but this is fairly minor and "fixes itself" after the initial loading. This should also cut down on the high volume of image requests mentioned earlier in this post.
Update(s) at end of post The underlying cause is, of course, "my ten-year-old code was never designed to handle this." But short of rewriting the whole thing, I can make some tweaks to squeeze some performance out of things. The database is definitely the biggest bottleneck right now. The major lag spikes occur when, for whatever reason, the "sessions" table falls out of active cache. When this happens, every single user gets lag because their requests to the server are trying to query the "sessions" table for authentication and updating online status. Imagine a door that's supposed to stay open, but closes for some reason as thousands of people are trying to get through it at once... Even when the door opens again, it's a mess for some time afterwards. That's the kind of thing going on here. To address this, configuration settings are being tweaked, that should hopefully increase the "open table cache" and either prevent this issue outright, or at least significantly reduce its impact. A consequence of the database "misbehaving" is that this leads to desyncing of data between it and the server's Memcache. I've been using Memcache to store copies of data in memory rather than having to query the database for them. However, this can lead to issues if the Memcache and database disagree on what the value should be. In the best case, things will just look a bit weird for a while. Most commonly, Pokémon may be "stuck" in a Scour mission that they aren't actually on, for instance. But in the worst case, if you buy ZC and it gets desynced, and then you do something else with ZC, the desynced value would then be saved back to the database. Thankfully I can fix ZC in particular, but it's not good! To try and reduce the impact of this, I have changed the default time that Memcached holds on to data, reducing it from one hour all the way down to one minute. The downside is that the database may need to be queried more often for the data, the upside is that any desyncs will fix themselves in a minute rather than an hour! I'm going to run the game like this for a while, maybe a few days, and see what impact this has on the game's performance. Hopefully, it won't affect much, and we'll just get the benefits of "less desync." We'll see!
Update Feb 27th A number of factors have been successfully identified as contributing to lag. • Pokémon Platform images each caused the full session handler to load. This is almost certainly a major culprit in why the session table was getting so hammered. • Additionally, forum posts with very large numbers of images would cause an excessive number of connections to the server.
I forgot to mention possibly the biggest adjustment: The Shelter updates its contents every 15 minutes, including at 00:04. This is right after the Shelter "opens" for the day at 00:02, causing a lot of people to be trying to use the Shelter while it performs its first update of the day. To address this, I have hopefully simplified the Shelter update code, and also temporarily the Shelter will remain closed until 00:10. I will be looking to bring this back to opening as early as possible, but for now this should give the script plenty of time to run. Also, the Shelter page will be much more forward with its "Shelter is closed for Reset" text. Further adjustments made today: • The "Clickback" and "Open 10" links will now disable themselves and show a "Loading..." text when clicked. This will prevent issues involving certain QoL/CSS codes that pull it in to your "no mouse move clicking" script area, as no matter how many times you click them, it will only cause one load. • Images on the forum have had the loading="lazy" attribute added to them. This instructs the browser to only load them when they become visible on screen rather than all at once. This doesn't always play nice with Accordions when they are opened for the first time (because the images haven't been loaded yet, it can't animate to the correct height) but this is fairly minor and "fixes itself" after the initial loading. This should also cut down on the high volume of image requests mentioned earlier in this post.
Clip from Pokémon anime, re-lined by me
-- OMNOMNOM!Featured story:
Injustice
Feedback welcome!
