Single post in Lag issues, progress, and planned maintenance [TBD]

Original post: Investigation continues into the lag, here's what I've got in my crosshairs.
Update post: Post
Followup post: Post-mortem
Post-sleep update: Next steps I am always apprehensive of touching the foundations of PFQ's code. Shaky as they are, they at least work, and one wrong move could bring the whole thing crashing down. However, there comes a time when it must be done. Following the test of disabling the server cache and seeing what it did to the server load, I have identified some ways in which the core parts of the code could be better. Firstly, the ability to load objects in bulk. Right now, to load a Field of Pokémon, the current code queries the database for the IDs of things in that Field, and then (after checking if they exist in cache, which they don't any more) fetch each of them, one by one, from the database. This is obviously inefficient when there's no cache (with the cache, it would only be inefficient once, and then very fast). To address this, I am writing new code that enables me to create objects from data, rather than just by ID. This means that if I already have the data from some other query (such as "get all the things in this Field"), I can just create the objects from that in one step. Next up, I noticed that some objects were being loaded even if they weren't being used. Again this wasn't such a big deal when the server cache was involved because it would just fetch it from memory and discard it later, no biggie, but now it's fetching it from the database every time, which is a problem. To address this, I've added a new way to get object properties, whereby they can be "lazy loaded" - meaning it only loads them if some other part of the code actually asks for it, rather than loading it up front when it isn't even needed. A notable example for this would be the Forums, where every single user link on the page would actually be loading the entire user's data, stats and profile, all just to render one little link. This is fixed now, as the stats and profile are only loaded if they're actually needed. Some other performance issues were similarly masked by the use of server cache, such as loading the Type Race team leaders. This was actually being done twice, for every single user link on the page, instead of just once ever. This has been addressed, along with other similar bugs. Still to do, I will be re-implementing the server cache to handle static data, such as the Pokémon species data, item data etc.. Things that don't change often, or at all, can be held in memory and used when needed without having to ask the database for them every time. Another thing I want to implement is an inventory manager class. Currently each item in your inventory is a separate entity with an ID, owner, and quantity. Loading your inventory, or loading those forum posts with "item tables" inside, or the Notification system loading your Wishforge progress... all of these and more will make heavy use of many items, sometimes even all items, and they're all being asked for one by one. The inventory manager will load your entire inventory in a single go, and then provide hooks for get/use/receive function calls. There are many more things besides, but the key point all of them have in common is that none of these changes will affect code in other places. For example the inventory code will still use getItem() to get the quantity of the given item in your inventory. Fetching $user->profile will "magically" load the profile data belonging to that user (whereas previously it was pre-loaded every time). This means that very little, if anything, will change elsewhere in the code. But the performance should be much better. In order to push these updates live, however, I will need a maintenance period - and since we have an MCW going on right now, it's going to have to wait! This gives me time to implement more improvements anyway, so it's fine. With any luck, this might even fix some of the weirdness people have been having with some features like the Scour mission energy. If not, it'll at least be easier to work with. I just wanted to make sure you all could be made aware of what I'm working on. I do occasionally drop some information over in our Discord server so come on by if you'd like to see that. As for the maintenance window, I'm thinking maybe Wednesday, but that'll depend on whether or not I'm satisfied the updates are working on the dev server, so I'll keep you updated if things change. It'll likely be around the middle of the day (Server Time) when fewer users are online. That's it for now, happy to answer any questions you may have!