Original post: Investigation continues into the lag, here's what I've got in my crosshairs.
Update post: Post
Followup post: Post-mortem
Post-sleep update: Next steps I am always apprehensive of touching the foundations of PFQ's code. Shaky as they are, they at least work, and one wrong move could bring the whole thing crashing down. However, there comes a time when it must be done. Following the test of disabling the server cache and seeing what it did to the server load, I have identified some ways in which the core parts of the code could be better. Firstly, the ability to load objects in bulk. Right now, to load a Field of Pokémon, the current code queries the database for the IDs of things in that Field, and then (after checking if they exist in cache, which they don't any more) fetch each of them, one by one, from the database. This is obviously inefficient when there's no cache (with the cache, it would only be inefficient once, and then very fast). To address this, I am writing new code that enables me to create objects from data, rather than just by ID. This means that if I already have the data from some other query (such as "get all the things in this Field"), I can just create the objects from that in one step. Next up, I noticed that some objects were being loaded even if they weren't being used. Again this wasn't such a big deal when the server cache was involved because it would just fetch it from memory and discard it later, no biggie, but now it's fetching it from the database every time, which is a problem. To address this, I've added a new way to get object properties, whereby they can be "lazy loaded" - meaning it only loads them if some other part of the code actually asks for it, rather than loading it up front when it isn't even needed. A notable example for this would be the Forums, where every single user link on the page would actually be loading the entire user's data, stats and profile, all just to render one little link. This is fixed now, as the stats and profile are only loaded if they're actually needed. Some other performance issues were similarly masked by the use of server cache, such as loading the Type Race team leaders. This was actually being done twice, for every single user link on the page, instead of just once ever. This has been addressed, along with other similar bugs. Still to do, I will be re-implementing the server cache to handle static data, such as the Pokémon species data, item data etc.. Things that don't change often, or at all, can be held in memory and used when needed without having to ask the database for them every time. Another thing I want to implement is an inventory manager class. Currently each item in your inventory is a separate entity with an ID, owner, and quantity. Loading your inventory, or loading those forum posts with "item tables" inside, or the Notification system loading your Wishforge progress... all of these and more will make heavy use of many items, sometimes even all items, and they're all being asked for one by one. The inventory manager will load your entire inventory in a single go, and then provide hooks for get/use/receive function calls. There are many more things besides, but the key point all of them have in common is that none of these changes will affect code in other places. For example the inventory code will still use getItem() to get the quantity of the given item in your inventory. Fetching $user->profile will "magically" load the profile data belonging to that user (whereas previously it was pre-loaded every time). This means that very little, if anything, will change elsewhere in the code. But the performance should be much better. In order to push these updates live, however, I will need a maintenance period - and since we have an MCW going on right now, it's going to have to wait! This gives me time to implement more improvements anyway, so it's fine. With any luck, this might even fix some of the weirdness people have been having with some features like the Scour mission energy. If not, it'll at least be easier to work with. I just wanted to make sure you all could be made aware of what I'm working on. I do occasionally drop some information over in our Discord server so come on by if you'd like to see that. As for the maintenance window, I'm thinking maybe Wednesday, but that'll depend on whether or not I'm satisfied the updates are working on the dev server, so I'll keep you updated if things change. It'll likely be around the middle of the day (Server Time) when fewer users are online. That's it for now, happy to answer any questions you may have!

really, really appreciate the updates, Niet! it's such a treat to be let in on what you're doing behind the scenes; it gives me a greater appreciation of the work that goes into this site, and I'm very grateful that you're actively looking into making things better! it really is a breath of fresh air compared to how most of the internet is these days. thanks for the update! :>

So I wasn't imagining the scour energy thing where it was depleting much faster than before. Good to know!

yessss i’m excited for you to mess with the base code !! it needs it! i’d happily wait a few days of down time while stuff breaks if needed

I do not have the technical understanding to know what basically any of that means, but I do really appreciate the explanations being given for those of us who /do/ know.

Good luck, keep up the good work. :D

Thanks for the work you're putting in. It's all going straight over my head but it sounds complicated so appreciate it!

Can I just say, I love all the positivity on these kinds of threads lately. I can remember a time (not that long ago, really) when nearly every announcement or update post was filled with angry, negative comments, no matter what it was about. I'm not sure what's changed- increased transparency from staff, mods cracking down on unproductive negativity, or something else- but it's just such a breath of fresh air, even as a spectator. Can't even imagine what a change it is to be on the receiving end of such comments. Sure there's going to be people who don't like each update, but I'm not seeing the anger I used to see, more "this is why I don't like this thing" comments. Again that could just be mods deleting posts faster, but from a reader perspective all the update posts seem to have a much healthier vibe to them.

The updates sound amazing! For someone who works in IT it sounds like the code grew much since first implementation. It happens often, when a function is needed in one place but in the course of changes, it grows to cover more area. Lazy Loading is super useful for reducing server lag, especially for data-heavy sections, such as Fields.❤️ I'm more interested in the thing about the Fields: loading of its content sounds rather confusing, but it may be due to my english being a second language. I would expect that a series of Pokemon in a field would be loaded as collection - is it currently being loaded as an array of Pokemon ids and later loaded again as Pokemon objects? Oh! It actually makes sense with cache! If you have an id in cache you don't have to query for an entire object, just look it up 😀 PokeFarm cache is fascinating, I would love to learn more :3

As someone who does not understand code at all, i really appreciate these updates and explanations! The work you do for PFQ is so great!