This topic is locked

Optimizing queries for list views

5/25/2023 4:47:33 AM
PHPRunner General questions
mbintex author

Hi at all,
when I look at the queries built by PHPRunner with the $dDebug setting on, I see that it fetches all the fields from the query screen for the list view, even when only some fields are actually shown in the list view (search reduced to only one field too) and all fields only needed in add and edit pages. This can lead to longer query and especially fetch times in MySQL.
Just as an example: A query of mine with round about 20 fields and a limit of 1000 is queried in 0.062 sec and fetched in 0.438 sec. If I only select 5 fields from the same query, the query time still is 0.062, but the fetch time is now down to 0.031 sec.
So I wonder, how I could get more efficient list views and still have the capability of adding and editing all fields of a table?
I could create one list view with only some fields used for listings and then jump from there to the add and edit pages based on the complete query. But then I would have two PHPRunner views for a table - a lot of overhead.
Any alternatives?

Sergey Kornilov admin 5/25/2023

Right now creating a custom view for this kind of task is the only option.

mbintex author 5/25/2023

@admin
I gave a try for the SQL view with querying only two fields in list view and a bunch more in view/update view.
Regarding $dDebug it seems, that this way the list view really only queries two fields.
Can you confirm, admin?

fhumanes 5/25/2023

Hello,
I understand what you ask and that is different from what I am going to propose.
It is very important to know the consultations made to the database manager and many of these consultations are created by PHPRUNNER, so it is not so simple to identify them.
Once they identify you can analyze the different costs and optimize the accesses.
For MySQL, I have written an article where I expose how these different activities can be done. https://fhumanes.com/blog/guias-desarrollo/guia-12-phprunner-optimizar-accesos-a-mysql/
Greetings,
fernando

mbintex author 5/26/2023

Thanks for your link.
In my case it is not so much the complexity of the table structure, but the shear amount of data. I am talking about transactions data in a bookkeeping system. Of course this data is growing on a daily basis and due to legislation and taxation requirements we have to archive the data for ten years.
In the early stages of our SaaS offering all this was no real problem, since we only had new or "young" customers. But now we are in our fifth year and the amount of data is rising steadily at least for some users.
Best idea I had in the last weeks for optimizing performance was to only query the data of the last three years in standard. And I found an easy way for doing that (without doing any duplication of PHPRunner views).
In the login event I have
$_SESSION["archivdatum"]=date('Y-12-31', strtotime('-3 years'));giving me the last day of the year three years ago.
I use this session variable then in my queries on the transaction table like this
WHERE (Datum >':session.archivdatum')and only get the data of the last three years. So there is no problem anymore with the ever growing amount of data.
But if someone wants or has to see the older data, I have custom button, that will show all data on demand. The code is quite easy:
Client before just tells the user what will happen:
`return ctrl.dialog({
title: 'Archivierte Daten',
header: '<table style=\'border: none;\'><td style=\'color: var(--akzentfarbe); font-size: 32px; width: 50px;\'><i class=\'glyphicon glyphicon-exclamation-sign\'></i></td><td>Archivierte Daten (älter als drei Jahre) einblenden.</td></table>

',
ok: '<i class=\'glyphicon glyphicon-plus\'></i> Einblenden',
cancel: 'Abbrechen',
height: 300
});Then the server part changes the session variable $_SESSION["archivdatum"]=date('Y-12-31', strtotime('-10 years'));
and the client after event just reloads the page location.reload();`The reload repeats the query, but this time queries all data upto 10 years.
The concentration on the current data gave my application an enourmous speed bump for heavy users and it makes it somehow future safe, no matter how many records a user might enter. Plus I don´t have any overhead for archive views in my PHPRunner project.

fhumanes 5/26/2023

Thanks for sharing your solution.
Greetings