GUI/API scalability for huge workflows

[LuiggiTenorioK]

For some time, inside the Climate DT VM, there have been many experiments that handle more than 100,000 jobs. This is a problem because many operators cannot use the API/GUI to have insights about these huge experiments.

To address that, we need to redesign some GUI pages to not retrieve and show all the information at once, and rather show small groups of information that are searchable. For that, we need to create some new API endpoints to serve experiment data in smaller chunks and use strategies like pagination or maybe caching some request responses.

To ensure this strategy actually improves the user experience, it is necessary that BSC-ES/autosubmit#2391 is ready and becomes the new way to store the job data. This is because the current job data can only be loaded as a whole and not in a segmented way.

For the first phase, the Table and Quick View of the GUI are targeted in this redesign since they seem to be the ones operators use most when going to huge workflows, and are less complex to modify.

[LuiggiTenorioK]

Issue to track graph view page optimization: #267

[LuiggiTenorioK]

@youcheng-csc @TracyMcBean this is the issue related to what was mentioned in https://jira.eduuni.fi/browse/CSCDESTINCLIMADT-778. As mentioned above, the Quick and Table views are targeted for an update. However, you might find other features/views more useful than others.

We would appreciate hearing any feedback or suggestions to improve the GUI and make your tasks easier.

cc @mcastril

[youcheng-csc]

Hi @LuiggiTenorioK Thank you for creating this issue. For me, the priority is to organize the quick view. Every time, it only show 100 jobs with running, failled, queueing job in the front of the list.

It would be better, if

1. The priority is to make it fast to show up, especailly for quick view. Make it as fast as possible. I have never been able to visit tree views for these long simulations. It could be slow but they should be there after 10 mins waiting.

2. we can classify the jobs in the similar way as tree view, distribute these jobs into a mother direcotry for example SIM, AQUA, ....... Now I use the browse function to load all SIM jobs and all AQUA jobs......

3. It would be good to allow the quick view load more jobs (more than 100 jobs). IT is good to load all jobs into one page.

4. When these names of the job show up in the quick view or tree view, it would be much more convenient to make it possible for users to copy these job names.

Above is what I can think about right now. Probably one or some of them are impossible to realize, but anyway it is feedback. Thank you again and hope we can improve GUI.

[LuiggiTenorioK]

Thanks for the feedback @youcheng-csc! 🙌

1. The priority is to make it fast to show up, especailly for quick view. Make it as fast as possible. I have never been able to visit tree views for these long simulations. It could be slow but they should be there after 10 mins waiting.

To accomplish that, we are planning to load first the structure of the tree (date, member, chunks). Then, the jobs will be loaded when a section of the tree is expanded. The main problem is to render the whole tree. This is not feasible when the tree has a large number of leaves, so we will try to minimize the number of visible nodes.

2. we can classify the jobs in the similar way as tree view, distribute these jobs into a mother direcotry for example SIM, AQUA, ....... Now I use the browse function to load all SIM jobs and all AQUA jobs......

Yes, in the Table View, we will apply that strategy. The idea is to keep the summary on the upper side. Then, the bottom side will have the job list but separated by sections (SIM, SETUP, OPA, etc) with a search bar and pagination (limit to maybe 100 jobs per page).

3. It would be good to allow the quick view load more jobs (more than 100 jobs). IT is good to load all jobs into one page.

As I mentioned in the previous points, it is not feasible to render all the jobs at once. As an alternative, we can give an option to increase the number of rendered jobs to maybe 100, 500, or 1000 jobs. Higher numbers will be difficult to render, especially in browsers used on low-spec machines.

4. When these names of the job show up in the quick view or tree view, it would be much more convenient to make it possible for users to copy these job names.

This problem has already been addressed in recent versions of the GUI. Unfortunately, CSC was not able to update the GUI to this new version yet. We have been unsuccessfully pushing for that for a couple of months now.

[kinow]

This problem has already been addressed in recent versions of the GUI. Unfortunately, CSC was not able to update the GUI to this new version yet. We have been unsuccessfully pushing for that for a couple of months now.

cc @sigurbrynjar