Feature request: Total CPU and GPU hours for experiment

[TracyMcBean]

Hello autosubmit-gui team (@LuiggiTenorioK @mcastril et al),

we regularly have to report the amount of CPU and GPU hours used for the ClimateDT runs. We are asked to report the core hours used for specific experiments, e.g. the different operational runs. Currently, we have the CHSY (core hours per simulated year I believe) but this does not include the failed sim chunks, as for a failed chunk one cannot easily check how far the simulation got.
Additionally, on LUMI-G the computation of the billing units for core hours is different for GPUs than for CPUs. I am not sure if this makes a difference here, but it would be worth checking if for GPU hours on LUMI a different approach would be needed for the computation (@JuhaTonttila for details).

The wish would be to have a total CPU/GPU hours for the entire experiment (without APPS should be good enough for now though) that one can easily find.
Currently, for instance we use sacct if we need more precise information or as cross evaluation point for instance for the GPU khours.

Do you think this is doable?

FYI @sigurbrynjar

[JuhaTonttila]

It's mostly about the billing rules. They're listed here. https://docs.lumi-supercomputer.eu/runjobs/lumi_env/billing/. For standard-g it's pretty straightforward for an individual big run.

[LuiggiTenorioK]

Hi @JuhaTonttila. The GUI reports a CHSY based on the requested JOBS.[JOB_SECTION].PROCESSORS declared in the experiment configuration files, which doesn't necessarily reflect the actual CPU/GPU hours used.

If this value can be used anyway, you can check the job history in the Tree/Graph view of the experiment. Like this:

There you'll get all the run times of the job, whether it is successful or failed. Then, you can also use the NCpus columns, which refer to the requested PROCESSORS as explained above.

However, if you need the real used cores value, we'll need to implement a way to get that data in Autosubmit. Nowadays, it only uses sacct to get the consumed energy information. @kinow @dbeltrankyl can give a better opinion if that is feasible for a future version of Autosubmit.