Implement total core-hours, comparative footprint and energy and distribution of time of an experiment in Performance section

[MissterP]

Hi!

The predecessor of this issue is this one.

I plan to enhance the GUI Performance section of an experiment by adding the total core-hours, comparative footprint and energy and distribution of time of an experiment. I would like to do the implementation to the file ExperimentPerformance.jsx, this way:

• Total core-hours: Add next to Simulated time of the section Summary an attribute called Total core-hours; the value would be obtained with the variable data which would have the result of the API call. The motive to implement this metric and not use the CPU Consumptionis because it doesn't take into account retrials and shows only the core-hours consumption for the last successful SIM jobs of each chunk. An explanation about this would be added at the ?.

• Comparative footprint and energy: To facilitate the comprehension of the results of Total energy consumed and Total footprint generated; I would like to add at the Sustainability section a barplot (to show the difference with common references i.g carbon footprint generated by 1kg of beef) and a table (to show the equivalence with common references i.g the total footprint of the experiment is equivalent to X kg of beef) for each of the metrics. The plots used would be be from a new dependency called recharts. Moreover, to show a warning message in case there is no energy or footprint the dependencies antd and @ant-design/icons would be used.

• Distribution of time: To understand the distribution of time for the different phases of an experiment; I would like to add a new section called Distribution of time which will show two plots (once decided which type, it will be edited) of the distribution of the three phases of an experiment (jobs before sim jobs, sim jobs and jobs after sim jobs). I would like to show two approaches; the distribution of time of the all the time used in the different phases and the critical path.

[mcastril]

I repeat here the comment I added in the original issue:

Concerning Total simulated years. Do I understand right that you consider only the time from the start of the simulation to the last completed chunk?

This is because we have this disclaimer "statement in the Performance view that the jobs displayed are not all the jobs run but only the last successful SIM of each CHUNK. This is to avoid misunderstandings"

Then I assume that we don't count repeated nor failed CHUNK, is it right?

The same applies to the Total energy consumed

Even so, I find the names, with the "Total" a little bit misleading because of this. With the ASYPD or SYPD we didn't have this issue because they were just averages.

What do you think about changing "Total simulated years" for "Years simulated" and "Total energy consumed" to "Consumed energy". It's only a detail. In addition to the statement (which I think it's a good idea), would it be possible to add a tooltip with some additional explanations?

For example, in "Years simulated", saying: "Amount of years from the Start Date to the last completed chunk". And, in "Consumed energy": Sum of energy consumed for all last successful SIM attempts of each CHUNK

[mcastril]

Actually, considering the information that you want to add... I think that we have to reconsider the Performance tab, because it only shows the latest successful SIM of each CHUNK. This was not to bad for showing averages, but since you are now focusing in totals I find that it can be very misleading for users.

I suggest:

1. Adding all the SIM retries in the SIM list
2. Leaving the SYPD, ASYPD, CHPSY empty in the FAILED ones (since we don't know the fraction of the chunk completed), but print the energy
3. Then, in total energy and footprint, consider all the jobs. This way we would have an idea of the total cost of the experiment.

@LuiggiTenorioK do you consider this is feasible?

If it's not feasible then I am not sure that we should add these total energy values in the Performance tab. We should find a better place.

[mcastril]

(to show the difference with common references i.g carbon footprint generated by 1kg of beef) and a table (to show the equivalence with common references i.g the total footprint of the experiment is equivalent to X kg of beef) f

@MissterP what do 1kg of beef stand for here? Do you mean the cost of growing the animal, cooking it, etc?

Maybe we can consider more "neutral" metrics like kms consumed by a diesel car or by a plane. To avoid also issues due to food preferences, religion, culture.... ?

[MissterP]

I repeat here the comment I added in the original issue:

Hi @mcastril!

"Concerning Total simulated years. Do I understand right that you consider only the time from the start of the simulation to the last completed chunk?" -> Yes. The value is calculated directly from the valid jobs (jobs considered).

"This is because we have this disclaimer "statement in the Performance view that the jobs displayed are not all the jobs run but only the last successful SIM of each CHUNK. This is to avoid misunderstandings"" -> This is indicated for all metrics in the considered jobs section.

"Then I assume that we don't count repeated nor failed CHUNK, is it right?" -> Yes.

"The same applies to the Total energy consumed" -> Yes. Like the time simulated it comes from the valid jobs.

"What do you think about changing "Total simulated years" for "Years simulated" " -> Cool! I can change it in this issue.

" "Total energy consumed" to "Consumed energy". " -> @LuiggiTenorioK had the same though so he suggested me to change it to "Energy Consumed".

"In addition to the statement (which I think it's a good idea), would it be possible to add a tooltip with some additional explanations?" -> I think so! Right now this is the information showed related to the energy.

"For example, in "Years simulated", saying: "Amount of years from the Start Date to the last completed chunk". And, in "Consumed energy": Sum of energy consumed for all last successful SIM attempts of each CHUNK" -> Cool!

[MissterP]

(to show the difference with common references i.g carbon footprint generated by 1kg of beef) and a table (to show the equivalence with common references i.g the total footprint of the experiment is equivalent to X kg of beef) f

@MissterP what do 1kg of beef stand for here? Do you mean the cost of growing the animal, cooking it, etc?

Maybe we can consider more "neutral" metrics like kms consumed by a diesel car or by a plane. To avoid also issues due to food preferences, religion, culture.... ?

The information about the footprint of 1 kg of beef comes from here. I didn't see how many processes are included. The example of the beef is just an example of reference to understand the motivation of the issue; it can be changed to another metrics to avoid the issues that you stated .

[mcastril]

The information about the footprint of 1 kg of beef comes from here. I didn't see how many processes are included. The example of the beef is just an example of reference to understand the motivation of the issue; it can be changed to another metrics to avoid the issues that you stated .

It just came to my mind. Mainly most of the measures there are targetted to Western people and we have to take into account this cultural facets.

Thanks

[LuiggiTenorioK]

I suggest:

1. Adding all the SIM retries in the SIM list
2. Leaving the SYPD, ASYPD, CHPSY empty in the FAILED ones (since we don't know the fraction of the chunk completed), but print the energy
3. Then, in total energy and footprint, consider all the jobs. This way we would have an idea of the total cost of the experiment.

@LuiggiTenorioK do you consider this is feasible?

I think we don't store data of the failed retrials since the job_data table is at run level. So, we only have the data of the latest retrial per run.

Maybe adding a disclaimer explaining that this total doesn't count the failed ones could work to give a lower bound of consumption/footprint to the users, which is still useful. If that is the case, we have to carefully explain it to avoid misunderstandings.

[LuiggiTenorioK]

• The plots used would be be from a new dependency called recharts. Moreover, to show a warning message in case there is no energy or footprint the dependencies antd and @ant-design/icons would be used.

I would avoid adding more dependencies to the project if it is not completely necessary. Is it possible you can achieve this using d3.js which is already included in the project?

Same with antd and @ant-design/icons. We have fontawesome for icons, and if you really need a specific icon, you can provide the SVG instead of the whole library.

[mcastril]

I suggest:

1. Adding all the SIM retries in the SIM list
2. Leaving the SYPD, ASYPD, CHPSY empty in the FAILED ones (since we don't know the fraction of the chunk completed), but print the energy
3. Then, in total energy and footprint, consider all the jobs. This way we would have an idea of the total cost of the experiment.

@LuiggiTenorioK do you consider this is feasible?

I think we don't store data of the failed retrials since the job_data table is at run level. So, we only have the data of the latest retrial per run.

Maybe adding a disclaimer explaining that this total doesn't count the failed ones could work to give a lower bound of consumption/footprint to the users, which is still useful. If that is the case, we have to carefully explain it to avoid misunderstandings.

How this works Luiggi?

I understand then that you talk about job runs, not experiment runs. Because one experiment run may have different retrials.

Then the disclaimer seems ok.

[LuiggiTenorioK]

How this works Luiggi?

I understand then that you talk about job runs, not experiment runs. Because one experiment run may have different retrials.

No, I was referring to the combination of jobs and experiment runs, which is what we have in the job_data table. Then, what I understood from the Autosubmit code about the retrials mechanism is that a job is resubmitted without creating a new experiment run.

Then, I assume that when a retrial is done, the job_data table is not updated (maybe @dbeltrankyl can confirm that) because the data is updated when the job change is status to a final status (COMPLETED, FAILED, UNKNOWN). This means we don't have information of failed retrials, just the latest retrial.

[dbeltrankyl]

I think we don't store data of the failed retrials since the job_data table is at run level. So, we only have the data of the latest retrial per run.

We store them

Then, I assume that when a retrial is done, the job_data table is not updated

It is

updated when the job change is status to a final status (COMPLETED, FAILED, UNKNOWN).

It is a new entry. Each time the job is in any final status it is updated in the job_data with a new row.

For example, in the integration test for failed jobs:

You can see 3 entries of the same job_name and the same run_id.

These rows are never updated once all timers are written

[LuiggiTenorioK]

@dbeltrankyl thanks for the clarification! It was hard to understand the retrial mechanics without a proper example.

Just to confirm, the counter column identifies the retrial number, right?

[dbeltrankyl]

Just to confirm, the counter column identifies the retrial number, right?

I wouldn't rely on that...

I think it only counts the number of times this job name has run but not the retrial number.

But there is the trick:

• Run_id is only updated on a new create + run ( without recovery )
• job fail_count is reset each time the run is stopped, so under the same run_id, there could be more than the retrials configured

The solution would be to post-process it and get the run_id, job_name, and job_id to determine the retrial number.

The job_id is to determine which one is the first retrial, second... ( higher == higher retrial number )

[MissterP]

@dbeltrankyl thank you for the explanation! I would like to ask when is accessed the information in job_data during the extraction of the valid SIM jobs of the Performance view, because I saw that only is accessed the pkl of an experiment; and the model has this information: