A very expensive part of the chemistry is the solver (ASAD_SPIMPMJP which calls SPINSLV2 and SPFULJAC). How many times it is called depends on the IREDO variable which is set in ASAD_SPMJPDRIV. IREDO starts at 1 and doubles until convergence is met or we reach 128, when it gives up. Hence, if IREDO = 16 then 16+8+4+2+1=31 calculation steps have been made to get here. If we knew that 16 calculation steps were needed, we could have saved 15 calculation steps which aren't used.
The code is massively imbalanced, such as chemistry happening more in lit areas. The code is only really as fast as it's slowest PE, because of all the MPI barriers in the code, so speeding up the code requires speeding up the slowest PEs. If we slow down the faster PEs in the process, this does not matter provided they are no slower than the slowest PEs.
On this page, I want to see if there's anything we can do to reduce the calculation steps of slowest PEs.
If we could assume that
then we could set IREDO to be the maximum calculation steps per level for all PEs and so cut down on the calculations steps which aren't used. This could greatly slow down the fastest PEs, but the main aim to speed up the slowest PEs.
See if the assumption above are valid if run a copy of mi-aj670, mi-aq835, for one month and outputed the values of IREDO for every timestep and every level. If we assume that each calculation step takes about the same amount of time, the slowest PE for each timestep will be the PE completing the most calculation steps. The top plot below shows log2(IREDO) for the slowest PE for chemistry timestep (0-7199) versus level (0-84). The red points around level fifty near the end of the run show where there was no convergence (IREDO=128, and log2(IREDO)=7). The plot shows that IREDO=1 (log2(IREDO)=0) is usually sufficient for convergence, except around level 40 and towards the surface.
The bottom plot shows the maximum calculation steps at every timestep at every level across all PEs. Showing again that is around level 40 where the most calculation steps are needed.
By taking the slowest PE each time and calculating how many calculation steps it required for all the levels, I found the calculation steps for one month was 87,490.
If we thought we could ancipate the maximum IREDO needed for each level and just this use this across all PEs, the calculation steps would this one month would be 99,163. This is slower and so my assumption are not sufficiently correct to begin using knowledge of the maximum IREDO per level to speed-up the solver.
It should be possible to use this information to be begin IREDO at a greater value than 1 around the 40th level, but it's unlikely to save much time for both the effort involved in coding and the extra complication it adds.
