Different results with different environments and processors

[cyy]

Hello,

When run our Baltic Sea model on parallel, we have the same results with
1 to 32 processors but we get some discrepancies when we try to run it
with 64 processors. The temperature differences start on the coastal
areas, but propagate during the summer and are higher in the open sea
(higher than 20% during the summer)

Also when we run it on 2 different systems, but with the same compilers
and optimizations we have different results as well.

Do you know if this is normal or if there is something wrong with our
model definition?

Thank you for your help

[Katrijn]

Hi cyy,

Which version of coherens are you using?

[cyy]

It is V2.7.

Hi cyy,

Which version of coherens are you using?

[steph]

Hello,

Concerning the different results from 32 to 64 processors, what kind of partitioning do you use, I mean are the processors aligned or not ?

Concerning the discrepancies from one machine to another when using the same compilers and optimizations,
- do you run the code in parallel ?
- what is the magnitude of the differences ?

Cheers,

Stephanie

[cyy]

Hello Stephanie,

The processors are divided in 8 nodes of 8 processors and for the partitioning, nprocsx and nprocsy were let as 0 so the partition should have been done as 8*8. The difference also happened when I tried with 64 processors on my computer (which has only 4 physical processors). The size of the grid we are using for the model is 194*286.

Concerning the differences between the machines, they were obtained both with single processor and in parallel. For a 1 day simulation of the Baltic Sea, the differences of temperature were between -0.15 to 0.2 and the temperature was ranging from 0 to 10.4°C. I also did the same test with the flood3d test case but I did not get any differences.

I think the differences are first located on shallow areas and then spread to the open sea.

Thank you for your help

[steph]

Hello,

There are at least two possible reasons to this problem. Firstly, it can stem from a problem with communications in MPI as implemented in COHERENS. Patrick has promised to run additional debugging to investigate this issue. A second possible cause is the way you impose temperature at the boundaries.

A "COHERENS users conference" should take place in Finland, if you wish to attend it to further discuss these issues with Patrick, please send an email to katrijn.baetens@naturalsciences.be

I hope this helps,

Stephanie

[pluyten]

Hi,

As I understand, the temperature discrepancies also occur in serial mode. The problem may be be related ot which type of grid you are using. The reason is that we has a similar problem for salinity in an estuary and we believe it is related to the use of a curvilinear grid.

Patrick

[cyy]

Hello,

Thank you, I think we will attend the COHERENS user conference.

For the temperature boundary conditions, we specify a temperature profile.

Yes, the discrepancies also occur in serial mode, between 2 working environments with the same compiler (gcc 4.8) and also between gcc and intel on the same environment.
We use a uniform rectangular grid in spherical coordinate and a uniform σ-grid on the vertical.

The problem might come from different reactions of compilers when confronted to uninitialised variables.

Do you think the results can still be used?

Thank you for your help.

[steph]

Hello,

Updates to the transport module are ongoing. They will soon be available in the new release of the code. We hope they will solve the problems you report.

Cheers,

Stephanie

[Katrijn]

Hi cyy,

Version 2.8 is made available on the site, with some fixes and changes. Maybe a good idea to try it out and see if it improves? Please keep us updated!

bw,

Katrijn