Page 1 of 1

Different results with different environments and processors

Posted: Wed May 13, 2015 9:46 am
by cyy
Hello,

When run our Baltic Sea model on parallel, we have the same results with
1 to 32 processors but we get some discrepancies when we try to run it
with 64 processors. The temperature differences start on the coastal
areas, but propagate during the summer and are higher in the open sea
(higher than 20% during the summer)

Also when we run it on 2 different systems, but with the same compilers
and optimizations we have different results as well.

Do you know if this is normal or if there is something wrong with our
model definition?

Thank you for your help

Re: Different results with different environments and proces

Posted: Wed May 13, 2015 1:44 pm
by Katrijn
Hi cyy,

Which version of coherens are you using?

Re: Different results with different environments and proces

Posted: Fri May 15, 2015 8:35 am
by cyy
It is V2.7.
Katrijn wrote:Hi cyy,

Which version of coherens are you using?

Re: Different results with different environments and proces

Posted: Mon May 18, 2015 8:04 am
by steph
Hello,

Concerning the different results from 32 to 64 processors, what kind of partitioning do you use, I mean are the processors aligned or not ?

Concerning the discrepancies from one machine to another when using the same compilers and optimizations,
- do you run the code in parallel ?
- what is the magnitude of the differences ?

Cheers,

Stephanie

Re: Different results with different environments and proces

Posted: Tue May 19, 2015 9:22 am
by cyy
Hello Stephanie,

The processors are divided in 8 nodes of 8 processors and for the partitioning, nprocsx and nprocsy were let as 0 so the partition should have been done as 8*8. The difference also happened when I tried with 64 processors on my computer (which has only 4 physical processors). The size of the grid we are using for the model is 194*286.

Concerning the differences between the machines, they were obtained both with single processor and in parallel. For a 1 day simulation of the Baltic Sea, the differences of temperature were between -0.15 to 0.2 and the temperature was ranging from 0 to 10.4°C. I also did the same test with the flood3d test case but I did not get any differences.

I think the differences are first located on shallow areas and then spread to the open sea.

Thank you for your help

Re: Different results with different environments and proces

Posted: Wed May 20, 2015 1:47 pm
by steph
Hello,

There are at least two possible reasons to this problem. Firstly, it can stem from a problem with communications in MPI as implemented in COHERENS. Patrick has promised to run additional debugging to investigate this issue. A second possible cause is the way you impose temperature at the boundaries.

A "COHERENS users conference" should take place in Finland, if you wish to attend it to further discuss these issues with Patrick, please send an email to katrijn.baetens@naturalsciences.be

I hope this helps,

Stephanie

Re: Different results with different environments and proces

Posted: Thu May 21, 2015 9:09 am
by pluyten
Hi,

As I understand, the temperature discrepancies also occur in serial mode. The problem may be be related ot which type of grid you are using. The reason is that we has a similar problem for salinity in an estuary and we believe it is related to the use of a curvilinear grid.

Patrick

Re: Different results with different environments and proces

Posted: Mon May 25, 2015 6:27 am
by cyy
Hello,

Thank you, I think we will attend the COHERENS user conference.

For the temperature boundary conditions, we specify a temperature profile.

Yes, the discrepancies also occur in serial mode, between 2 working environments with the same compiler (gcc 4.8) and also between gcc and intel on the same environment.
We use a uniform rectangular grid in spherical coordinate and a uniform σ-grid on the vertical.

The problem might come from different reactions of compilers when confronted to uninitialised variables.

Do you think the results can still be used?

Thank you for your help.

Re: Different results with different environments and proces

Posted: Wed May 27, 2015 7:57 am
by steph
Hello,

Updates to the transport module are ongoing. They will soon be available in the new release of the code. We hope they will solve the problems you report.

Cheers,

Stephanie

Re: Different results with different environments and proces

Posted: Mon Jun 01, 2015 2:12 pm
by Katrijn
Hi cyy,

Version 2.8 is made available on the site, with some fixes and changes. Maybe a good idea to try it out and see if it improves? Please keep us updated!

bw,

Katrijn