Different results with different environments and processors

Questions related to the implemented of numerical techniques, or if you have ideas to implement and test new numerical techniques should be posted here.

Moderator: mumm-sebastien

cyy
Posts: 5
Joined: Wed Apr 29, 2015 11:59 am
Full name: Yuanying Chen

Different results with different environments and processors

Post by cyy »

Hello,

When run our Baltic Sea model on parallel, we have the same results with
1 to 32 processors but we get some discrepancies when we try to run it
with 64 processors. The temperature differences start on the coastal
areas, but propagate during the summer and are higher in the open sea
(higher than 20% during the summer)

Also when we run it on 2 different systems, but with the same compilers
and optimizations we have different results as well.

Do you know if this is normal or if there is something wrong with our
model definition?

Thank you for your help
Katrijn
Posts: 87
Joined: Fri Nov 29, 2013 3:04 pm
Full name: Katrijn Baetens

Re: Different results with different environments and proces

Post by Katrijn »

Hi cyy,

Which version of coherens are you using?
cyy
Posts: 5
Joined: Wed Apr 29, 2015 11:59 am
Full name: Yuanying Chen

Re: Different results with different environments and proces

Post by cyy »

It is V2.7.
Katrijn wrote:Hi cyy,

Which version of coherens are you using?
steph
Posts: 6
Joined: Thu Oct 09, 2014 11:11 am
Full name: Stephanie Ponsar

Re: Different results with different environments and proces

Post by steph »

Hello,

Concerning the different results from 32 to 64 processors, what kind of partitioning do you use, I mean are the processors aligned or not ?

Concerning the discrepancies from one machine to another when using the same compilers and optimizations,
- do you run the code in parallel ?
- what is the magnitude of the differences ?

Cheers,

Stephanie
cyy
Posts: 5
Joined: Wed Apr 29, 2015 11:59 am
Full name: Yuanying Chen

Re: Different results with different environments and proces

Post by cyy »

Hello Stephanie,

The processors are divided in 8 nodes of 8 processors and for the partitioning, nprocsx and nprocsy were let as 0 so the partition should have been done as 8*8. The difference also happened when I tried with 64 processors on my computer (which has only 4 physical processors). The size of the grid we are using for the model is 194*286.

Concerning the differences between the machines, they were obtained both with single processor and in parallel. For a 1 day simulation of the Baltic Sea, the differences of temperature were between -0.15 to 0.2 and the temperature was ranging from 0 to 10.4°C. I also did the same test with the flood3d test case but I did not get any differences.

I think the differences are first located on shallow areas and then spread to the open sea.

Thank you for your help
steph
Posts: 6
Joined: Thu Oct 09, 2014 11:11 am
Full name: Stephanie Ponsar

Re: Different results with different environments and proces

Post by steph »

Hello,

There are at least two possible reasons to this problem. Firstly, it can stem from a problem with communications in MPI as implemented in COHERENS. Patrick has promised to run additional debugging to investigate this issue. A second possible cause is the way you impose temperature at the boundaries.

A "COHERENS users conference" should take place in Finland, if you wish to attend it to further discuss these issues with Patrick, please send an email to katrijn.baetens@naturalsciences.be

I hope this helps,

Stephanie
pluyten
Posts: 6
Joined: Wed Aug 13, 2014 2:38 pm
Full name: Patrick Luyten

Re: Different results with different environments and proces

Post by pluyten »

Hi,

As I understand, the temperature discrepancies also occur in serial mode. The problem may be be related ot which type of grid you are using. The reason is that we has a similar problem for salinity in an estuary and we believe it is related to the use of a curvilinear grid.

Patrick
cyy
Posts: 5
Joined: Wed Apr 29, 2015 11:59 am
Full name: Yuanying Chen

Re: Different results with different environments and proces

Post by cyy »

Hello,

Thank you, I think we will attend the COHERENS user conference.

For the temperature boundary conditions, we specify a temperature profile.

Yes, the discrepancies also occur in serial mode, between 2 working environments with the same compiler (gcc 4.8) and also between gcc and intel on the same environment.
We use a uniform rectangular grid in spherical coordinate and a uniform σ-grid on the vertical.

The problem might come from different reactions of compilers when confronted to uninitialised variables.

Do you think the results can still be used?

Thank you for your help.
steph
Posts: 6
Joined: Thu Oct 09, 2014 11:11 am
Full name: Stephanie Ponsar

Re: Different results with different environments and proces

Post by steph »

Hello,

Updates to the transport module are ongoing. They will soon be available in the new release of the code. We hope they will solve the problems you report.

Cheers,

Stephanie
Katrijn
Posts: 87
Joined: Fri Nov 29, 2013 3:04 pm
Full name: Katrijn Baetens

Re: Different results with different environments and proces

Post by Katrijn »

Hi cyy,

Version 2.8 is made available on the site, with some fixes and changes. Maybe a good idea to try it out and see if it improves? Please keep us updated!

bw,

Katrijn
Post Reply