Hi Sukanta,<br><br>I am not sure what may be the problem but you are right, 24GB is more than enough for your domain size. I have been running larger WRF domains in dmpar mode on various architectures including Cray for quite some time. For me, it suffices to have the following in my .bashrc<br>
<br>unset limits<br>export MP_STACK_SIZE=64000000<br><br>Are you using FPMPI? Is yes, what happens if you dont use the profiler?<br><br>Regards<br>Preeti<br><br><div class="gmail_quote">On Thu, Feb 9, 2012 at 7:55 PM, Sukanta Basu <span dir="ltr"><<a href="mailto:sukanta.basu@gmail.com">sukanta.basu@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Gus,<br>
<br>
The Cray has 4 nodes (each containing 8 cores, 24 GB RAM). The nodes<br>
are connected by gigE. I need to use dmpar option.<br>
<br>
Regards,<br>
Sukanta<br>
<div class="HOEnZb"><div class="h5"><br>
On Thu, Feb 9, 2012 at 8:59 AM, Gustavo Correa <<a href="mailto:gus@ldeo.columbia.edu">gus@ldeo.columbia.edu</a>> wrote:<br>
> Hi Basu<br>
><br>
> Sorry, I missed the 'dmpar' information.<br>
> I am not familiar to it, but I guess it is the Cray trick to make the CX1 machine<br>
> look like a standard distributed memory environment?<br>
> [As opposed to a full shared memory environment across all nodes,<br>
> which would be 'smpar', right?]<br>
><br>
> If 'dmpar' it is a standard distributed memory environment, I presume all that<br>
> I said before still holds. I would just try to set KMP_STACKSIZE to 16m or more on<br>
> all nodes, and run WRF again.<br>
><br>
> FYI, I had some issues compiling some models with Intel 12.0, and in other mailing<br>
> lists I saw people that had issues with version 12.1.<br>
> However, I compiled some models with Intel 11.1 correctly, but as I said before, not WRF.<br>
><br>
> BTW, we're cheap here. No funding for fancy machines, no Cray, no IBM, no SGI.<br>
> The top thing we can buy is a standard Linux cluster once in a while. :)<br>
><br>
> Good luck,<br>
> Gus Correa<br>
><br>
> On Feb 9, 2012, at 8:25 AM, Sukanta Basu wrote:<br>
><br>
>> Hi Gus,<br>
>><br>
>> Thanks for your email.<br>
>><br>
>> I am compiling WRF with dmpar option (distributed memory). WRF has a<br>
>> different option for hybrid openmp+mpi (they call it dmpar+smpar). To<br>
>> the best of my knowledge, openmp is not invoked.<br>
>><br>
>> I do understand the distinction between openmp and openmpi. Yesterday,<br>
>> I uninstalled mpich2 and installed openmpi. I compiled and ran wrf<br>
>> jobs. As I mentioned before, I faced different types of problems.<br>
>><br>
>> I have been using WRF on various clusters for ~6-7 years. I bought a<br>
>> Cray CX1 recently and trying to set it up myself for running WRF<br>
>> locally. Now, I am suspecting that there is some compatibility issues<br>
>> between WRF and the Intel Composer. I used to use Intel 11.1 compiler.<br>
>><br>
>> I will set KMP_STACKSIZE and re-run the simulations with wrf+mpich2+intel.<br>
>><br>
>> Best regards,<br>
>> Sukanta<br>
>><br>
>> On Thu, Feb 9, 2012 at 8:02 AM, Gustavo Correa <<a href="mailto:gus@ldeo.columbia.edu">gus@ldeo.columbia.edu</a>> wrote:<br>
>>> Hi Sukanta<br>
>>><br>
>>> Did you read the final part of my previous email about KMP_STACKSIZE?<br>
>>> This is how Intel calls the OpenMP threads stack size.<br>
>>> I think you misspelled that environment variable [it is not MP_STACKSIZE as your email says].<br>
>>><br>
>>> Did you compile WRF with OpenMP turned on and with the Intel compiler?<br>
>>> If you did, you certainly need to increase also the threads' stack size.<br>
>>><br>
>>> I had experiences similar to yours, with other models compiled with Intel ifort,<br>
>>> and OpenMP, i.e., unexplained segmentation faults, even though the stacksize was<br>
>>> set to unlimited.<br>
>>><br>
>>> Some time ago I posted this same solution in this mailing list to somebody<br>
>>> at LLNL or ANL, I think, who was having this type of problem as well.<br>
>>> It is common in hybrid MPI+OpenMP programs.<br>
>>><br>
>>> I would set KMP_STACKSIZE to 16m at least *on all nodes*, maybe in your .bashrc, or in the script that launches the job. I don't remember the syntax on top of my head,<br>
>>> but the MPICH2 mpiexec [hydra] probably has a way to export the environment variables<br>
>>> to all processes. Check 'man mpiexec'.<br>
>>> You must ensure that the environment variable is set *on all nodes*.<br>
>>><br>
>>> You may need more than 16m, depending on how fine a grid you are using.<br>
>>> In another model here I had to use 512m, but this also depends<br>
>>> on how much memory/RAM your nodes have available per core.<br>
>>> You could try increasing it step by step, say, doubling each time:<br>
>>> 16m, 32m, 64m, ...<br>
>>><br>
>>> Anyway, this is a guess based on what happened here.<br>
>>> There is no guarantee that it will work, although it may be worth trying it.<br>
>>> The problem you see may also be a bug in WRF, or an input/forcing file that is missing, etc.<br>
>>><br>
>>> I hope this helps,<br>
>>> Gus Correa<br>
>>><br>
>>> PS - Note: Just to avoid confusion with names.<br>
>>> OpenMP and OpenMPI [or Open MPI] are different things.<br>
>>> The former is the thread-based standard for parallelization:<br>
>>> <a href="http://openmp.org/wp/" target="_blank">http://openmp.org/wp/</a><br>
>>> The latter is another open source MPI, like MPICH2:<br>
>>> <a href="http://www.open-mpi.org/" target="_blank">http://www.open-mpi.org/</a><br>
>>><br>
>>><br>
>>> On Feb 8, 2012, at 10:33 PM, Sukanta Basu wrote:<br>
>>><br>
>>>> Hi Gus,<br>
>>>><br>
>>>> I tried setting the stack option in limits.conf. No change. I logged<br>
>>>> on to each nodes and checked that the ulimit is indeed unlimited.<br>
>>>><br>
>>>> I just installed openmpi and recompiled WRF. It now runs with any<br>
>>>> array sizes. However, I have a different problem. Now, one of the<br>
>>>> processes quits suddenly during the run (with a segmentation fault<br>
>>>> error). I think both the mpich2 and openmpi problems are somewhat<br>
>>>> related.<br>
>>>><br>
>>>> Best regards,<br>
>>>> Sukanta<br>
>>>><br>
>>>> On Wed, Feb 8, 2012 at 6:20 PM, Gustavo Correa <<a href="mailto:gus@ldeo.columbia.edu">gus@ldeo.columbia.edu</a>> wrote:<br>
>>>>> Hi Sukanta<br>
>>>>><br>
>>>>> Did you set the stacksize [not only memlock] to unlimited in<br>
>>>>> /etc/security/limits.conf on all nodes?<br>
>>>>><br>
>>>>> Not sure this will work, but you could try to run 'ulimit -s' and 'ulimit -l' via mpiexec, just to check:<br>
>>>>><br>
>>>>> mpiexec -prepend-rank -f hostfile -np 32 ulimit -s<br>
>>>>> mpiexec -prepend-rank -f hostfile -np 32 ulimit -l<br>
>>>>><br>
>>>>> Or just login to each node and check.<br>
>>>>><br>
>>>>> Also, if your WRF is compiled with OpenMP,<br>
>>>>> I think the Intel-specific environment variable for OMP_STACKSIZE is<br>
>>>>> KMP_STACKSIZE [not MP_STACKSIZE], although they should also accept<br>
>>>>> the portable/standard OMP_STACKSIZE [but I don't know if they do].<br>
>>>>> For some models here I had to make is as big as 512m [I don't run wrf, though].<br>
>>>>> 'man ifort' should tell more about it [at the end of the man page].<br>
>>>>><br>
>>>>> I hope this helps,<br>
>>>>> Gus Correa<br>
>>>>><br>
>>>>> On Feb 8, 2012, at 4:23 PM, Anthony Chan wrote:<br>
>>>>><br>
>>>>>><br>
>>>>>> There is fpi, Fortran counterpart of cpi, you can try that.<br>
>>>>>> Also, there is MPICH2 testsuite which is located in<br>
>>>>>> mpich2-xxx/test/mpi can be invoked by "make testing".<br>
>>>>>> It is unlikely those tests will reveal anything.<br>
>>>>>> The testsuite is meant to test the MPI implementation<br>
>>>>>> not your app.<br>
>>>>>><br>
>>>>>> As what you said earlier, your difficulty in running WRF<br>
>>>>>> with larger dataset is memory related. You should contact WRF<br>
>>>>>> emailing list for more pointers.<br>
>>>>>><br>
>>>>>> ----- Original Message -----<br>
>>>>>>> Hi Anthony,<br>
>>>>>>><br>
>>>>>>> Is there any other mpi example code (other than cpi.c) that I could<br>
>>>>>>> test which will give me more information about my mpich setup?<br>
>>>>>>><br>
>>>>>>> Here is the output from cpi (using 32 cores on 4 nodes):<br>
>>>>>>><br>
>>>>>>> mpiuser@crayN1-5150jo:~/Misc$ mpiexec -f mpd.hosts -n 32 ./cpi<br>
>>>>>>> Process 1 on crayN1-5150jo<br>
>>>>>>> Process 18 on crayN2-5150jo<br>
>>>>>>> Process 2 on crayN2-5150jo<br>
>>>>>>> Process 26 on crayN2-5150jo<br>
>>>>>>> Process 5 on crayN1-5150jo<br>
>>>>>>> Process 14 on crayN2-5150jo<br>
>>>>>>> Process 21 on crayN1-5150jo<br>
>>>>>>> Process 22 on crayN2-5150jo<br>
>>>>>>> Process 25 on crayN1-5150jo<br>
>>>>>>> Process 6 on crayN2-5150jo<br>
>>>>>>> Process 9 on crayN1-5150jo<br>
>>>>>>> Process 17 on crayN1-5150jo<br>
>>>>>>> Process 30 on crayN2-5150jo<br>
>>>>>>> Process 10 on crayN2-5150jo<br>
>>>>>>> Process 29 on crayN1-5150jo<br>
>>>>>>> Process 13 on crayN1-5150jo<br>
>>>>>>> Process 8 on crayN3-5150jo<br>
>>>>>>> Process 20 on crayN3-5150jo<br>
>>>>>>> Process 4 on crayN3-5150jo<br>
>>>>>>> Process 12 on crayN3-5150jo<br>
>>>>>>> Process 0 on crayN3-5150jo<br>
>>>>>>> Process 24 on crayN3-5150jo<br>
>>>>>>> Process 16 on crayN3-5150jo<br>
>>>>>>> Process 28 on crayN3-5150jo<br>
>>>>>>> Process 3 on crayN4-5150jo<br>
>>>>>>> Process 7 on crayN4-5150jo<br>
>>>>>>> Process 11 on crayN4-5150jo<br>
>>>>>>> Process 23 on crayN4-5150jo<br>
>>>>>>> Process 27 on crayN4-5150jo<br>
>>>>>>> Process 31 on crayN4-5150jo<br>
>>>>>>> Process 19 on crayN4-5150jo<br>
>>>>>>> Process 15 on crayN4-5150jo<br>
>>>>>>> pi is approximately 3.1416009869231249, Error is 0.0000083333333318<br>
>>>>>>> wall clock time = 0.009401<br>
>>>>>>><br>
>>>>>>> Best regards,<br>
>>>>>>> Sukanta<br>
>>>>>>><br>
>>>>>>> On Wed, Feb 8, 2012 at 1:19 PM, Anthony Chan <<a href="mailto:chan@mcs.anl.gov">chan@mcs.anl.gov</a>> wrote:<br>
>>>>>>>><br>
>>>>>>>> Hmm.. Not sure what is happening.. I don't see anything<br>
>>>>>>>> obviously wrong in your mpiexec verbose output (though<br>
>>>>>>>> I am not hydra expert). Your code now is killed because of<br>
>>>>>>>> segmentation fault. Naively, I would recompile WRF with -g<br>
>>>>>>>> and use a debugger to see where segfault is. If you don't want<br>
>>>>>>>> to mess around WRF source code, you may want to contact WRF<br>
>>>>>>>> developers to see if they have encountered similar problem<br>
>>>>>>>> before.<br>
>>>>>>>><br>
>>>>>>>> ----- Original Message -----<br>
>>>>>>>>> Dear Anthony,<br>
>>>>>>>>><br>
>>>>>>>>> Thanks for your response. Yes, I did try MP_STACK_SIZE and<br>
>>>>>>>>> OMP_STACKSIZE. The error is still there. I have attached a log file<br>
>>>>>>>>> (I<br>
>>>>>>>>> ran mpiexec with -verbose option). May be this will help.<br>
>>>>>>>>><br>
>>>>>>>>> Best regards,<br>
>>>>>>>>> Sukanta<br>
>>>>>>>>><br>
>>>>>>>>> On Tue, Feb 7, 2012 at 3:28 PM, Anthony Chan <<a href="mailto:chan@mcs.anl.gov">chan@mcs.anl.gov</a>><br>
>>>>>>>>> wrote:<br>
>>>>>>>>>><br>
>>>>>>>>>> I am not familar with WRF, and not sure if WRF uses any thread<br>
>>>>>>>>>> in dmpar mode. Did you try setting MP_STACK_SIZE or OMP_STACKSIZE<br>
>>>>>>>>>> ?<br>
>>>>>>>>>><br>
>>>>>>>>>> see: <a href="http://forum.wrfforum.com/viewtopic.php?f=6&t=255" target="_blank">http://forum.wrfforum.com/viewtopic.php?f=6&t=255</a><br>
>>>>>>>>>><br>
>>>>>>>>>> A.Chan<br>
>>>>>>>>>><br>
>>>>>>>>>> ----- Original Message -----<br>
>>>>>>>>>>> Hi,<br>
>>>>>>>>>>><br>
>>>>>>>>>>> I am using a small cluster of 4 nodes (each with 8 cores + 24 GB<br>
>>>>>>>>>>> RAM).<br>
>>>>>>>>>>> OS: Ubuntu 11.10. The cluster uses nfs file system and gigE<br>
>>>>>>>>>>> connections.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> I installed mpich2 and ran cpi.c program successfully.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> I installed WRF (<a href="http://www.wrf-model.org/index.php" target="_blank">http://www.wrf-model.org/index.php</a>) using the<br>
>>>>>>>>>>> intel<br>
>>>>>>>>>>> compilers (dmpar option)<br>
>>>>>>>>>>> I set ulimit -l and -s to be unlimited in .bashrc (all nodes)<br>
>>>>>>>>>>> I set memlock to be unlimited in limits.conf (all nodes)<br>
>>>>>>>>>>> I have password-less ssh (public key sharing) on all the nodes<br>
>>>>>>>>>>> I ran parallel jobs with 40x40x40, 40x40x50, and 40x40x60 grid<br>
>>>>>>>>>>> points<br>
>>>>>>>>>>> successfully. However, when I utilize 40x40x80 grid points, I<br>
>>>>>>>>>>> get<br>
>>>>>>>>>>> the<br>
>>>>>>>>>>> following MPI error:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> **********************************************************<br>
>>>>>>>>>>> Fatal error in PMPI_Wait: Other MPI error, error stack:<br>
>>>>>>>>>>> PMPI_Wait(183)............: MPI_Wait(request=0x34e83a4,<br>
>>>>>>>>>>> status=0x7fff7b24c400) failed<br>
>>>>>>>>>>> MPIR_Wait_impl(77)........:<br>
>>>>>>>>>>> dequeue_and_set_error(596): Communication error with rank 8<br>
>>>>>>>>>>> **********************************************************<br>
>>>>>>>>>>> Given that I can run the exact simulation with slightly lesser<br>
>>>>>>>>>>> number<br>
>>>>>>>>>>> of grid points without any problem, this error is related to<br>
>>>>>>>>>>> stack<br>
>>>>>>>>>>> size. What could be the problem?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Thanks,<br>
>>>>>>>>>>> Sukanta<br>
>>>>>>>>>>><br>
>>>>>>>>>>> --<br>
>>>>>>>>>>> Sukanta Basu<br>
>>>>>>>>>>> Associate Professor<br>
>>>>>>>>>>> North Carolina State University<br>
>>>>>>>>>>> <a href="http://www4.ncsu.edu/%7Esbasu5/" target="_blank">http://www4.ncsu.edu/~sbasu5/</a><br>
>>>>>>>>>>> _______________________________________________<br>
>>>>>>>>>>> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
>>>>>>>>>>> To manage subscription options or unsubscribe:<br>
>>>>>>>>>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
>>>>>>>>>> _______________________________________________<br>
>>>>>>>>>> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
>>>>>>>>>> To manage subscription options or unsubscribe:<br>
>>>>>>>>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> --<br>
>>>>>>>>> Sukanta Basu<br>
>>>>>>>>> Associate Professor<br>
>>>>>>>>> North Carolina State University<br>
>>>>>>>>> <a href="http://www4.ncsu.edu/%7Esbasu5/" target="_blank">http://www4.ncsu.edu/~sbasu5/</a><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> --<br>
>>>>>>> Sukanta Basu<br>
>>>>>>> Associate Professor<br>
>>>>>>> North Carolina State University<br>
>>>>>>> <a href="http://www4.ncsu.edu/%7Esbasu5/" target="_blank">http://www4.ncsu.edu/~sbasu5/</a><br>
>>>>>> _______________________________________________<br>
>>>>>> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
>>>>>> To manage subscription options or unsubscribe:<br>
>>>>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
>>>>> To manage subscription options or unsubscribe:<br>
>>>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>> --<br>
>>>> Sukanta Basu<br>
>>>> Associate Professor<br>
>>>> North Carolina State University<br>
>>>> <a href="http://www4.ncsu.edu/%7Esbasu5/" target="_blank">http://www4.ncsu.edu/~sbasu5/</a><br>
>>>> _______________________________________________<br>
>>>> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
>>>> To manage subscription options or unsubscribe:<br>
>>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
>>><br>
>>> _______________________________________________<br>
>>> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
>>> To manage subscription options or unsubscribe:<br>
>>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
>><br>
>><br>
>><br>
>> --<br>
>> Sukanta Basu<br>
>> Associate Professor<br>
>> North Carolina State University<br>
>> <a href="http://www4.ncsu.edu/%7Esbasu5/" target="_blank">http://www4.ncsu.edu/~sbasu5/</a><br>
>> _______________________________________________<br>
>> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
>> To manage subscription options or unsubscribe:<br>
>> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
><br>
> _______________________________________________<br>
> mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
> To manage subscription options or unsubscribe:<br>
> <a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
<br>
<br>
<br>
--<br>
Sukanta Basu<br>
Associate Professor<br>
North Carolina State University<br>
<a href="http://www4.ncsu.edu/%7Esbasu5/" target="_blank">http://www4.ncsu.edu/~sbasu5/</a><br>
_______________________________________________<br>
mpich-discuss mailing list <a href="mailto:mpich-discuss@mcs.anl.gov">mpich-discuss@mcs.anl.gov</a><br>
To manage subscription options or unsubscribe:<br>
<a href="https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss" target="_blank">https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss</a><br>
<br>
--<br>
This message has been scanned for viruses and<br>
dangerous content by MailScanner, and is<br>
believed to be clean.<br>
<br>
</div></div></blockquote></div><br>