<font size=2 face="sans-serif">If you are interested in investigating
- the following wrapper may give you some information about the number
of types used.</font>
<br>
<br><font size=2 face="sans-serif">mpicc -c wrap_some.c</font>
<br><font size=2 face="sans-serif">ar -rcv libwrap.a wrap_some.o</font>
<br>
<br><font size=2 face="sans-serif">and then some experimenting on how to
get your libwrap.a into the linking after the Fortran MPI library, but
before the C implementations of it.</font>
<br>
<br><font size=2 face="sans-serif">It will print a summary at the end of
type commits/frees. And if you want intermediate values you can use MPI_Type_Stats_{commit,free,delta,peak}</font>
<br>
<br><font size=2 face="sans-serif">/Nils</font>
<br>
<br><font size=2 face="sans-serif">$ cat wrap_some.c </font>
<br><font size=2 face="sans-serif">#include "mpi.h"</font>
<br><font size=2 face="sans-serif">#include <stdio.h></font>
<br>
<br><font size=2 face="sans-serif">static long ncommit=0,nfree=0,ndelta=0,npeak=0;</font>
<br>
<br><font size=2 face="sans-serif">int MPI_Type_commit( MPI_Datatype *datatype
) {</font>
<br><font size=2 face="sans-serif"> ncommit++;</font>
<br><font size=2 face="sans-serif"> if(ndelta==npeak) npeak++;</font>
<br><font size=2 face="sans-serif"> ndelta++;</font>
<br><font size=2 face="sans-serif"> return PMPI_Type_commit(datatype);</font>
<br><font size=2 face="sans-serif">}</font>
<br>
<br><font size=2 face="sans-serif">int MPI_Type_free( MPI_Datatype *datatype
) {</font>
<br><font size=2 face="sans-serif"> nfree--;</font>
<br><font size=2 face="sans-serif"> ndelta--;</font>
<br><font size=2 face="sans-serif"> return PMPI_Type_free(datatype);</font>
<br><font size=2 face="sans-serif">}</font>
<br>
<br><font size=2 face="sans-serif">int MPI_Finalize(){</font>
<br><font size=2 face="sans-serif"> int myid;</font>
<br><font size=2 face="sans-serif"> MPI_Comm_rank(MPI_COMM_WORLD,&myid);</font>
<br><font size=2 face="sans-serif"> printf("Rank: %4d
Free: %8ld Commit: %8ld Balance: %8ld Peak: %8ld\n",</font>
<br><font size=2 face="sans-serif"> myid,nfree,ncommit,ndelta,npeak);</font>
<br><font size=2 face="sans-serif"> return PMPI_Finalize();</font>
<br><font size=2 face="sans-serif">}</font>
<br>
<br><font size=2 face="sans-serif">long MPI_Type_Stats_commit() {</font>
<br><font size=2 face="sans-serif"> return ncommit;</font>
<br><font size=2 face="sans-serif">}</font>
<br>
<br><font size=2 face="sans-serif">long MPI_Type_Stats_free() {</font>
<br><font size=2 face="sans-serif"> return nfree;</font>
<br><font size=2 face="sans-serif">}</font>
<br>
<br><font size=2 face="sans-serif">long MPI_Type_Stats_peak() {</font>
<br><font size=2 face="sans-serif"> return npeak;</font>
<br><font size=2 face="sans-serif">}</font>
<br>
<br><font size=2 face="sans-serif">long MPI_Type_Stats_delta() {</font>
<br><font size=2 face="sans-serif"> return ndelta;</font>
<br><font size=2 face="sans-serif">}</font>
<br><font size=2 face="sans-serif">______________________________________________<br>
Nils Smeds, IBM Deep Computing / World Wide Coordinated Tuning Team<br>
IT Specialist, Mobile phone: +46-70-793 2639<br>
Fax. +46-8-793 9523<br>
Mail address: IBM Sweden; Loc. 5-03; 164 92 Stockholm; SWEDEN</font>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Maxwell Kelley <kelley@giss.nasa.gov></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">Rob Latham <robl@mcs.anl.gov></font>
<br><font size=1 color=#5f5f5f face="sans-serif">Cc:
</font><font size=1 face="sans-serif">parallel-netcdf@lists.mcs.anl.gov</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">10/06/2010 11:11 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">Re: MPI_TYPE_MAX
limit using MPT</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Sent by:
</font><font size=1 face="sans-serif">parallel-netcdf-bounces@lists.mcs.anl.gov</font>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2><br>
FWIW, the same test using the blocking interface hit the MPI_TYPE_MAX <br>
limit at about the same point.<br>
<br>
On Wed, 6 Oct 2010, Maxwell Kelley wrote:<br>
<br>
><br>
> My test was indeed using the nonblocking interface; I could re-code
with the <br>
> blocking interface if you think that would shed some light. The same
test run <br>
> with mvapich2 didn't encounter any problem. The MPI_TYPE_MAX
issue is <br>
> mentioned here<br>
><br>
> </font></tt><a href="http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-February/006647.html"><tt><font size=2>http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-February/006647.html</font></tt></a><tt><font size=2><br>
><br>
> so perhaps it's not pnetcdf that is forgetting to free datatypes.<br>
><br>
> -Max<br>
><br>
> On Wed, 6 Oct 2010, Rob Latham wrote:<br>
><br>
>> On Wed, Oct 06, 2010 at 12:29:50PM -0400, Maxwell Kelley wrote:<br>
>>> Is this normal? Setting MPI_TYPE_MAX to 65536 simply
allowed more<br>
>>> I/O to be performed before the error appears. The limit is
reached<br>
>>> more quickly using more processors. Assuming that this
is a case of<br>
>>> types not being freed after use, should I just set this limit
high<br>
>>> enough that it will never be exceeded during a 12-hour batch
job?<br>
>> <br>
>> I wish we knew more about where the extra data types came from.<br>
>> <br>
>> I imagine there is some cost to setting MPI_TYPE_MAX to 2 billion.<br>
>> Hopefully, you can find a value that lets you complete your work
while<br>
>> I try to find the places where pnetcdf forgets to free datatypes.<br>
>> <br>
>> Are you still using the nonblocking interface?<br>
>> <br>
>> ==rob<br>
>> <br>
>> -- <br>
>> Rob Latham<br>
>> Mathematics and Computer Science Division<br>
>> Argonne National Lab, IL USA<br>
>> <br>
>> <br>
><br>
<br>
</font></tt>
<br><font size=2 face="sans-serif"><br>
<br>
Såvida annat inte anges ovan: / Unless stated otherwise above:<br>
IBM Svenska AB<br>
Organisationsnummer: 556026-6883<br>
Adress: 164 92 Stockholm<br>
</font>