[MOAB-dev] Fwd: Re: Job exiting early [ThermHydraX]

Grindeanu, Iulian R. iulian at mcs.anl.gov
Wed May 21 10:32:02 CDT 2014


I am looking at this problem;
it is a memory leak somewhere, in exchange tags. After 200000 iterations even a few bytes per iteration is enough to blow up.
is shows on my laptop that the memory usage is increasing, with number of iterations :(
Actually, after 300K iterations, memory is almost 1Gb higher per process. 
1Gb/300K ~= 300Kb 
I will run valgrind and let you know.

....
//BEGIN ITER
  for (int iter = 1; iter <= 1000000; iter++){
        if(rank ==0 && iter%100==1) std::cout << "---ITER " << iter << " ---" << std::endl;
    
        result = pcomm->exchange_tags(celltagsn, celltagsn, ents);
        PRINT_LAST_ERROR;

        result = pcomm->exchange_tags(celltags, celltags, ents);
        PRINT_LAST_ERROR; 

        result = pcomm->exchange_tags(facetags, facetags, adjs);
        PRINT_LAST_ERROR;

} //END ITER
________________________________________
From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf of Tim Tautges [timothy.tautges at cd-adapco.com]
Sent: Wednesday, May 21, 2014 10:11 AM
To: moab-dev at mcs.anl.gov
Subject: Re: [MOAB-dev] Fwd: Re: Job exiting early [ThermHydraX]

IIRC, there's an option to wait at a barrier where appropriate.  All
sends/receives should have matches, so if it doesn't make it through a
barrier something is wrong.  There's no algorithm in there that has
non-matching sends/receives (i.e. no polling for random messages).

- tim

On 05/21/2014 09:57 AM, Grindeanu, Iulian R. wrote:
> Anton's issue is with doing exchange tags repeatedly; it seems to be
> mismatched isend/ireceives; as they accumulate, they run the memory out.
> Or something else is happening with memory, maybe we need to do some
> flushes of the buffers.
> Jane's issue is different; assign ids might have problems.
> All these should deserve tickets.
> Iulian
> ------------------------------------------------------------------------
> *From:* Vijay S. Mahadevan [vijay.m at gmail.com]
> *Sent:* Wednesday, May 21, 2014 9:48 AM
> *To:* Jiangtao Hu
> *Cc:* Grindeanu, Iulian R.; kanaev at ibrae.ac.ru; MOAB dev
> *Subject:* Re: [MOAB-dev] Fwd: Re: Job exiting early [ThermHydraX]
>
> Jane,
>
> //              Global Ids
>    //   2.0   4 ------- 3 -------- 6 -------- 10 --------- 14
>    //          |         |          |          |                 |
>    //          |         |          |          |                 |
>    //          |         |          |          |                 |
>    //   1.0   1 ------- 2 -------- 5 -------- 9 --------- 13
>    //        0.0       1.0        2.0        3.0          4.0
>
> This is certainly a bug and should not happen to start with. If you have
> this test case available, do send it to the list so that we can find out
> the actual reason for the misnumber. Anton's issue might or not be
> directly related to this bug but his reference to GID being
> discontinuous shows that there is an outstanding issue here.
>
> Vijay
>
> On Wed, May 21, 2014 at 8:06 PM, Jiangtao Hu <jiangtao_ma at yahoo.com
> <mailto:jiangtao_ma at yahoo.com>> wrote:
>
>     On unit test for my project, I am trying to use
>     ParallelComm::assign_global_id() to get the global ids for vertices
>     of a simple grid
>        //              Mesh Ids
>        //   2.0   6 ------- 7 -------- 8 -------- 9 --------- 10
>        //          |        |          |          |                 |
>        //          |    1  |    2    |     3    |      4         |
>        //          |        |          |          |                 |
>        //   1.0   1 ------- 2 -------- 3 -------- 4 --------- 5
>        //        0.0       1.0        2.0        3.0          4.0
>
>     and got global ids as following
>     //              Global Ids
>        //   2.0   4 ------- 3 -------- 6 -------- 10 --------- 14
>        //          |         |          |          |                 |
>        //          |         |          |          |                 |
>        //          |         |          |          |                 |
>        //   1.0   1 ------- 2 -------- 5 -------- 9 --------- 13
>        //        0.0       1.0        2.0        3.0          4.0
>
>     Don't know if this is what Anton got through.
>
>     Iulian, if you are interested to see the test, please let me know,
>     and I'll send to you.
>     Jane
>
>
>     Asst. Researcher
>     Dept. of Engineering Physics
>     UW @ Madison
>
>
>     "And we know that for those who love God, that is, for those who are
>     called according to his purpose, all things are working together for
>     good." (Romans 8:28)
>     On Tuesday, May 20, 2014 5:36 PM, "Grindeanu, Iulian R."
>     <iulian at mcs.anl.gov <mailto:iulian at mcs.anl.gov>> wrote:
>
>
>     hmmm,
>     this is not good :(
>
>     Are you running this on mira? Do you have a small file for a
>     laptop/workstation?
>     Maybe I can create one similar.
>
>     Do you see this only on 1024 processes or can it be lower count?
>
>     How does your model look like?
>     Any processor should not communicate with more than 64 other
>     processes, maybe after ghosting this number is reached.
>
>     Can you run a debug version of this ? maybe some asserts are not
>     triggered in optimized mode.
>
>     Is your file somewhere on mira I can get to it?
>
>     Iulian
>
>     ------------------------------------------------------------------------
>     *From:* moab-dev-bounces at mcs.anl.gov
>     <mailto:moab-dev-bounces at mcs.anl.gov> [moab-dev-bounces at mcs.anl.gov
>     <mailto:moab-dev-bounces at mcs.anl.gov>] on behalf of
>     kanaev at ibrae.ac.ru <mailto:kanaev at ibrae.ac.ru> [kanaev at ibrae.ac.ru
>     <mailto:kanaev at ibrae.ac.ru>]
>     *Sent:* Tuesday, May 20, 2014 5:05 PM
>     *To:* MOAB dev
>     *Subject:* Re: [MOAB-dev] Fwd: Re: Job exiting early [ThermHydraX]
>
>     The problem is still here
>     I've made a simple program performing certain numbers of
>     exchange_tags calls within a loop
>     If you run it on several processors with any mesh file it will
>     eventually crash with the following message from every core:
>     Fatal error in PMPI_Isend: Internal MPI error!, error stack:
>     PMPI_Isend(148): MPI_Isend(buf=0xd0f300, count=4, MPI_UNSIGNED_CHAR,
>     dest=1, tag=6, MPI_COMM_WORLD, request=0xcde354) failed
>     (unknown)(): Internal MPI error!
>     Thanks
>     Anton
>
>     On Tue, 20 May 2014 04:40:03 -0400, wrote:
>
>         Please disregard that, the global_id space for Quads was
>         incontinuous in my mesh file
>         Will check back with correct mesh
>         Anton
>
>         On Mon, 19 May 2014 15:17:14 -0400, wrote:
>
>             Hello MOAB dev,
>             I've attached a simplified version of my program that
>             crashes presumably after particular numbers calls of
>             exchange_tags
>             I ran it couple of times on Mira on 1024 cores (64 nodes in
>             --mode c16)
>             It performs around 524378 iterations and then crushes (error
>             file attached)
>             Can you please take a look at what Scott Parker from ALCF
>             suggests about it:
>             -------- Original Message --------
>             Subject:  Re: Job exiting early [ThermHydraX]
>             Date:     Fri, 9 May 2014 18:48:25 -0500
>             From:     Scott Parker
>             To:
>
>
>
>             Anton-
>
>             I took a look at the core files and from the stack trace it
>             appears that the code is failing in an MPI_Isend call
>             that is called from moab.ParallelComm::recv_buffer which is
>             called from moab::ParallelComm::exchange_tags
>             called from main(). The stack looks like:
>
>                main
>                  moab::ParallelComm::exchange_tags
>                    moab.ParallelComm::recv_buffer
>                      MPI_Isend
>
>             I've been able to get the same failure and error message you
>             are seeing by having an MPI process call MPI_Isend
>             when there are no matching receives. After 2 million Isends
>             the program exits with the error you are seeing. So
>             I'm pretty sure your ending up with a large number of
>             outstanding requests and the program is failing because
>             it can't allocate space for new MPI_Request objects.
>
>             I'd suggest looking at how Moab is calling MPI and how many
>             requests might be outstanding at any one time.
>             Since the code is running for 5 hours and looks to be
>             executing hundreds of thousands of iterations I wonder
>             if there is some sort of send-receive mismatch that is
>             letting requests accumulate. I think your best bet is to
>             talk to the Moab folks and see if they have any ideas about
>             why this might be happening.
>
>             One possibility is a load imbalance between processes - if
>             you don't have any MPI_Barriers or other collectives in
>             your code you could try adding a barrier to synchronize the
>             processes.
>
>             If the Moab guys can't help you and adding a barrier doesn't
>             help I can work with you to instrument the code to
>             collect more information on how MPI is being called and we
>             could possibly pin down the source of the problem
>             that way.
>
>
>             Scott
>
>
>             On 5/2/14, 11:14 PM, kanaev at ibrae.ac.ru
>             <mailto:kanaev at ibrae.ac.ru> wrote:
>
>                 Hello Scott
>                 The dir is cd
>                 /gpfs/mira-fs0/projects/ThermHydraX/kanaev/pinit
>                 The run produced core files is  253035
>                 I took another run with the line
>                   MPI_Allreduce(&r,&rmax,1,MPI_REAL8,MPI_MAX,MPI_COMM_WORLD); commented and it stopped the same very time iteration #524378,  just passed some more lines
>                 I use MOAB library and its function of exchanging data
>                 between processors so i think i cannot really count MPI
>                 requests
>                 Anton
>
>                 On Mon, 28 Apr 2014 16:45:41 -0500, Scott Parker wrote:
>
>
>                     Can you point me to the directory where your binary
>                     and core files are?
>
>                     The stack trace you sent shows a call to
>                     MPI_Waitany, do you know how many MPI requests
>                     the code generally has outstanding at any time?
>
>                     -Scott
>
>                     On 4/28/14, 4:30 PM, kanaev at ibrae.ac.ru
>                     <mailto:kanaev at ibrae.ac.ru> wrote:
>
>                         Hello Scott,
>                         I took rerun with the mentioned keys. The code
>                         was freshly compiled with makefile attached just
>                         in case.
>                         I've got 1024 core files. Two of them are attached.
>                         I run bgq_stack for core.0 and here's what i got:
>                           [akanaev at miralac1 pinit]$bgq_stack pinit core.0
>                         ------------------------------------------------------------------------
>
>                         Program   :
>                         /gpfs/mira-fs0/projects/ThermHydraX/kanaev/pinit/pinit
>
>                         ------------------------------------------------------------------------
>
>                         +++ID Rank: 0, TGID: 1, Core: 0, HWTID:0 TID: 1
>                         State: RUN
>                         00000000018334c0
>                         _ZN4PAMI6Device2MU7Factory12advance_implEPNS1_7ContextEmm
>
>                         /bgsys/source/srcV1R2M1.17463/comm/sys/buildtools/pami/components/devices/bgq/mu2/Factory.h:269
>
>                         000000000170da28
>                         PAMI_Context_trylock_advancev
>                         /bgsys/source/srcV1R2M1.17463/comm/sys/buildtools/pami/api/c/pami.cc:554
>
>                         000000000155d0dc
>                         PMPI_Waitany
>                         /bgsys/source/srcV1R2M1.17463/comm/lib/dev/mpich2/src/mpid/pamid/include/../src/mpid_progress.h:239
>
>                         00000000010e84e4
>                         00000072.long_branch_r2off.H5Dget_space+0
>                         :0
>                         0000000001042be0
>                         00000012.long_branch_r2off._ZNSt14basic_ifstreamIcSt11char_traitsIcEEC1EPKcSt13_Ios_Openmode+0
>
>                         :0
>                         00000000019de058
>                         generic_start_main
>                         /bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/glibc-2.12.2/csu/../csu/libc-start.c:226
>
>                         00000000019de354
>                         __libc_start_main
>                         /bgsys/drivers/V1R2M1/ppc64/toolchain/gnu/glibc-2.12.2/csu/../sysdeps/unix/sysv/linux/powerpc/libc-start.c:194
>
>                         0000000000000000
>                         ??
>                         ??:0
>
>                          >   Have these sort of runs succeeded in the
>                         past using the same code base with no changes
>                         and similar input data?
>                         That is the first time i'm trying to run this
>                         code for that long time
>                         Thanks
>                         Anton
>
>                         On Thu, 24 Apr 2014 15:45:49 -0500, Scott
>                         Parker wrote:
>
>
>                             Anton-
>
>                             Thanks, it's aborting because of a runtime
>                             error that appears to be in the mpich layer.
>
>                             Can you rerun with  "--env
>                             BG_COREDUMPONEXIT=1:BG_COREDUMPONERROR=1"
>                             added to your qsub line - that should
>                             generate some core files on which you can
>                             run bgq_stack.
>
>                             The system software (driver) on Mira was
>                             updated this week and I'd like to get a
>                             clearer picture of
>                             whether that could be related to you problem, so
>
>                                 Has your code been recompiled since
>                             Monday? If not can you recompile and try
>                             running again
>
>                                 Have these sort of runs succeeded in the
>                             past using the same code base with no
>                             changes and similar input data?
>
>                             -Scott
>
>
>
>                             On 4/24/14, 2:59 PM, kanaev at ibrae.ac.ru
>                             <mailto:kanaev at ibrae.ac.ru> wrote:
>
>                                 Sorry about the attached files, here the
>                                 are
>                                 There's no core files after exiting,
>                                 looks like stopping because of time
>                                 requested expires but you can see from
>                                 cobaltlog it's about 5 hours passed (10
>                                 hours was requested) before exit
>                                 Anton
>                                 On Thu, 24 Apr 2014 14:07:07 -0500,
>                                 Scott Parker wrote:
>
>
>                                     Anton-
>
>                                     Please send these emails to
>                                     support at alcf.anl.gov
>                                     <mailto:support at alcf.anl.gov> as I
>                                     may not always be available to
>                                     investigate.
>
>                                     I don't see the cobalt or error
>                                     files attached, so I can't really
>                                     say anything about why your job may be
>                                     failing. Do you get core files when
>                                     the job crashed? If so I'd recommend
>                                     using 'bgq_stack '
>                                     to try and get the file and line
>                                     number where the failure occurred.
>                                     Knowing the line may be enough
>                                     to let you figure it out, if not
>                                     you'll need to dump the values of
>                                     the variables at the time of the crash
>                                     to get a clearer picture of what is
>                                     going on.
>
>                                     Scott
>
>                                     On 4/24/14, 1:36 PM,
>                                     kanaev at ibrae.ac.ru
>                                     <mailto:kanaev at ibrae.ac.ru> wrote:
>
>                                         Hello Scott,
>                                         I've tried twice to run 10 hours
>                                         1024 cores job on Mira in mode
>                                         c16 with
>                                         qsub -n 64 -t 10:00:00 --mode
>                                         c16 -A ThermHydraX pinit
>                                         Lid_128x128x1p1024.h5m
>                                         Both times job exited earlier
>                                         than expected on the same
>                                         iteration after the same error
>                                         during executing the following
>                                         section (it's between two couts):
>                                           ...
>                                         //LOOP OVER OWNED CELLS
>                                               double r = 0;
>                                               for (moab::Range::iterator
>                                         it = owned_ents.begin(); it !=
>                                         owned_ents.end(); it++) {
>                                                EntityHandle ent = *it;
>                                                int cellid =
>                                         mb->id_from_handle(ent);
>                                                double Vol_;
>                                                double u_;
>                                                double v_;
>                                                double w_;
>                                                double r1,r2,r3;
>                                                double tmp;
>                                                result =
>                                         mb->tag_get_data(u, &ent, 1, &u_);
>                                                PRINT_LAST_ERROR;
>                                                result =
>                                         mb->tag_get_data(v, &ent, 1, &v_);
>                                                PRINT_LAST_ERROR;
>                                                result =
>                                         mb->tag_get_data(w, &ent, 1, &w_);
>                                                PRINT_LAST_ERROR;
>                                                double result;
>
>                                         SCALAR_PRODUCT(result,u_,v_,w_,CG[cellid][2].lx,CG[cellid][2].ly,CG[cellid][2].lz);
>
>                                                r1 = (sound +
>                                         fabs(result))/CG[cellid][2].length;
>
>                                         SCALAR_PRODUCT(result,u_,v_,w_,CG[cellid][3].lx,CG[cellid][3].ly,CG[cellid][3].lz);
>
>                                                r2 = (sound +
>                                         fabs(result))/CG[cellid][3].length;
>
>                                         SCALAR_PRODUCT(result,u_,v_,w_,CG[cellid][5].lx,CG[cellid][5].ly,CG[cellid][5].lz);
>
>                                                r3 = (sound +
>                                         fabs(result))/CG[cellid][5].length;
>                                                tmp = MAX3(r1,r2,r3);
>                                                r = MAX(tmp,r);
>                                           }
>                                           double rmax;
>                                           MPI_Allreduce(&r,&rmax,1,MPI_REAL8,MPI_MAX,MPI_COMM_WORLD);
>                                           tau = CFL/rmax;
>                                           ttime+=tau;
>                                         ...
>                                         So it may be Allreduce
>                                         I've attached cobaltlog and
>                                         error files of both runs
>                                         Can you please take a look and
>                                         suggest a further debugging
>                                         Thanks
>                                         Anton
>
>
>
>
>
>

--
Timothy J. Tautges
Manager, Directed Meshing, CD-adapco
Phone: 608-354-1459
timothy.tautges at cd-adapco.com


More information about the moab-dev mailing list