<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Barry,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Apologies for the delay in getting back to you, the mail somehow missed my attention when it first arrived,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
but thanks for this tip - this would allow the <span style="color:rgb(23,78,134);font-family:Calibri,Helvetica,sans-serif">
PetscIntView</span> to be used even when in practice only one process <br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
was actually writing out information.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Dan.<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Barry Smith <bsmith@petsc.dev><br>
<b>Sent:</b> Thursday, May 20, 2021 5:00 PM<br>
<b>To:</b> dazza simplythebest <sayosale@hotmail.com><br>
<b>Cc:</b> PETSc users list <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)</font>
<div> </div>
</div>
<div class="" style="word-wrap:break-word; line-break:after-white-space">
<div class=""><br class="">
</div>
You can also have the processes with no values print an array of length zero. Like
<div class=""><br class="">
</div>
<div class=""> <span class="" style="color:rgb(23,78,134); font-family:Calibri,Helvetica,sans-serif">if (rank3 == PROC_ROW) then ! IF mpi PROCESS OWNS THIS ROW THEN ..</span></div>
<div class=""><font class="" face="Calibri, Helvetica, sans-serif" color="#174e86"><span class="" style=""> ..</span></font></div>
<div class=""><font class="" face="Calibri, Helvetica, sans-serif" color="#174e86"><span class="" style=""> else </span></font></div>
<div class=""><font class="" face="Calibri, Helvetica, sans-serif" color="#174e86"><span class="" style=""> </span></font><span class="" style="color:rgb(23,78,134); font-family:Calibri,Helvetica,sans-serif">NO_A_ENTRIES = 0</span></div>
<div class=""><font class="" face="Calibri, Helvetica, sans-serif" color="#174e86"><span class="" style=""> </span></font><span class="" style="color:rgb(23,78,134); font-family:Calibri,Helvetica,sans-serif">call PetscIntView(NO_A_ENTRIES,JALOC(1:NO_A_ENTRIES),
&</span>
<div class=""><span class="" style="color:rgb(23,78,134); font-family:Calibri,Helvetica,sans-serif"> & PETSC_VIEWER_STDOUT_WORLD, ierr_pets)</span></div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On May 20, 2021, at 5:31 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div>
<br class="x_Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div dir="ltr" class="">On Thu, May 20, 2021 at 5:32 AM dazza simplythebest <<a href="mailto:sayosale@hotmail.com" class="">sayosale@hotmail.com</a>> wrote:<br class="">
</div>
<div class="x_gmail_quote">
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">Dear Jose,</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"> Many thanks for the prompt explanation - that would definitely explain what is going on,</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt">I will adjust my code accordingly .</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">If you want to print different things from each process in parallel, I suggest</div>
<div class=""><br class="">
</div>
<div class=""> <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscSynchronizedPrintf.html" class="">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscSynchronizedPrintf.html</a></div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr" class="">
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"> Thanks again,</div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"> Dan.<br class="">
</div>
<div class="">
<div id="x_gmail-m_-632613171368988235appendonsend" class=""></div>
<div class="" style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt"><br class="">
</div>
<hr class="" style="display:inline-block; width:98%">
<div id="x_gmail-m_-632613171368988235divRplyFwdMsg" dir="ltr" class=""><font class="" style="font-size:11pt" face="Calibri, sans-serif"><b class="">From:</b> Jose E. Roman <<a href="mailto:jroman@dsic.upv.es" target="_blank" class="">jroman@dsic.upv.es</a>><br class="">
<b class="">Sent:</b> Thursday, May 20, 2021 9:06 AM<br class="">
<b class="">To:</b> dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank" class="">sayosale@hotmail.com</a>><br class="">
<b class="">Cc:</b> PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" class="">petsc-users@mcs.anl.gov</a>><br class="">
<b class="">Subject:</b> Re: [petsc-users] Code hangs when calling PetscIntView (MPI, fortran)</font>
<div class=""> </div>
</div>
<div class=""><font class="" size="2"><span class="" style="font-size:11pt">
<div class="">If you look at the manpage <a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscIntView.html" target="_blank" class="">
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscIntView.html</a> you will see that PetscIntView() is collective. This means that all MPI processes must call this function, so it is forbidden to call it within an IF rank==...<br class="">
<br class="">
Jose<br class="">
<br class="">
> El 20 may 2021, a las 10:25, dazza simplythebest <<a href="mailto:sayosale@hotmail.com" target="_blank" class="">sayosale@hotmail.com</a>> escribió:<br class="">
> <br class="">
> Dear All,<br class="">
> As part of preparing a code to call the SLEPC eigenvalue solving library,<br class="">
> I am constructing a matrix in sparse CSR format row-by-row. Just for debugging <br class="">
> purposes I write out the column values for a given row, which are stored in a <br class="">
> PetscInt allocatable vector, using PetscIntView.<br class="">
> <br class="">
> Everything works fine when the number of MPI processes exactly divide the<br class="">
> number of rows of the matrix, and so each process owns the same number of rows.<br class="">
> However, when the number of MPI processes does not exactly divide the<br class="">
> number of rows of the matrix, and so each process owns a different number of rows,<br class="">
> the code hangs when it reaches the line that calls PetscIntView.<br class="">
> To be precise the code hangs on the final row that a process, other than root, owns.<br class="">
> If I however comment out the call to PetscIntView the code completes without error,<br class="">
> and produces the correct eigenvalues (hence we are not missing a row / miswriting a row).<br class="">
> Note also that a simple direct writeout of this same array using a plain fortran command<br class="">
> will write out the array without problem.<br class="">
> <br class="">
> I have attached below a small code that reproduces the problem.<br class="">
> For this code we have nominally assigned 200 rows to our matrix. The code runs without<br class="">
> problem using 1,2,4,5,8 or 10 MPI processes, all of which precisely divide 200,<br class="">
> but will hang for 3 MPI processes for example.<br class="">
> For the case of 3 MPI processes the subroutine WHOSE_ROW_IS_IT allocates the rows
<br class="">
> to each process as :<br class="">
> process no first row last row no. of rows<br class="">
> 0 1 66 66<br class="">
> 1 67 133 67<br class="">
> 2 134 200 67<br class="">
> <br class="">
> The code will hang when process 1 calls PetscIntView for its last row, row 133 for example.<br class="">
> <br class="">
> One piece of additional information that may be relevant is that the code does run to completion
<br class="">
> without hanging if I comment out the final slepc/MPI finalisation command<br class="">
> CALL SlepcFinalize(ierr_pets) <br class="">
> (I of course I get ' bad termination' errors, but the otherwise the run is successful.)<br class="">
> <br class="">
> I would appreciate it if anyone has any ideas on what is going wrong!<br class="">
> Many thanks,<br class="">
> Dan.<br class="">
> <br class="">
> <br class="">
> code:<br class="">
> <br class="">
> MODULE ALL_STAB_ROUTINES<br class="">
> IMPLICIT NONE<br class="">
> CONTAINS<br class="">
> <br class="">
> SUBROUTINE WHOSE_ROW_IS_IT(ROW_NO, TOTAL_NO_ROWS, NO_PROCESSES, &<br class="">
> & OWNER)<br class="">
> ! THIS ROUTINE ALLOCATES ROWS EVENLY BETWEEN mpi PROCESSES<br class="">
> #include <slepc/finclude/slepceps.h><br class="">
> use slepceps<br class="">
> IMPLICIT NONE<br class="">
> PetscInt, INTENT(IN) :: ROW_NO, TOTAL_NO_ROWS, NO_PROCESSES<br class="">
> PetscInt, INTENT(OUT) :: OWNER<br class="">
> PetscInt :: P, REM<br class="">
> <br class="">
> P = TOTAL_NO_ROWS / NO_PROCESSES ! NOTE INTEGER DIVISION<br class="">
> REM = TOTAL_NO_ROWS - P*NO_PROCESSES<br class="">
> IF (ROW_NO < (NO_PROCESSES - REM)*P + 1 ) THEN<br class="">
> OWNER = (ROW_NO - 1)/P ! NOTE INTEGER DIVISION<br class="">
> ELSE<br class="">
> OWNER = ( ROW_NO + NO_PROCESSES - REM -1 )/(P+1) ! NOTE INTEGER DIVISION<br class="">
> ENDIF <br class="">
> END SUBROUTINE WHOSE_ROW_IS_IT<br class="">
> END MODULE ALL_STAB_ROUTINES<br class="">
> <br class="">
> <br class="">
> PROGRAM trialer<br class="">
> USE MPI<br class="">
> #include <slepc/finclude/slepceps.h><br class="">
> use slepceps<br class="">
> USE ALL_STAB_ROUTINES<br class="">
> IMPLICIT NONE<br class="">
> PetscMPIInt rank3, total_mpi_size<br class="">
> PetscInt nl3, code, PROC_ROW, ISTATUS, jm, N_rows,NO_A_ENTRIES<br class="">
> PetscInt, ALLOCATABLE, DIMENSION(:) :: JALOC<br class="">
> PetscInt, PARAMETER :: ZERO = 0 , ONE = 1, TWO = 2, THREE = 3 <br class="">
> PetscErrorCode ierr_pets<br class="">
> <br class="">
> ! Initialise sleps/mpi<br class="">
> call SlepcInitialize(PETSC_NULL_CHARACTER,ierr_pets) ! note that this initialises MPI<br class="">
> call MPI_COMM_SIZE(MPI_COMM_WORLD, total_mpi_size, ierr_pets) !! find total no of MPI processes
<br class="">
> nL3= total_mpi_size<br class="">
> call MPI_COMM_RANK(MPI_COMM_WORLD,rank3,ierr_pets) !! find my overall rank -> rank3<br class="">
> write(*,*)'Welcome: PROCESS NO , TOTAL NO. OF PROCESSES = ',rank3, nl3<br class="">
> <br class="">
> N_rows = 200 ! NUMBER OF ROWS OF A NOTIONAL MATRIX<br class="">
> NO_A_ENTRIES = 12 ! NUMBER OF ENTRIES FOR JALOC<br class="">
> <br class="">
> ! LOOP OVER ROWS <br class="">
> do jm = 1, N_rows<br class="">
> <br class="">
> CALL whose_row_is_it(JM, N_rows , NL3, PROC_ROW) ! FIND OUT WHICH PROCESS OWNS ROW<br class="">
> if (rank3 == PROC_ROW) then ! IF mpi PROCESS OWNS THIS ROW THEN ..<br class="">
> ! ALLOCATE jaloc ARRAY AND INITIALISE<br class="">
> <br class="">
> allocate(jaloc(NO_A_ENTRIES), STAT=ISTATUS )<br class="">
> jaloc = three<br class="">
> <br class="">
> <br class="">
> WRITE(*,*)'JALOC',JALOC ! THIS SIMPLE PLOT ALWAYS WORKS<br class="">
> write(*,*)'calling PetscIntView: PROCESS NO. ROW NO.',rank3, jm<br class="">
> ! THIS CALL TO PetscIntView CAUSES CODE TO HANG WHEN E.G. total_mpi_size=3, JM=133<br class="">
> call PetscIntView(NO_A_ENTRIES,JALOC(1:NO_A_ENTRIES), &<br class="">
> & PETSC_VIEWER_STDOUT_WORLD, ierr_pets)<br class="">
> CHKERRA(ierr_pets)<br class="">
> deallocate(jaloc)<br class="">
> endif<br class="">
> enddo<br class="">
> <br class="">
> CALL SlepcFinalize(ierr_pets)<br class="">
> end program trialer<br class="">
<br class="">
</div>
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br class="" clear="all">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">
-- Norbert Wiener</div>
<div class=""><br class="">
</div>
<div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</body>
</html>