<div dir="ltr">
<span style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Hey All,</span><div style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial">I have been stuck in this problem for a while now but I think it should be simple and probably you can tell where I am going wrong.</div><div style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial">- So I have a forcing function on a finite difference X-Y grid, which I am reading in as a text file in the 'userdat2' subroutine. </div><div style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial">- I am then parallelizing it using a previous suggestion by Paul, here is a snippet of the routine.</div><div style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">
<div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font face="verdana, sans-serif">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------</font></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> subroutine usrdat2</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"><br></font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> include 'SIZE'</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> include 'TOTAL'</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> double precision :: my_array(651)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> integer m</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> common /myarray/ my_array</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><font face="verdana, sans-serif"><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> parameter (my_array_size=651)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> real work(my_array_size)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> n = my_array_size</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> call rzero(my_array,n)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> call rzero(work,n)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> if (nid.eq.0) then</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> open (unit=49,file='fint41.txt')</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> do k = 1,n</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> read(49,*) my_array(k)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> enddo</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> close(49)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> endif</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> call gop(my_array,work,'+ ',n) ! Sum over all processors</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><br><span style="font-variant-ligatures:no-common-ligatures"></span></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> return</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> end</font></span></p></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>---------------</font></span></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"><br></font></span></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace">
<span style="font-family:arial,sans-serif;font-size:12.8px;font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">- Next I am interpolating this forcing function to the GLL grid points in the userf routine, by a piecewise linear interpolant, here's how,</span>
<br></font></span></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-variant-ligatures:no-common-ligatures"><font face="monospace, monospace"><br></font></span></div><div style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-variant-ligatures:no-common-ligatures">
<div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font face="verdana, sans-serif">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>----------------</font></div><div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font face="verdana, sans-serif"><br></font></div><div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> subroutine userf (ix,iy,iz,ieg)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"><br></font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> include 'SIZE'</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> include 'TOTAL'</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> include 'NEKUSE'</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> EQUIVALENCE (prandtl,param(2))</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"><br></font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> double precision :: my_array(651)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> double precision :: shaped_array(31,21)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> real new_array(nx1,ny1,nz1,nelv)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> double precision :: xin(21),yin(31)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> double precision :: xn(nx1*ny1*nelv),yn(nx1*ny1*ne<wbr>lv)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> double precision :: fo(nx1*ny1*nelv)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> integer i,j,k,l,p,q</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> integer e,eg</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> common /myarray/ my_array</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> open(16,file='xin.txt')</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> do p=1,21</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> read(16,*) xin(p)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> end do</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> close(16)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> open(12,file='yin.txt')</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> do q=1,31</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> read(12,*) yin(q)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> end do</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> close(12)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><span style="font-variant-ligatures:no-common-ligatures"></span><br></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> shaped_array = reshape(my_array,(/31,21/))</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><font face="verdana, sans-serif"><br><span style="font-variant-ligatures:no-common-ligatures"></span></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> do i=1,nx1*ny1*nelv</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> xn(i) = xm1(i,1,1,1)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> yn(i) = ym1(i,1,1,1)</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> end do</font></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"> call pwl_interp_2d(31,21,yin,xin,sh<wbr>aped_array,nx1*ny1*nelv,yn,xn,<wbr>fo)</font></span></p></div><div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"><br></font></span></div><div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-variant-ligatures:no-common-ligatures"><font face="verdana, sans-serif"><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> new_array=reshape(fo,(/nx1,n<wbr>y1,nz1,nelv/))</span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;min-height:13px"><span style="color:rgb(0,0,0)"> <span> </span></span><font color="#cc0000"> do e=1,nelv<br><span style="font-variant-ligatures:no-common-ligatures"></span></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;min-height:13px"><font color="#cc0000"> eg = lglel(e)<br><span style="font-variant-ligatures:no-common-ligatures"></span></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;min-height:13px"><font color="#cc0000"> ffx = new_array(ix,iy,iz,eg)<br><span style="font-variant-ligatures:no-common-ligatures"></span></font></p><p style="margin:0px;font-stretch:normal;line-height:normal;min-height:13px"><font color="#cc0000"> end do</font></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><br></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"> ffy = 0.0<br><span style="font-variant-ligatures:no-common-ligatures"></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"> ffz = 0.0<br><span style="font-variant-ligatures:no-common-ligatures"></span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> return</span></p><p style="margin:0px;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> end</span></p><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></font></span></div><div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font face="verdana, sans-serif"><br></font></div><div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font face="verdana, sans-serif">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>---------------</font></div><div style="font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><font face="verdana, sans-serif"><br></font></div>
<div style="font-size:12.8px;font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">This is a test domain I made, with 6 elements. It seems to be working when I am using only 1 core, i.e., in series. In parallel, it seems to be giving me a different flowfield. My actual domain has 252 elements which I intend to parallelize with 48 cores. So would like to ask you how to proceed. I think I am close but I am slightly confused about part of the code that I have made red. </div><div style="font-size:12.8px;font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:12.8px;font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">When I am trying this in my actual domain, I get an error, "Segmentation fault occurred." I think the key here is parallelization and identifying the elements correctly which I am not sure I am doing right. Thanks for reading this lengthy email. Will be eager to hear from you.</div><div style="font-size:12.8px;font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:12.8px;font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Thanks,</div><div style="font-size:12.8px;font-variant-ligatures:normal;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">Saikat</div>
<br class="gmail-Apple-interchange-newline">
<br></span></div>
<br></div>
<br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div></div></div></div></div></div></div></div>
</div>