Arrghhh, some bad compilers use SIGUSR1 to communicate with themselves. I have<br>
had this. Just keep typing 'cont' until the SEGV.<br>
<br>
Matt<br><br><div><span class="gmail_quote">On 5/28/06, <b class="gmail_sendername">Randall Mackie</b> <<a href="mailto:randy@geosystem.us">randy@geosystem.us</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
Satish,<br><br>Yes, PETSc was compiled in debug mode.<br><br>Since I'm simply storing vectors in a temporary file, could I get<br>around this by using VecView and writing each vector to the<br>same Viewer in binary format, then reading them later?
<br><br>In other words:<br><br> do loop=1,n<br><br> call VecView (xvec(:,loop).....)<br><br> end do<br><br><br><br>then later<br><br><br> do loop=1,n<br><br> call VecLoad (xvec(:,loop)....)
<br><br> end do<br><br><br>Randy<br><br>ps. I'll try your other suggestions as well. However, this code has worked<br>flawlessly until now, with a model much much larger than I've used in the<br>past.<br><br><br><br>
Satish Balay wrote:<br>> - Not sure what SIGUSR1 means in this context.<br>><br>> - The stack doesn't show any PETSc/user code. Was<br>> this code compiled with debug version of PETSc?<br>><br>> - it could be that gdb is unable to look at intel compilers stack
<br>> [normally gdb should work]. If thats the case - you could run with<br>> '-start_in_debugger idb']<br>><br>> - It appears that this breakage is from usercode which calls fortran<br>> I/O [for_write_dir_xmit()]. There is no fortran I/O from PETSc side
<br>> of the code. I think it could still be a bug in the usercode.<br>><br>> However PETSc does try to detect the availability of<br>> _intel_fast_memcpy() and use it from C side. I don't think this is the<br>
> cause. But to verify you could remove the flag<br>> PETSC_HAVE__INTEL_FAST_MEMCPY from petscconf.h and rebuild libraries.<br>><br>> Satish<br>><br>><br>> On Sun, 28 May 2006, Randall Mackie wrote:<br>
><br>>> Satish,<br>>><br>>> Thanks, using method (2) worked. However, when I run a bt in gdb,<br>>> I get the following output:<br>>><br>>> Loaded symbols for /lib/libnss_files.so.2<br>
>> 0x080b2631 in d3inv_3_3 () at d3inv_3_3.F:2063<br>>> 2063 call VecAssemblyBegin(xyz,ierr)<br>>> (gdb) cont<br>>> Continuing.<br>>><br>>> Program received signal SIGUSR1, User defined signal 1.
<br>>> [Switching to Thread 1082952160 (LWP 23496)]<br>>> 0x088cd729 in _intel_fast_memcpy.J ()<br>>> Current language: auto; currently fortran<br>>> (gdb) bt<br>>> #0 0x088cd729 in _intel_fast_memcpy.J ()
<br>>> #1 0x40620628 in for_write_dir_xmit ()<br>>> from /opt/intel_fc_80/lib/libifcore.so.5<br>>> #2 0xbfffa6b0 in ?? ()<br>>> #3 0x00000008 in ?? ()<br>>> #4 0xbfff986c in ?? ()<br>>> #5 0xbfff9890 in ?? ()
<br>>> #6 0x406873a8 in __dtors_list_end () from /opt/intel_fc_80/lib/libifcore.so.5<br>>> #7 0x00000002 in ?? ()<br>>> #8 0x00000000 in ?? ()<br>>> (gdb)<br>>><br>>> This all makes me think this is an INTEL compiler bug, and has nothing to
<br>>> do with my code.<br>>><br>>> Any ideas?<br>>><br>>> Randy<br>>><br>>><br>>> Satish Balay wrote:<br>>>> Looks like you have direct access to all the cluster nodes. Perhaps
<br>>>> you have admin access? You can do either of the following:<br>>>><br>>>> * if the cluster frontend/compute nodes have common filesystem [i.e<br>>>> all machines can see the same file for ~/.Xauthority] and you can get
<br>>>> 'sshd' settings on the frontend changed - then:<br>>>><br>>>> - configure sshd with 'X11UseLocalhost no' - this way xterms on the<br>>>> compute-nodes can connect to the 'ssh-x11' port on the frontend - run
<br>>>> the PETSc app with: '-display frontend:ssh-x11-port'<br>>>><br>>>> * However if the above is not possible - but you can ssh directly to<br>>>> all the the compute nodes [perhaps from the frontend] then you can
<br>>>> cascade x11 forwarding with:<br>>>><br>>>> - ssh from desktop to frontend<br>>>> - ssh from frontend to node-9 [if you know which machine is node9<br>>>> from the machine file.]
<br>>>> - If you don't know which one is the node-9 - then ssh from frontend<br>>>> to all the nodes :). Mostlikely all nodes will get a display<br>>>> 'localhost:l0.0'<br>>>> - so now you can run the executable with the option
<br>>>> -display localhost:10.0<br>>>><br>>>> The other alternative that might work [for interactive runs] is:<br>>>><br>>>> -start_in_debugger noxterm -debugger_nodes 9
<br>>>><br>>>> Satish<br>>>><br>>>> On Sat, 27 May 2006, Randall Mackie wrote:<br>>>><br>>>>> I can't seem to get the debugger to pop up on my screen.<br>>>>>
<br>>>>> When I'm logged into the cluster I'm working on, I can<br>>>>> type xterm &, and an xterm pops up on my display. So I know<br>>>>> I can get something from the remote cluster.
<br>>>>><br>>>>> Now, when I try this using PETSc, I'm getting the following error<br>>>>> message, for example:<br>>>>><br>>>>> ------------------------------------------------------------------------
<br>>>>> [17]PETSC ERROR: PETSC: Attaching gdb to<br>>>>> /home/randy/d3inv/PETSC_V3.3/d3inv_3_3_petsc of pid 3628 on display<br>>>>> <a href="http://24.5.142.138:0">24.5.142.138:0</a>.0 on machine
compute-0-23.local<br>>>>> ------------------------------------------------------------------------<br>>>>><br>>>>> I'm using this in my command file:<br>>>>><br>>>>> source ~/.bashrc
<br>>>>> time /opt/mpich/intel/bin/mpirun -np 20 -nolocal -machinefile machines \<br>>>>> /home/randy/d3inv/PETSC_V3.3/d3inv_3_3_petsc \<br>>>>> -start_in_debugger \<br>
>>>> -debugger_node 1 \<br>>>>> -display <a href="http://24.5.142.138:0">24.5.142.138:0</a>.0 \<br>>>>> -em_ksp_type bcgs \<br>>>>> -em_sub_pc_type ilu \
<br>>>>> -em_sub_pc_factor_levels 8 \<br>>>>> -em_sub_pc_factor_fill 4 \<br>>>>> -em_sub_pc_factor_reuse_ordering \<br>>>>> -em_sub_pc_factor_reuse_fill \
<br>>>>> -em_sub_pc_factor_mat_ordering_type rcm \<br>>>>> -divh_ksp_type cr \<br>>>>> -divh_sub_pc_type icc \<br>>>>> -ppc_sub_pc_type ilu \
<br>>>>> << EOF<br>>><br>><br><br>--<br>Randall Mackie<br>GSY-USA, Inc.<br>PMB# 643<br>2261 Market St.,<br>San Francisco, CA 94114-1600<br>Tel (415) 469-8649<br>Fax (415) 469-5044<br><br>California Registered Geophysicist
<br>License No. GP 1034<br><br></blockquote></div><br><br clear="all"><br>-- <br>"Failure has a thousand explanations. Success doesn't need one" -- Sir Alec Guiness