<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""> The test program has</div>
<div class=""><br class="">
</div>
<div class="">integer mpistatus</div>
<div class=""><br class="">
</div>
<div class=""> It should be</div>
<div class=""><br class="">
</div>
<div class="">integer(MPI_STATUS_SIZE) mpistatus</div>
<div class=""><br class="">
</div>
<div class=""> I don’t have a GPFS filesystem to test on, but it runs fine on NFS and Lustre for me.</div>
<br class="">
<div class="">
<div class="">Michael A. Raymond</div>
<div class="">SGI MPT Team Leader</div>
<div class="">1 (651) 683-7523</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 4, 2016, at 07:48, Michael Raymond <<a href="mailto:mraymond@sgi.com" class="">mraymond@sgi.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class=""> If I compile the test program with -g, it runs fine. Digging deeper.</div>
<br class="">
<div class="">
<div class="">Michael A. Raymond</div>
<div class="">SGI MPT Team Leader</div>
<div class="">1 (651) 683-7523</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Feb 3, 2016, at 14:26, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] <<a href="mailto:eric.kemp@nasa.gov" class="">eric.kemp@nasa.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 16px; font-family: Arial, sans-serif;" class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">Hi Wei-keng:</div>
<div class=""><br class="">
</div>
<div class="">Your program (successfully?) triggers two errors when I run it with 'mpiexec_mpt –np 2 ./mpi_io_hint'</div>
<div class=""><br class="">
</div>
<div class="">(1) This error message:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">MPI_File_f2c: Invalid file handle</div>
<div class=""> Error at MPI_File_close Invalid file handle</div>
</div>
<div class=""><br class="">
</div>
<div class="">(2) The program then hangs and must be manually killed.</div>
<div class=""><br class="">
</div>
<div class="">The same behavior occurs when increasing the number of MPI processes. However, when I only run with 1 process (serially), I only see the error message (1) but the program stops gracefully. These errors are with SGI MPT 2.12.</div>
<div class=""><br class="">
</div>
<div class="">The program runs w/o issue if I use Intel MPI 15.0.3.187. </div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class=""><br class="">
</div>
<div class="">-Eric</div>
<div class=""><br class="">
</div>
</div>
<span id="OLK_SRC_BODY_SECTION" class="">
<div style="font-family: 'Lucida Grande'; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223);" class="">
<span style="font-weight:bold" class="">From: </span>Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>><br class="">
<span style="font-weight:bold" class="">Date: </span>Wednesday, February 3, 2016 2:32 PM<br class="">
<span style="font-weight:bold" class="">To: </span>Eric Kemp <<a href="mailto:eric.kemp@nasa.gov" class="">eric.kemp@nasa.gov</a>><br class="">
<span style="font-weight:bold" class="">Cc: </span>"<a href="mailto:parallel-netcdf@mcs.anl.gov" class="">parallel-netcdf@mcs.anl.gov</a>" <<a href="mailto:parallel-netcdf@mcs.anl.gov" class="">parallel-netcdf@mcs.anl.gov</a>><br class="">
<span style="font-weight:bold" class="">Subject: </span>Re: Problems with testing PNETCDF 1.6.1<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div class="BodyFragment"><font size="2" class=""><span style="font-size:10pt;" class="">
<div class="PlainText">Hi, Eric<br class="">
<br class="">
Many thanks for digging the problem. It is great to know the cause.<br class="">
<br class="">
In PnetCDF, writing the file header is done by root process using an<br class="">
independent write. So, this line is not necessary. However, setting<br class="">
this hint should not cause the program to hang or break anything.<br class="">
<br class="">
This indicates a bug in the MPI-IO component of the SGI MPT.<br class="">
<br class="">
Attached is a small test program that should reproduce the hang<br class="">
problem you encountered.<br class="">
<br class="">
Thanks again for put more effort on this!<br class="">
<br class="">
Wei-keng<br class="">
<br class="">
</div>
</span></font></div>
<div class="BodyFragment"><font size="2" class=""><span style="font-size:10pt;" class="">
<div class="PlainText"><br class="">
On Feb 3, 2016, at 12:41 PM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote:<br class="">
<br class="">
> <br class="">
> Hi Wei-keng:<br class="">
> <br class="">
> Good news: I went through the source code, adding print statements, and<br class="">
> traced the hang to the nfmpi_wait_all invocation in<br class="">
> checkpoint_ncmpi_parallel.F90. After some further trial-and-error, I<br class="">
> fixed the problem by commenting out this line in subroutine<br class="">
> checkpoint_wr_ncmpi_par:<br class="">
> <br class="">
> call MPI_Info_set(file_info, 'romio_no_indep_rw', 'true', err)<br class="">
> <br class="">
> I don't understand what is going on under the hood, but perhaps the<br class="">
> combination of SGI MPT and GPFS (the file system on NASA's supercomputer)<br class="">
> makes this setting undesirable?<br class="">
> <br class="">
> Anyway, after commented out this line, both SGI MPT and Intel MPI tests<br class="">
> work, and produce FLASH output files with the appropriate sizes. So I'm<br class="">
> going to declare victory and install with this change.<br class="">
> <br class="">
> Thanks for your help!<br class="">
> <br class="">
> -Eric<br class="">
> <br class="">
> Eric M. Kemp (SSAI)<br class="">
> NASA/GSFC <br class="">
> Mail Code: 606 <br class="">
> Greenbelt, MD 20771<br class="">
> 301.286.9768 <br class="">
> <a href="mailto:eric.kemp@nasa.gov" class="">eric.kemp@nasa.gov</a><br class="">
> <a href="mailto:eric.kemp@ssaihq.com" class="">eric.kemp@ssaihq.com</a><br class="">
> <br class="">
> <br class="">
> <br class="">
> On 2/3/16 11:24 AM, "Wei-keng Liao" <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>> wrote:<br class="">
> <br class="">
>> Hi, Eric<br class="">
>> <br class="">
>> The file size 2964 is the file header size supposed to be. This<br class="">
>> indicates the program has reached the call to nfmpi_enddef() at<br class="">
>> line 155 of file checkpoint_ncmpi_parallel.F90.<br class="">
>> The file size also tells that no data has been written to the file yet.<br class="">
>> In this case, two possible places can cause the program to hang:<br class="">
>> line 155 and line 600.<br class="">
>> <br class="">
>> If you put a print statement after line 156, then it can check if<br class="">
>> the program returns from it correctly or hangs at nfmpi_enddef().<br class="">
>> <br class="">
>> The other possible hangs is at line 600, the call to nfmpi_wait_all.<br class="">
>> <br class="">
>> Wei-keng<br class="">
>> <br class="">
>> On Feb 3, 2016, at 7:08 AM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS<br class="">
>> AND APPLICATIONS INC] wrote:<br class="">
>> <br class="">
>>> <br class="">
>>> Good morning Wei-keng:<br class="">
>>> <br class="">
>>> I reran 1.7.0.pre1 FLASH-IO again. The only output from the program is:<br class="">
>>> <br class="">
>>> rw-r--r-- 1 emkemp k3002 2964 Feb 3 07:53<br class="">
>>> flash_io_test_ncmpi_chk_0000.nc<br class="">
>>> <br class="">
>>> I will tinker with the source code to see if I can identify where it<br class="">
>>> hangs.<br class="">
>>> <br class="">
>>> <br class="">
>>> Thanks,<br class="">
>>> <br class="">
>>> -Eric<br class="">
>>> <br class="">
>>> <br class="">
>>> <br class="">
>>> On 2/2/16 5:36 PM, "Wei-keng Liao" <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>> wrote:<br class="">
>>> <br class="">
>>>> Hi, Eric<br class="">
>>>> <br class="">
>>>> Unfortunately, I do not have access to an SGI machine. We usually<br class="">
>>>> rely on our users to do some initial debugging for the situation like<br class="">
>>>> this.<br class="">
>>>> I know this can be too much to ask, but if you did not encounter any<br class="">
>>>> problem when running your program, maybe you can ignore this test.<br class="">
>>>> <br class="">
>>>> However, since the hanging occurs only with SGI MPT, I suspect it is<br class="">
>>>> related to MPT.<br class="">
>>>> <br class="">
>>>> Could you check one thing for me? After you kill the FLASH-IO job,<br class="">
>>>> could<br class="">
>>>> you check if any netCDF files were created? The expected files and<br class="">
>>>> their<br class="">
>>>> sizes are<br class="">
>>>> <br class="">
>>>> -rw------- 1 254075392 Feb 1 21:58 flash_io_test_ncmpi_chk_0000.nc<br class="">
>>>> -rw------- 1 21208576 Feb 1 21:58 flash_io_test_ncmpi_plt_cnt_0000.nc<br class="">
>>>> -rw------- 1 25431372 Feb 1 21:58 flash_io_test_ncmpi_plt_crn_0000.nc<br class="">
>>>> <br class="">
>>>> <br class="">
>>>> Wei-keng<br class="">
>>>> <br class="">
>>>> On Feb 2, 2016, at 3:53 PM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS<br class="">
>>>> AND APPLICATIONS INC] wrote:<br class="">
>>>> <br class="">
>>>>> <br class="">
>>>>> Hi Wei-keng:<br class="">
>>>>> <br class="">
>>>>> I tried rerunning the entire installation with PNETCDF_SAFE_MODE=1.<br class="">
>>>>> FLASH-IO still hangs with SGI MPT (with no error message), but it<br class="">
>>>>> works<br class="">
>>>>> fine with Intel MPI.<br class="">
>>>>> <br class="">
>>>>> -Eric<br class="">
>>>>> <br class="">
>>>>> <br class="">
>>>>> From: Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>><br class="">
>>>>> Date: Tuesday, February 2, 2016 12:39 PM<br class="">
>>>>> To: Eric Kemp <<a href="mailto:eric.kemp@nasa.gov" class="">eric.kemp@nasa.gov</a>><br class="">
>>>>> Cc: "<a href="mailto:parallel-netcdf@mcs.anl.gov" class="">parallel-netcdf@mcs.anl.gov</a>" <<a href="mailto:parallel-netcdf@mcs.anl.gov" class="">parallel-netcdf@mcs.anl.gov</a>><br class="">
>>>>> Subject: Re: Problems with testing PNETCDF 1.6.1<br class="">
>>>>> <br class="">
>>>>> Hi, Eric<br class="">
>>>>> <br class="">
>>>>> Sorry for sending the wrong file. The correct one is attached, in case<br class="">
>>>>> you would like<br class="">
>>>>> to use it.<br class="">
>>>>> <br class="">
>>>>> I check your config.log file but could not find any thing fishy.<br class="">
>>>>> I just now tested it with Intel compiler 16.0.0.109 without a problem.<br class="">
>>>>> Could you try running FLASH-IO under the safe mode? i.e. set the<br class="">
>>>>> environment<br class="">
>>>>> variable PNETCDF_SAFE_MODE to 1. It will enable internal checking for<br class="">
>>>>> data inconsistency.<br class="">
>>>>> <br class="">
>>>>> Just want to make sure for 1.7.0.pre1 that your "make ptest" failed<br class="">
>>>>> only on FLASH-IO.<br class="">
>>>>> Because FLAH-IO is the last test program, this means all other tests<br class="">
>>>>> have passed.<br class="">
>>>>> Let me know. Thanks.<br class="">
>>>>> <br class="">
>>>>> Wei-keng<br class="">
>>>>> <br class="">
>>>>> <br class="">
>>>>> On Feb 2, 2016, at 8:27 AM, Kemp, Eric M. (GSFC-606.0)[SCIENCE SYSTEMS<br class="">
>>>>> AND APPLICATIONS INC] wrote:<br class="">
>>>>> <br class="">
>>>>>> <br class="">
>>>>>> Hi Wei-keng:<br class="">
>>>>>> <br class="">
>>>>>> I think you sent me the wrong copy of that file ‹ it was identical to<br class="">
>>>>> what is in 1.7.0.pre1. But I went ahead and added "cmd" as an<br class="">
>>>>> argument<br class="">
>>>>> to subroutine check_err, and that test code compiles and runs.<br class="">
>>>>>> <br class="">
>>>>>> The large file tests pass in 1.7.0.pre1 as you indicated. However,<br class="">
>>>>> FLASH-IO still hangs with SGI MPT. I took your suggestion and tried<br class="">
>>>>> running this test separately (cd benchmarks/FLASH-IO ; make ptest) but<br class="">
>>>>> the code still hangs.<br class="">
>>>>>> <br class="">
>>>>>> I've attached the (gzipped) config.log file from the 1.7.0pre1<br class="">
>>>>> installations.<br class="">
>>>>>> <br class="">
>>>>>> Thanks,<br class="">
>>>>>> <br class="">
>>>>>> -Eric<br class="">
>>>>>> <br class="">
>>>>>> Eric M. Kemp (SSAI)<br class="">
>>>>>> NASA/GSFC <br class="">
>>>>>> Mail Code: 606<br class="">
>>>>>> Greenbelt, MD 20771<br class="">
>>>>>> 301.286.9768 <br class="">
>>>>>> <a href="mailto:eric.kemp@nasa.gov" class="">eric.kemp@nasa.gov</a><br class="">
>>>>>> <a href="mailto:eric.kemp@ssaihq.com" class="">eric.kemp@ssaihq.com</a><br class="">
>>>>>> <br class="">
>>>>>> <br class="">
>>>>>> From: Wei-keng Liao <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>><br class="">
>>>>>> Date: Monday, February 1, 2016 5:12 PM<br class="">
>>>>>> To: Eric Kemp <<a href="mailto:eric.kemp@nasa.gov" class="">eric.kemp@nasa.gov</a>><br class="">
>>>>>> Cc: "<a href="mailto:parallel-netcdf@mcs.anl.gov" class="">parallel-netcdf@mcs.anl.gov</a>" <<a href="mailto:parallel-netcdf@mcs.anl.gov" class="">parallel-netcdf@mcs.anl.gov</a>><br class="">
>>>>>> Subject: Re: Problems with testing PNETCDF 1.6.1<br class="">
>>>>>> <br class="">
>>>>>> Hi, Eric,<br class="">
>>>>>> <br class="">
>>>>>> Thanks for reporting the error. This is another oversight, Sorry.<br class="">
>>>>>> The fixed file, bigrecords.f, is attached.<br class="">
>>>>>> <br class="">
>>>>>> <br class="">
>>>>>> Wei-keng<br class="">
>>>>>> <br class="">
>>>>>> <br class="">
>>>>>> <br class="">
>>>>>> On Feb 1, 2016, at 2:41 PM, Kemp, Eric M. (GSFC-606.0)[SCIENCE<br class="">
>>>>> SYSTEMS AND APPLICATIONS INC] wrote:<br class="">
>>>>>> <br class="">
>>>>>>> <br class="">
>>>>>>> Hi Wei-keng:<br class="">
>>>>>>> <br class="">
>>>>>>> Thanks for your quick response. I tried installing 1.7.0.pre1 but I<br class="">
>>>>> get a<br class="">
>>>>>>> different error when compiling the tests:<br class="">
>>>>>>> <br class="">
>>>>>>> /usr/local/intel/2016/impi/5.1.2.150/bin64/mpif90 -I../../src/lib<br class="">
>>>>>>> -I./../common -I../../src/libf -I../../src/libf90 -fpic -O2<br class="">
>>>>> -fp-model<br class="">
>>>>>>> strict -c bigrecords.f<br class="">
>>>>>>> bigrecords.f(333): error #6514: A substring must be of type<br class="">
>>>>> CHARACTER.<br class="">
>>>>>>> [CMD]<br class="">
>>>>>>> msg = '*** TESTING F77 '//cmd(1:XTRIM(cmd))//<br class="">
>>>>>>> ------------------------------------^<br class="">
>>>>>>> bigrecords.f(333): error #6054: A CHARACTER data type is required<br class="">
>>>>> in this<br class="">
>>>>>>> context. [CMD]<br class="">
>>>>>>> msg = '*** TESTING F77 '//cmd(1:XTRIM(cmd))//<br class="">
>>>>>>> ------------------------------------^<br class="">
>>>>>>> compilation aborted for bigrecords.f (code 1)<br class="">
>>>>>>> <br class="">
>>>>>>> <br class="">
>>>>>>> <br class="">
>>>>>>> This appears to be a legitimate syntax error in the test program, in<br class="">
>>>>>>> subroutine check_err. "cmd" is not defined in that subroutine, nor<br class="">
>>>>> is it<br class="">
>>>>>>> a global variable.<br class="">
>>>>>>> <br class="">
>>>>>>> I will try patching 1.6.1 with the NC_64BIT_DATA constant instead.<br class="">
>>>>>>> <br class="">
>>>>>>> -Eric<br class="">
>>>>>>> <br class="">
>>>>>>> <br class="">
>>>>>>> <br class="">
>>>>>>> <br class="">
>>>>>>> On 2/1/16 12:02 PM, "Wei-keng Liao" <<a href="mailto:wkliao@eecs.northwestern.edu" class="">wkliao@eecs.northwestern.edu</a>><br class="">
>>>>> wrote:<br class="">
>>>>>>> <br class="">
>>>>>>>> Hi, Eric<br class="">
>>>>>>>> <br class="">
>>>>>>>> For the large file tests, the error is caused by a oversight of<br class="">
>>>>> using a<br class="">
>>>>>>>> wrong flag.<br class="">
>>>>>>>> Line 81 of file large_files.c should have used NC_64BIT_DATA,<br class="">
>>>>> instead of<br class="">
>>>>>>>> NC_64BIT_OFFSET.<br class="">
>>>>>>>> This error has been fixed in the pre-release of 1.7.0.pre1. Could<br class="">
>>>>> you<br class="">
>>>>>>>> give it a try?<br class="">
>>>>>>>> <a href="http://cucis.ece.northwestern.edu/projects/PnetCDF/download.html" class="">
http://cucis.ece.northwestern.edu/projects/PnetCDF/download.html</a><br class="">
>>>>>>>> <br class="">
>>>>>>>> As for the FLASH-IO test, could you try running it alone? I.e. cd<br class="">
>>>>> to the<br class="">
>>>>>>>> folder<br class="">
>>>>>>>> benchmarks/FLASH-IO and run "make ptest" there. In the meantime,<br class="">
>>>>> please<br class="">
>>>>>>>> send me<br class="">
>>>>>>>> the file config.log.<br class="">
>>>>>>>> <br class="">
>>>>>>>> <br class="">
>>>>>>>> Wei-keng<br class="">
>>>>>>>> <br class="">
>>>>>>>> On Feb 1, 2016, at 7:32 AM, Kemp, Eric M. (GSFC-606.0)[SCIENCE<br class="">
>>>>> SYSTEMS<br class="">
>>>>>>>> AND APPLICATIONS INC] wrote:<br class="">
>>>>>>>> <br class="">
>>>>>>>>> <br class="">
>>>>>>>>> Dear PNETCDF developers:<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> I'm attempting to install PNETCDF1.6.1 on a Linux cluster running<br class="">
>>>>> SLES<br class="">
>>>>>>>>> 11.3. I'm using Intel 15 Fortran and C compilers (no C++), and<br class="">
>>>>> I'm<br class="">
>>>>>>>>> trying to install for two separate MPI implementations (SGI MPT<br class="">
>>>>> 2.12 and<br class="">
>>>>>>>>> Intel MPI 5.1.2).<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> I'm encountering two problems when I run 'make ptest'.<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> 1) For both MPI implementations, the large file tests fail with<br class="">
>>>>> an<br class="">
>>>>>>>>> integer overflow. The error message is:<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> *** Testing large files, slowly.<br class="">
>>>>>>>>> line 116 of large_files.c: Overflow when type cast to 4-byte<br class="">
>>>>> integer.<br class="">
>>>>>>>>> *** Creating large file ./testfile.nc...srun.slurm: error:<br class="">
>>>>> borgo018:<br class="">
>>>>>>>>> task 0: Exited with exit code 1<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> I reviewed the README.large_files for guidance, and I can confirm<br class="">
>>>>> that<br class="">
>>>>>>>>> both 'MPI_Offset' and 'off_t' are 8 bytes.<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> 2) For SGI MPT only, if I disable support for large file tests,<br class="">
>>>>> 'make<br class="">
>>>>>>>>> ptest' hangs when testing FLASH-IO:<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> make -w -C FLASH-IO ptest<br class="">
>>>>>>>>> make[2]: Entering directory<br class="">
>>>>>>>>> <br class="">
>>>>> <br class="">
>>>>> `/gpfsm/dnb32/emkemp/NUWRFLIB/svn/trunk/builds/parallel-netcdf-1.6.1/be<br class="">
>>>>> nc<br class="">
>>>>>>>>> hmarks/FLASH-IO'<br class="">
>>>>>>>>> mpiexec_mpt -n 4 ./flash_benchmark_io ./flash_io_test_<br class="">
>>>>>>>>> srun.slurm: cluster configuration lacks support for cpu binding<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> The earlier tests with both single and multiple processes work<br class="">
>>>>> for SGI<br class="">
>>>>>>>>> MPT. And all tests (again, excluding large file tests) work for<br class="">
>>>>> Intel<br class="">
>>>>>>>>> MPI.<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> I can provide more information (e.g., output from the configure<br class="">
>>>>> script)<br class="">
>>>>>>>>> upon request.<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> Thanks,<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> -Eric<br class="">
>>>>>>>>> <br class="">
>>>>>>>>> Eric M. Kemp (SSAI)<br class="">
>>>>>>>>> NASA/GSFC <br class="">
>>>>>>>>> Mail Code: 606<br class="">
>>>>>>>>> Greenbelt, MD 20771<br class="">
>>>>>>>>> 301.286.9768<br class="">
>>>>>>>>> <a href="mailto:eric.kemp@nasa.gov" class="">eric.kemp@nasa.gov</a><br class="">
>>>>>>>>> <a href="mailto:eric.kemp@ssaihq.com" class="">eric.kemp@ssaihq.com</a><br class="">
>>>>>>>>> <br class="">
>>>>>>>> <br class="">
>>>>>>> <br class="">
>>>>>> <br class="">
>>>>>> <config.log.gz><br class="">
>>>>> <br class="">
>>>> <br class="">
>>> <br class="">
>> <br class="">
> <br class="">
<br class="">
</div>
</span></font></div>
</div>
</div>
</span></div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>