[Darshan-users] dxt_analyzer.py: 'invalid continuation byte' error

[Snyder, Shane]

Thanks for the suggestion, Sam! I'll make sure that particular warning is cleared up in our next release, because that is confusing (since, as you point out, all of our other utilities work directly on Darshan log files, not text output).

We also will have new Python bindings to our C utility library in our next release, that should allow us to simplify things like dxt_analyzer -- that would allow us to skip the step of converting the log to text, as Python applications would be able to operate on Darshan logs directly. We'll make sure to keep the community posted about developments on that front.

--Shane
________________________________
From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Omlin Samuel <omlin at cscs.ch>
Sent: Tuesday, October 13, 2020 6:01 AM
To: Harms, Kevin <harms at alcf.anl.gov>; Carns, Philip H. <carns at mcs.anl.gov>; darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
Subject: Re: [Darshan-users] dxt_analyzer.py: 'invalid continuation byte' error


Hi Kevin,


thanks a lot for this clarification on the usage of the DXT-related tools. This worked for me as well. Thus, it is only a documentation issue. Calling `dxt_analyzer.py --help` prints:
```

-i DXT_LOGNAME, --input DXT_LOGNAME
                        dxt log path
```
It would be good probably to indicate here that this input file should be the output of the darshan-dxt-parser and not a .darshan file,  in particular as, e.g., darshan-job-summary.pl takes directly a .darshan file as input.

Thanks a lot!

Sam

PS:
> Looking at the plot from run, it looks like you were possibly doing I/O almost constantly during the run.
- Yes this is correct: the profiled application is an I/O-performance test.

________________________________
From: Harms, Kevin <harms at alcf.anl.gov>
Sent: Tuesday, October 13, 2020 1:19:02 AM
To: Omlin Samuel; Carns, Philip H.; darshan-users at lists.mcs.anl.gov
Subject: Re: [Darshan-users] dxt_analyzer.py: 'invalid continuation byte' error

Sam,

  I think there was just a misunderstanding on how to use the tools. Please follow this:

  # first parse the DXT log from the darshan log.
  $ ./darshan-dxt-parser ./omlins_a.out_id1174947_10-9-40448-1694713884344482404_1602235659.darshan > parsed-dxt.txt

  # then run the analyzer
  $ python3 dxt_analyzer.py -i parsed-dxt.txt

  # this produces a PDF with the plot
  $ ls *.pdf
  dxt_plot.pdf

  Looking at the plot from run, it looks like you were possibly doing I/O almost constantly during the run.

kevin

________________________________________
From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Omlin  Samuel <omlin at cscs.ch>
Sent: Monday, October 12, 2020 8:10 AM
To: Carns, Philip H.; darshan-users at lists.mcs.anl.gov
Subject: Re: [Darshan-users] dxt_analyzer.py: 'invalid continuation byte' error

Hi Phil,


thanks a lot. You can find the file attached.


Cheers,


Sam

________________________________
From: Carns, Philip H. <carns at mcs.anl.gov>
Sent: Friday, October 9, 2020 10:37:41 PM
To: Omlin Samuel; darshan-users at lists.mcs.anl.gov
Subject: Re: [Darshan-users] dxt_analyzer.py: 'invalid continuation byte' error

Hi Sam,

I'm not sure off hand what would cause that.  Can you share the log file by any chance (you can send it to me directly if you would prefer not to post it on the mailing list)?  I can try to reproduce on some other systems and see if that narrows down the problem.

thanks,
-Phil
________________________________
From: Darshan-users <darshan-users-bounces at lists.mcs.anl.gov> on behalf of Omlin Samuel <omlin at cscs.ch>
Sent: Friday, October 9, 2020 10:18 AM
To: darshan-users at lists.mcs.anl.gov <darshan-users at lists.mcs.anl.gov>
Subject: [Darshan-users] dxt_analyzer.py: 'invalid continuation byte' error

Hi all,

I have tried the following command both on Ubuntu 16.04 (python 3.5) and on SUSE Linux Enterprise Server 15 SP1 (python 3.6) and got the same error:
```
$ python3 /usr/local/bin/dxt_analyzer.py -i omlins_a.out_id1174947_10-9-40448-1694713884344482404_1602235659.darshan
Traceback (most recent call last):
  File "/usr/local/bin/dxt_analyzer.py", line 264, in <module>
    for line in infile:
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe6 in position 24: invalid continuation byte
```

With python 2.7.12 (on Ubuntu), I did not get the error, but the generated plot was empty.

Note that the input file does contain values generated by DXT:
```
$ darshan-dxt-parser omlins_a.out_id1174947_10-9-40448-1694713884344482404_1602235659.darshan | tail -3
 X_POSIX       7   read      197    211527180650      1073741824    804.8381    806.1340  [  0]
 X_POSIX       7   read      198    212600922683      1073741824    806.4404    807.7277  [  0]
 X_POSIX       7   read      199    213674664716      1073741824    808.0719    809.3472  [  0]
```

Can you fix this / help me to get this working?

Thanks!!

Sam




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/darshan-users/attachments/20201013/598cfb58/attachment-0001.html>