[MOAB-dev] parallel write and restart with variable length tags
Grindeanu, Iulian R.
iulian at mcs.anl.gov
Wed Dec 5 13:48:04 CST 2018
Hi Lukasz,
Thanks for explanations;
My guess is that the parallel writer has an issue with variable length tags
(not sure yet, I did not try)
when we write in parallel, we order the entities for these tags; if the same variable length tag appears on multiple types, the ordering probably fails (we have to order in parallel, but I think we just concatenate by processor, in the sense that each processor writes its var len tags in one array; as entities can belong to different processes, they will end up unordered in the larger dataset in the file; ) My guess is that the reader can be changed to read var len tags if entities are not ordered, but maybe it is not worth the effort
the quick solution is to give a different name by entity type; my understanding from your message is that this approach works for you
are you saving the piola stresses at integration points or at other nodes?
Your approximation order changes by element / face? What kind of formulation do you use?
I should probably read your code :)
Thanks,
Iulian
________________________________________
From: Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk>
Sent: Wednesday, December 5, 2018 11:54:28 AM
To: MOAB dev
Cc: Grindeanu, Iulian R.
Subject: Re: [MOAB-dev] parallel write and restart with variable length tags
Hello,
Thanks for the answer.
The code is quite complicated, is a mix formulation with six different fileds, I can easily reproduce the error, but not in a simple way. And the error is triggered for this particular configuration for more than one processor. For all other works.
In the meantime, I fixed the problem, by doing what you suggested, i.e.sperated tags, on variable tags and fixes size tags. However, the error is not exactly what you suggested. I have to use variable tags because the order of approximation is arbitrary can change during analysis.
The problem is when the variable tag is on vertices and other dimension entities. There is other filed, which is not picked up by debugging, which is on vertices, edges, which trigger the error. If for that field, tag on vertices is separated from all other entities, all works correctly.
In meantime, I changed the internal code, which was accidentally changed for the good. For special approximation which I have, tag on the nodes is always fixed size, so that split of tags makes perfect sense. And for some cases I can make the tag on vertices dense and exploit that fact.
Nevertheless, the problem is somewhere in the code. I will look at it and let you know if I learn more. Anyway, Moab can do what no other code can with tags :) I like it a lot.
BTW _App_DataP is a field of Piola Stress in HDiv space, that why is on Triangles (faces) and Volumes (tets).
Kind regards,
Lukasz
> On 5 Dec 2018, at 16:26, Grindeanu, Iulian R. <iulian at mcs.anl.gov> wrote:
>
> Hi Lukasz,
> Thanks for reporting the problem;
> Sorry I missed your message
>
> I cannot read the file at all in MOAB, even in serial
> I do get the same error;
> it is related to the fact that tag _App_DataP is variable length and it is defined on tetras and edges at the same time
>
> The list in the file starts with tetras and then edges, I am not sure why the reader does not like it;
>
> Can you separate the tag on edges and tetras? Or at least, start with edges, then with tetras, that might get over the hoop.
> it is possible that the writer does not rearrange correctly, if you write in parallel, it should have arranged them by type
>
> Is it important that it is the same name on both edges and tetras? Can you share the code that writes the file, or is it too complicated?
>
> Not sure what is the significance of that tag, _App_DataP
>
> probably it is not the only tag with this problem, I did not dig deeper.
>
> Iulian
>
> ________________________________________
> From: moab-dev <moab-dev-bounces at mcs.anl.gov> on behalf of Lukasz Kaczmarczyk via moab-dev <moab-dev at mcs.anl.gov>
> Sent: Tuesday, December 4, 2018 1:59:20 AM
> To: MOAB dev
> Cc: Lukasz Kaczmarczyk
> Subject: Re: [MOAB-dev] parallel write and restart with variable length tags
>
> Hello,
>
> I apologise for this overflow of information, I know something more, this error is triggered on 0 proc, when I have a variable tag on vertices and edges.
>
> 0 H5M (0.03 s) Reading tag "_App_Datax"
> 0 H5M *********** Debug Barrier 51 (@2706)***********
> 1 H5M (0.03 s) Reading tag "_App_Datax"
> 1 H5M *********** Debug Barrier 51 (@2706)***********
> 0 H5M Reading var-len sparse data for tag "_App_Datax"
> 1 H5M Reading var-len sparse data for tag "_App_Datax"
> 0 H5M Reading file ids for sparse tag "_App_Datax" in 1 chunks
> 0 H5M Reading chunk 1 of "_App_Datax" IDs
> 1 H5M Reading file ids for sparse tag "_App_Datax" in 1 chunks
> 1 H5M Reading chunk 1 of "_App_Datax" IDs
> 1 H5M Switching to unordered list for tag handle list
> [1]MOAB ERROR: --------------------- Error Message ------------------------------------
> [1]MOAB ERROR: Unordered file ids for variable length tag not supported!
> [1]MOAB ERROR: read_var_len_tag() line 3335 in src/io/ReadHDF5.cpp
> [1]MOAB ERROR: --------------------- Error Message ------------------------------------
> [1]MOAB ERROR: ReadHDF5 Failure!
> [1]MOAB ERROR: read_tag() line 2769 in src/io/ReadHDF5.cpp
> [1]MOAB ERROR: --------------------- Error Message ------------------------------------
> [1]MOAB ERROR: ReadHDF5 Failure!
> [1]MOAB ERROR: load_file_partial() line 1164 in src/io/ReadHDF5.cpp
> 1 H5M (0.04 s) Cleaning up
>
> In particular note, that proc 1 is ok, the problem is on proc 0. Mesh is partitioned with metis and entities are on both processors for both dimensions.
>
> I reading fike with options:
>
> option = "PARALLEL=READ_PART;"
> "PARALLEL_RESOLVE_SHARED_ENTS;"
> "PARTITION=PARALLEL_PARTITION;DEBUG_IO=5";
>
>
> Is this something wrong on my side, or rather need to be fixed in moab reader. I can have a go, but need to have some hint how to fix this.
>
> Lukasz
>
>
> On 4 Dec 2018, at 00:27, Lukasz Kaczmarczyk via moab-dev <moab-dev at mcs.anl.gov<mailto:moab-dev at mcs.anl.gov>> wrote:
>
> Hello,
>
> If that will help, this is debugging IO trace,
>
> H5M (0.02 s) Reading all nodes...
> H5M Reading 0 chunks for coordinate coordinates
> H5M (0.02 s) Reading chunk 1 of node coords
> H5M Closing node coordinate table
> H5M Updating ID to handle map for 59 nodes
> H5M (0.02 s) Reading all element connectivity...
> H5M (0.02 s) READING Edge2 CONNECTIVITY (236 elems in 1 selects)
> H5M Reading connectivity in 0 chunks for element group "Edge2"
> H5M Reading chunk 1 for "Edge2"
> H5M (0.02 s) READING Tet4 CONNECTIVITY (132 elems in 1 selects)
> H5M Reading connectivity in 0 chunks for element group "Tet4"
> H5M Reading chunk 1 for "Tet4"
> H5M (0.02 s) READING Tri3 CONNECTIVITY (310 elems in 1 selects)
> H5M Reading connectivity in 0 chunks for element group "Tri3"
> H5M Reading chunk 1 for "Tri3"
> H5M (0.02 s) Reading all sets...
> H5M Reading chunk 1 of set contents
> H5M Reading chunk 1 of set children
> H5M Reading chunk 1 of set parents
> H5M (0.02 s) Reading all adjacencies...
> H5M Reading adjacency list in 1 chunks
> H5M Reading chunk 1 of adjacency list
> H5M Reading adjacency list in 1 chunks
> H5M Reading chunk 1 of adjacency list
> H5M Reading adjacency list in 1 chunks
> H5M Reading chunk 1 of adjacency list
> H5M (0.03 s) Reading all tags...
> H5M (0.03 s) Reading tag "BLOCK_ATTRIBUTES"
> H5M Reading var-len sparse data for tag "BLOCK_ATTRIBUTES"
> H5M Reading file ids for sparse tag "BLOCK_ATTRIBUTES" in 0 chunks
> H5M Reading chunk 1 of "BLOCK_ATTRIBUTES" IDs
> H5M Reading BLOCK_ATTRIBUTES offsets in 0 chunks
> H5M Reading chunk 1 of BLOCK_ATTRIBUTES offsets
> H5M Reading BLOCK_ATTRIBUTES values in 0 chunks
> H5M Reading chunk 1 of BLOCK_ATTRIBUTES values
> H5M (0.03 s) Reading tag "BLOCK_HEADER"
> H5M Reading sparse data for tag "BLOCK_HEADER"
> H5M Reading file ids for sparse tag "BLOCK_HEADER" in 0 chunks
> H5M Reading chunk 1 of "BLOCK_HEADER" IDs
> H5M Reading sparse values for tag "BLOCK_HEADER" in 0 chunks
> H5M Reading chunk 1 of "BLOCK_HEADER" values
> H5M (0.03 s) Reading tag "BOX_DIMS"
> H5M (0.03 s) Reading tag "CATEGORY"
> H5M Reading sparse data for tag "CATEGORY"
> H5M Reading file ids for sparse tag "CATEGORY" in 0 chunks
> H5M Reading chunk 1 of "CATEGORY" IDs
> H5M Reading sparse values for tag "CATEGORY" in 0 chunks
> H5M Reading chunk 1 of "CATEGORY" values
> H5M (0.03 s) Reading tag "DIRICHLET_SET"
> H5M (0.03 s) Reading tag "DIRICHLET_SET__BC_DATA"
> H5M (0.03 s) Reading tag "ElemType"
> H5M (0.03 s) Reading tag "GEOM_DIMENSION"
> H5M Reading sparse data for tag "GEOM_DIMENSION"
> H5M Reading file ids for sparse tag "GEOM_DIMENSION" in 0 chunks
> H5M Reading chunk 1 of "GEOM_DIMENSION" IDs
> H5M Reading sparse values for tag "GEOM_DIMENSION" in 0 chunks
> H5M Reading chunk 1 of "GEOM_DIMENSION" values
> H5M (0.03 s) Reading tag "GEOM_SENSE_2"
> H5M Reading sparse data for tag "GEOM_SENSE_2"
> H5M Reading file ids for sparse tag "GEOM_SENSE_2" in 0 chunks
> H5M Reading chunk 1 of "GEOM_SENSE_2" IDs
> H5M Reading sparse values for tag "GEOM_SENSE_2" in 0 chunks
> H5M Reading chunk 1 of "GEOM_SENSE_2" values
> H5M (0.03 s) Reading tag "GEOM_SENSE_N_ENTS"
> H5M Reading var-len sparse data for tag "GEOM_SENSE_N_ENTS"
> H5M Reading file ids for sparse tag "GEOM_SENSE_N_ENTS" in 0 chunks
> H5M Reading chunk 1 of "GEOM_SENSE_N_ENTS" IDs
> H5M Reading GEOM_SENSE_N_ENTS offsets in 0 chunks
> H5M Reading chunk 1 of GEOM_SENSE_N_ENTS offsets
> H5M Reading GEOM_SENSE_N_ENTS values in 0 chunks
> H5M Reading chunk 1 of GEOM_SENSE_N_ENTS values
> H5M (0.03 s) Reading tag "GEOM_SENSE_N_SENSES"
> H5M Reading var-len sparse data for tag "GEOM_SENSE_N_SENSES"
> H5M Reading file ids for sparse tag "GEOM_SENSE_N_SENSES" in 0 chunks
> H5M Reading chunk 1 of "GEOM_SENSE_N_SENSES" IDs
> H5M Reading GEOM_SENSE_N_SENSES offsets in 0 chunks
> H5M Reading chunk 1 of GEOM_SENSE_N_SENSES offsets
> H5M Reading GEOM_SENSE_N_SENSES values in 0 chunks
> H5M Reading chunk 1 of GEOM_SENSE_N_SENSES values
> H5M (0.03 s) Reading tag "GLOBAL_ID"
> H5M Read dense data block for tag "GLOBAL_ID" on "sets"
> H5M Reading dense data for tag "GLOBAL_ID data for sets" and group "sets" in 0 chunks
> H5M Reading chunk 1 of "GLOBAL_ID data for sets" data
> H5M Read dense data block for tag "GLOBAL_ID" on "nodes"
> H5M Reading dense data for tag "GLOBAL_ID data for nodes" and group "nodes" in 0 chunks
> H5M Reading chunk 1 of "GLOBAL_ID data for nodes" data
> H5M Read dense data block for tag "GLOBAL_ID" on "Edge2"
> H5M Reading dense data for tag "GLOBAL_ID data for Edge2" and group "Edge2" in 0 chunks
> H5M Reading chunk 1 of "GLOBAL_ID data for Edge2" data
> H5M Read dense data block for tag "GLOBAL_ID" on "Tet4"
> H5M Reading dense data for tag "GLOBAL_ID data for Tet4" and group "Tet4" in 0 chunks
> H5M Reading chunk 1 of "GLOBAL_ID data for Tet4" data
> H5M Read dense data block for tag "GLOBAL_ID" on "Tri3"
> H5M Reading dense data for tag "GLOBAL_ID data for Tri3" and group "Tri3" in 0 chunks
> H5M Reading chunk 1 of "GLOBAL_ID data for Tri3" data
> H5M (0.03 s) Reading tag "MATERIAL_SET"
> H5M Reading sparse data for tag "MATERIAL_SET"
> H5M Reading file ids for sparse tag "MATERIAL_SET" in 0 chunks
> H5M Reading chunk 1 of "MATERIAL_SET" IDs
> H5M Reading sparse values for tag "MATERIAL_SET" in 0 chunks
> H5M Reading chunk 1 of "MATERIAL_SET" values
> H5M (0.03 s) Reading tag "MOFEM_VERSION"
> H5M (0.03 s) Reading tag "NAME"
> H5M Reading sparse data for tag "NAME"
> H5M Reading file ids for sparse tag "NAME" in 0 chunks
> H5M Reading chunk 1 of "NAME" IDs
> H5M Reading sparse values for tag "NAME" in 0 chunks
> H5M Reading chunk 1 of "NAME" values
> H5M (0.03 s) Reading tag "NEUMANN_SET"
> H5M (0.03 s) Reading tag "NEUMANN_SET__BC_DATA"
> H5M (0.03 s) Reading tag "OBB"
> H5M (0.03 s) Reading tag "PARALLEL_PARTITION"
> H5M Reading sparse data for tag "PARALLEL_PARTITION"
> H5M Reading file ids for sparse tag "PARALLEL_PARTITION" in 0 chunks
> H5M Reading chunk 1 of "PARALLEL_PARTITION" IDs
> H5M Reading sparse values for tag "PARALLEL_PARTITION" in 0 chunks
> H5M Reading chunk 1 of "PARALLEL_PARTITION" values
> H5M Read dense data block for tag "PARALLEL_PARTITION" on "Edge2"
> H5M Reading dense data for tag "PARALLEL_PARTITION data for Edge2" and group "Edge2" in 0 chunks
> H5M Reading chunk 1 of "PARALLEL_PARTITION data for Edge2" data
> H5M Read dense data block for tag "PARALLEL_PARTITION" on "Tet4"
> H5M Reading dense data for tag "PARALLEL_PARTITION data for Tet4" and group "Tet4" in 0 chunks
> H5M Reading chunk 1 of "PARALLEL_PARTITION data for Tet4" data
> H5M Read dense data block for tag "PARALLEL_PARTITION" on "Tri3"
> H5M Reading dense data for tag "PARALLEL_PARTITION data for Tri3" and group "Tri3" in 0 chunks
> H5M Reading chunk 1 of "PARALLEL_PARTITION data for Tri3" data
> H5M (0.03 s) Reading tag "PARTITION"
> H5M (0.03 s) Reading tag "UNIQUE_ID"
> H5M Reading sparse data for tag "UNIQUE_ID"
> H5M Reading file ids for sparse tag "UNIQUE_ID" in 0 chunks
> H5M Reading chunk 1 of "UNIQUE_ID" IDs
> H5M Reading sparse values for tag "UNIQUE_ID" in 0 chunks
> H5M Reading chunk 1 of "UNIQUE_ID" values
> H5M (0.03 s) Reading tag "_App_DataBUBBLE"
> H5M Reading var-len sparse data for tag "_App_DataBUBBLE"
> H5M Reading file ids for sparse tag "_App_DataBUBBLE" in 0 chunks
> H5M Reading chunk 1 of "_App_DataBUBBLE" IDs
> H5M Reading _App_DataBUBBLE offsets in 0 chunks
> H5M Reading chunk 1 of _App_DataBUBBLE offsets
> H5M Reading _App_DataBUBBLE values in 0 chunks
> H5M Reading chunk 1 of _App_DataBUBBLE values
> H5M (0.03 s) Reading tag "_App_DataG"
> H5M Reading var-len sparse data for tag "_App_DataG"
> H5M Reading file ids for sparse tag "_App_DataG" in 0 chunks
> H5M Reading chunk 1 of "_App_DataG" IDs
> H5M Reading _App_DataG offsets in 0 chunks
> H5M Reading chunk 1 of _App_DataG offsets
> H5M Reading _App_DataG values in 0 chunks
> H5M Reading chunk 1 of _App_DataG values
> H5M (0.03 s) Reading tag "_App_DataG0"
> H5M Reading var-len sparse data for tag "_App_DataG0"
> H5M Reading file ids for sparse tag "_App_DataG0" in 0 chunks
> H5M Reading chunk 1 of "_App_DataG0" IDs
> H5M Reading _App_DataG0 offsets in 0 chunks
> H5M Reading chunk 1 of _App_DataG0 offsets
> H5M Reading _App_DataG0 values in 0 chunks
> H5M Reading chunk 1 of _App_DataG0 values
> H5M (0.03 s) Reading tag "_App_DataP"
> H5M Reading var-len sparse data for tag "_App_DataP"
> H5M Reading file ids for sparse tag "_App_DataP" in 0 chunks
> H5M Reading chunk 1 of "_App_DataP" IDs
> H5M Switching to unordered list for tag handle list
> [0]MOAB ERROR: --------------------- Error Message ------------------------------------
> [0]MOAB ERROR: Unordered file ids for variable length tag not supported!
> [0]MOAB ERROR: read_var_len_tag() line 3335 in src/io/ReadHDF5.cpp
> [0]MOAB ERROR: --------------------- Error Message ------------------------------------
> [0]MOAB ERROR: ReadHDF5 Failure!
> [0]MOAB ERROR: read_tag() line 2769 in src/io/ReadHDF5.cpp
> [0]MOAB ERROR: --------------------- Error Message ------------------------------------
> [0]MOAB ERROR: ReadHDF5 Failure!
> [0]MOAB ERROR: load_file_impl() line 678 in src/io/ReadHDF5.cpp
> H5M (0.03 s) Cleaning up
> H5M (0.03 s) READ FAILED (ERROR CODE MB_FAILURE): ReadHDF5 Failure
> [0]MOAB ERROR: --------------------- Error Message ------------------------------------
> [0]MOAB ERROR: Failed to load file after trying all possible readers!
> [0]MOAB ERROR: serial_load_file() line 623 in src/Core.cpp
> [0]MOAB ERROR: --------------------- Error Message ------------------------------------
> [0]MOAB ERROR: Failed in step PARALLEL READ!
> [0]MOAB ERROR: load_file() line 612 in src/ReadParallel.cpp
> [0]MOAB ERROR: load_file() line 263 in src/ReadParallel.cpp
> [0]MOAB ERROR: load_file() line 503 in src/Core.cpp
> [0]MoFEM ERROR: --------------------- MoFEM Error Message ---------------------------------------------------
> [0]MoFEM ERROR: MoFEM version 0.8.17 (MOAB 5.0.2 Petsc Release Version 3.9.3, Jul, 02, 2018 )
> [0]MoFEM ERROR: MoFEM git commit id 9fe99b7e4d55b4d664a369dec823ed6f828cab58
> [0]MoFEM ERROR: See http://mofem.eng.gla.ac.uk/mofem/html/guidelines_bug_reporting.html for bug reporting.
> [0]MoFEM ERROR: Write to https://groups.google.com/forum/#!forum/mofem-group to seek help.
> [0]MoFEM ERROR: MOAB error (16) MB_FAILURE
> [0]MoFEM ERROR: #1 main() line 59 in /Users/likask/MyBuild/mofem-cephas/mofem/users_modules/eshelbian_plasticty/ep.cpp
>
>
>
> On 3 Dec 2018, at 22:23, Lukasz Kaczmarczyk <Lukasz.Kaczmarczyk at glasgow.ac.uk<mailto:Lukasz.Kaczmarczyk at glasgow.ac.uk>> wrote:
>
> This is for partitioning for two processors, and I have variable length tags on vertices, edges, faces and volumes.
>
> On 3 Dec 2018, at 22:15, Lukasz Kaczmarczyk via moab-dev <moab-dev at mcs.anl.gov<mailto:moab-dev at mcs.anl.gov>> wrote:
>
> Hello,
>
> I have this error for parallel when I write, then read in parallel, could you advise how to fix this error,
>
> [1]MOAB ERROR: --------------------- Error Message ------------------------------------
> [1]MOAB ERROR: Unordered file ids for variable length tag not supported!
> [1]MOAB ERROR: read_var_len_tag() line 3335 in src/io/ReadHDF5.cpp
> [1]MOAB ERROR: --------------------- Error Message ------------------------------------
> [1]MOAB ERROR: ReadHDF5 Failure!
> [1]MOAB ERROR: read_tag() line 2769 in src/io/ReadHDF5.cpp
> [1]MOAB ERROR: --------------------- Error Message ------------------------------------
> [1]MOAB ERROR: ReadHDF5 Failure!
> [1]MOAB ERROR: load_file_partial() line 1164 in src/io/ReadHDF5.cpp
>
>
>
> Kind regards,
> Lukasz
>
> <restart_1.h5m>
More information about the moab-dev
mailing list