[Mochi-devel] Serializing raw buffers in mercury

[Philip Davis]

Hi Matthieu,

That helps immensely. I had not considered that the user provides memory for the deserializer in margo_get_input.

Thanks,
Philip

> On Feb 6, 2020, at 4:34 PM, Dorier, Matthieu <mdorier at anl.gov> wrote:
> 
> The argument to the hg_proc_ function is a void* pointer which should be cast into a pointer to the structure you want to serialize/deserialize. The structure itself should be allocated by the called (usually it will be on the stack, though).
> 
> To take the example of kv_data_t, if you had an RPC taking as an argument such a data structure, you would have something like this:
> 
> char* buffer = ...;
> kv_data_t args = { .data = buffer; .size = bufferSize };
> margo_forward(h, &args);
> 
> margo_forward would call hg_proc_kv_data_t and, as you can see, while a pointer to the structure is passed to margo_forward, the structure itself is located on the stack. On the server side (in the RPC handler) you would have something like this:
> 
> kv_data_t args;
> margo_get_input(h, &args);
> ...
> margo_free_input(h, &args);
> 
> as you can see, once again the structure itself is already on the stack so the deserialization function doesn't have to allocate it. However the deserialization function does have to allocate the inner data buffer, and margo_free_input will call free() on this buffer later on.
> 
> I hope this helps.
> Matthieu
> 
> On 06/02/2020, 11:54, "Philip Davis" <philip.e.davis at rutgers.edu> wrote:
> 
>    Great, thanks for the example! One question: is memory being allocated for the second argument to hg_proc_* before it is called? I was thinking that on a DECODE operation that argument would be uninitialized, but in the KV example you sent, arg is being cast to a pointer to a kv_data_t struct, which is then dereferenced. Is it doing a malloc(sizeof(kv_data_t) or similar before the call?
> 
>> On Feb 6, 2020, at 12:45 PM, Dorier, Matthieu <mdorier at anl.gov> wrote:
>> 
>> Hi Philip,
>> 
>> Here is an example of how it's done in one of our services:
>> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxgitlab.cels.anl.gov%2Fsds%2Fsds-keyval%2Fblob%2Fmaster%2Fsrc%2Fsdskv-rpc-types.h%23L18&data=02%7C01%7Cphilip.e.davis%40rutgers.edu%7C928fdba90e964e9bba1208d7ab4c5b4a%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637166216753927847&sdata=xTkEWE4O%2BWScvfvJ3Pa0iPNiKRy0967UtnrSWIJgkx0%3D&reserved=0
>> 
>> Thanks,
>> 
>> Matthieu
>> 
>> On 06/02/2020, 10:12, "mochi-devel on behalf of Philip Davis" <mochi-devel-bounces at lists.mcs.anl.gov on behalf of philip.e.davis at rutgers.edu> wrote:
>> 
>>   Hello,
>> 
>>   I have a need to serialize a simple raw buffer (void * and length.) I’m looking at the “serializing complex data structures” page on the Margo documentation, and I want to make sure I’m doing this right. Should I be writing an hg_proc_* function that serializes the hg_size_t and uses hg_proc_raw to copy the buffer, or is there some existing hg_proc_* function I can leverage to encode/decode the buffer/length pair in a single call? It doesn’t look to me like hg_proc_raw can do this, since the length is copied by value.
>> 
>>   Thanks,
>>   Philip
>>   _______________________________________________
>>   mochi-devel mailing list
>>   mochi-devel at lists.mcs.anl.gov
>>   https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mcs.anl.gov%2Fmailman%2Flistinfo%2Fmochi-devel&data=02%7C01%7Cphilip.e.davis%40rutgers.edu%7C928fdba90e964e9bba1208d7ab4c5b4a%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637166216753927847&sdata=TxRCaUkmLOAdDuszdn%2ByEP9mJ7ETJc2JE%2BnbWtT2Z2A%3D&reserved=0
>>   https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mcs.anl.gov%2Fresearch%2Fprojects%2Fmochi&data=02%7C01%7Cphilip.e.davis%40rutgers.edu%7C928fdba90e964e9bba1208d7ab4c5b4a%7Cb92d2b234d35447093ff69aca6632ffe%7C1%7C0%7C637166216753927847&sdata=api6aqeZMQhcB4xMM%2Fs%2FXBFgHASyUqNJPerVvn0AiPk%3D&reserved=0
>> 
>> 
> 
> 
>