Data serialization for embedded platforms: ASN.1, JSON, Protocol Buffers....

It's a long time I work with embedded platforms, however I never used a 
well-defined and "standard" data serialization format. I often had the 
need to exchange data between platforms (Wifi, Ethernet, RS485...) and I 
always find a custom solution.

I perfectly understood the problem with this approach. Keeping the two 
(or more) side synchronized is very difficult and error-prone. When the 
sides are designed and maintaned by a single person/company, this 
approach could be sufficient. However I started to interface with other 
people, companies, very difficult techologies.

For example, I'm working on a IoT device that connects with AWS through 
MQTT protocol. I have to decide the data format of MQTT packets. They 
will be processed by AWS services, so JSON seems the perfect choice 
(maybe someone remember an old post where I was asking how to use a JSON 
in C for embedded platforms).
However JSON, considering it's a schema-less data format, is very 
complex to manage in C without using dynamic memory. So I'm thinking to 
use another data format, fithing with guys that will work with AWS, 
Javascript, Typescript, and so on (they doesn't understand why JSON 
isn't good on my side).

ASN.1 (mainly in uPER mode) and protobuf could be a good choice. However 
the backend/frontend guys don't know ASN.1, some has heard about protobuf.
I could insist to use ASN.1, however I'm not sure it is a good choice. 
And I don't know if they will be able to decode uPER messages in 
Typescript (for example). It seems they found something for BER, nothing 
for PER.

So this post just to discuss and share some experience on this topic.

pozz schreef:
> It's a long time I work with embedded platforms, however I never used a 
> well-defined and "standard" data serialization format. I often had the 
> need to exchange data between platforms (Wifi, Ethernet, RS485...) and I 
> always find a custom solution.
> 
> I perfectly understood the problem with this approach. Keeping the two 
> (or more) side synchronized is very difficult and error-prone. 

Or perhaps you're working with unreliable and/or incompetent people?  :P

> When the 
> sides are designed and maintaned by a single person/company, this 
> approach could be sufficient. However I started to interface with other 
> people, companies, very difficult techologies.
 >
> For example, I'm working on a IoT device that connects with AWS through 
> MQTT protocol. I have to decide the data format of MQTT packets. They 
> will be processed by AWS services, so JSON seems the perfect choice 
> (maybe someone remember an old post where I was asking how to use a JSON 
> in C for embedded platforms).
> However JSON, considering it's a schema-less data format, is very 
> complex to manage in C without using dynamic memory. 

If you think it's "very complex" then perhaps you're taking a suboptimal 
approach.  Even in a schema-less data format you can agree on how to lay 
out the necessary information.  If you have an event-based JSON parser 
then it should be pretty easy (even with static allocation).

> So I'm thinking to 
> use another data format, fithing with guys that will work with AWS, 
> Javascript, Typescript, and so on (they doesn't understand why JSON 
> isn't good on my side).
>
> ASN.1 (mainly in uPER mode) and protobuf could be a good choice. However 
> the backend/frontend guys don't know ASN.1, some has heard about protobuf.
> I could insist to use ASN.1, however I'm not sure it is a good choice. 
> And I don't know if they will be able to decode uPER messages in 
> Typescript (for example). It seems they found something for BER, nothing 
> for PER.
> 
> So this post just to discuss and share some experience on this topic.