10 Protobuf Versioning Best Practices

Protocol buffers (Protobuf) is a popular data serialization format that enables developers to exchange data between different systems. While Protobuf is a great tool, it can be challenging to keep track of different versions of your data schema.

In this article, we will share 10 best practices for versioning your Protobuf data schema. By following these best practices, you can avoid common versioning pitfalls and make it easier to manage different versions of your data schema.

1. Don’t change the meaning of a field

Suppose you have a field called “user_id” in your protobuf. In version 1.0.0 of your API, the user_id field is defined as the unique identifier for a user in your database. In version 2.0.0 of your API, you change the meaning of the user_id field to be the user’s social security number.

Now, any client that was using the user_id field in version 1.0.0 of your API will break when they try to use it in version 2.0.0, because the meaning of the field has changed.

To avoid this type of breaking change, always keep the meaning of a field the same when you version your protobufs.

2. Add new fields to the end of messages

When you add a new field to the end of a message, it doesn’t break compatibility with older versions of the message. That’s because protobuf uses varint encoding for fields, which means that each field is encoded as a variable-length integer. The length of the integer depends on the value of the field.

So, if you add a new field to the end of a message, and that field has a value of 0, it will take up less space than any other field in the message. This is because 0 is encoded as a single byte. However, if you add a new field to the middle of a message, it will break compatibility with older versions, because the order of the fields matters.

Therefore, when you’re adding new fields to a message, always add them to the end. This will ensure that your messages are backward-compatible.

3. Use oneof for optional fields

If you have an optional field in your protobuf message, and you want to change the type of that field, you can use oneof. Oneof allows you to define a field as being one of a number of different types. So if you want to change an optional int32 field to an optional string field, you can do so by using oneof like this:

message Foo {
oneof bar {
int32 baz = 1;
string qux = 2;
}
}

This is much easier than having to create a new field and deprecate the old field. It also has the added benefit of making it clear to anyone reading the code that the field can be either an int32 or a string.

4. Avoid changing types

When you change a type, it’s a breaking change for everyone who is using that type. That means that if you change the type of an existing field from string to int, anyone who was using that field will have to update their code to account for the new type.

It’s much better to add a new field with the new type than to change the type of an existing field. That way, old code can still continue to work without any changes, and new code can start using the new field.

5. Keep old code working with deprecated fields

Suppose you have a message type like this:

message Foo {
optional string name = 1;
}

And you want to change the field to be called “first_name” instead. If you simply rename the field, old code that’s still using the “name” field will break. Instead, you should do this:

message Foo {
optional string first_name = 1;
optional string name = 2 [deprecated=true];
}

This way, old code can still use the “name” field (although it will get a warning from the compiler), and new code can use the “first_name” field. When all the old code has been updated to use the new field, you can remove the deprecated field.

6. Version your API in multiple ways

When you make a breaking change to your API, it’s important to version not only the individual Protobuf files that have changed, but also the overall API. That way, clients can choose to update to the new version of the API, or they can continue using the old version and just update the individual Protobuf files that have changed.

It’s also important to use a consistent naming scheme for your versions. For example, you might use “v1”, “v2”, etc. for major versions, and “beta1”, “beta2”, etc. for minor versions. This will help clients know at a glance which version they should be using.

Finally, don’t forget to document your API! Include a description of each version, what has changed, and how clients can update.

7. Consider using custom options

When adding a new field to a message, you have two choices: make the field required or optional. If you make it required, then existing code that tries to parse the new version of the message will fail. If you make it optional, then existing code can still parse the new message, but will just ignore the new field.

There are times when making a field required is the right choice. For example, if the field is used in a critical part of the system, then not having the field would be a problem. However, there are also times when making a field optional is the better choice. For example, if the field is only used for logging or debugging, then not having the field wouldn’t be a big deal.

If you’re not sure which choice to make, then consider using a custom option. With a custom option, you can add the new field as an optional field, but also set a flag indicating that the field is required. That way, if you ever need to change the field to be required, you can do so without breaking backwards compatibility.

8. Update documentation and examples

When you make changes to your .proto files, it’s important that you also update any documentation or examples that use those types. Otherwise, people who are trying to use your types will get confused and might not be able to figure out how to use the new version of your types.

It can be tempting to skip this step, especially if you’re in a hurry, but it’s really important to take the time to do it right. Your users will appreciate it, and it will save you a lot of headaches down the road.

9. Write tests that check backwards compatibility

Suppose you have a service that uses protobufs for its API. Your service has been running for a while and has accumulated a lot of users. You want to release a new version of your protobufs, but you’re worried about breaking backwards compatibility and causing problems for your users.

The best way to avoid this problem is to write tests that check backwards compatibility. That way, you can be confident that your new protobufs will work with the old ones.

There are two ways to do this:

1. Use the protoc command to generate both the old and the new versions of your protobufs, and compare the generated code.

2. Use the Google Protocol Buffers library to serialize and deserialize both the old and the new versions of your protobufs, and compare the results.

Both approaches have their pros and cons, but in general, the second approach is more comprehensive and less error-prone.

10. Be careful when deleting fields

Suppose you have a message type like this:

message Foo {
optional string name = 1;
}

And you want to delete the name field in version 2. You might be tempted to do something like this:

message Foo {
optional string name = 1 [deprecated=true];
}

However, this is not the correct way to do it. The reason is that when protobufs are deserialized, fields that are marked as deprecated are still parsed and stored in memory. So if you were to receive a protobuf of type Foo with a name field, it would still be parsed and stored in memory, even though it’s marked as deprecated.

The correct way to delete a field is to remove it from the message type altogether:

message Foo {
}

By doing this, the protobuf will no longer be able to parse the name field, and any protobufs received with that field will cause an error.