Protobuf Backwards compatibility

APIs are fundamentally contracts with users, and users often write code against APIs that is then launched into a production service with the expectation that it continues to work (unless the API has a stability level that indicates otherwise). Therefore, it is important to understand what constitutes a backwards compatible change and what constitutes a backwards incompatible change.

Guidance

Existing client code must not be broken by a service updating to a new minor or patch release. Old clients must be able to work against newer servers (with the same major version number).

Important

It is not always clear whether a change is compatible or not. The guidance here should be treated as indicative, rather than as a comprehensive list of every possible change.

There are three distinct types of compatibility to consider:

1. Source compatibility: Code written against a previous version must compile against a newer version, and successfully run with a newer version of the client library.
2. Wire compatibility: Code written against a previous version must be able to communicate correctly with a newer server. In other words, not only are inputs and outputs compatible, but the serialization and deserialization expectations continue to match.
3. Semantic compatibility: Code written against a previous version must continue to receive what most reasonable developers would expect. (This can be tricky in practice, however, and sometimes determining what users will expect can involve a judgment call.)

Note

This guidance assumes that APIs are intended to be called from a range of consumers, written in multiple languages and with no control over how and when consumers update. Any API which has a more limited scope (for example, an API which is only called by client code written by the same team as the API producer, or deployed in a way which can enforce updates) should carefully consider its own compatibility requirements.

Adding components

In general, new components (interfaces, methods, messages, fields, enums, or enum values) may be added to existing APIs in the same major version.

However, keep the following guidelines in mind when doing this:

• Code written against the previous surface (and thus is unaware of the new components) must continue to be treated the same way as before.
  • New required fields must not be added to existing request messages or resources.
  • Any field being populated by clients must have a default behavior matching the behavior before the field was introduced.
  • Any field previously populated by the server must continue to be populated, even if it introduces redundancy.
• For enum values specifically, be aware that it is possible that user code does not handle new values gracefully.
  • Enum values may be freely added to enums which are only used in request messages.
  • Enums that are used in response messages or resources and which are expected to receive new values should document this. Enum values still may be added in this situation; however, appropriate caution should be used.

Note

It is possible when adding a component closely related to an existing component (for example, string foo_value when string foo already exists) to enter a situation where generated code will conflict. Service owners should be aware of subtleties in the tooling they or their users are likely to use (and tool authors should endeavor to avoid such subtleties if possible).

Removing or renaming components

Existing components (interfaces, methods, messages, fields, enums, or enum values) must not be removed from existing APIs in the same major version. Removing a component is a backwards incompatible change.

Important

Renaming a component is semantically equivalent to “remove and add”. In cases where these sorts of changes are desirable, a service may add the new component, but must not remove the existing one. In situations where this can allow users to specify conflicting values for the same semantic idea, the behavior must be clearly specified.

Moving components between files

Existing components must not be moved between files.

Moving a component from one proto file to another within the same package is wire compatible, however, the code generated for languages like C++ or Python will result in breaking change since import and #include will no longer point to the correct code location.

Moving into oneofs

Existing fields must not be moved into or out of a oneof. This is a backwards-incompatible change in the Go protobuf stubs.

Changing the type of fields

Existing fields and messages must not have their type changed, even if the new type is wire-compatible, because type changes alter generated code in a breaking way.

Changing resource paths

A resource must not change its path.

Unlike most breaking changes, this affects major versions as well: in order for a client to expect to use v2.0 access a resource that was created in v1.0 or vice versa, the same resource name must be used in both versions.

More subtly, the set of valid resource paths should not change either, for the following reasons:

• If resource name formats become more restrictive, a request that would previously have succeeded will now fail.
• If resource name formats become less restrictive than previously documented, then code making assumptions based on the previous documentation could break. Users are very likely to store resource names elsewhere, in ways that may be sensitive to the set of permitted characters and the length of the name. Alternatively, users might perform their own resource name validation to follow the documentation.
  • For example, Amazon gave customers a lot of warning and had a migration period when they started allowing longer EC2 resource IDs.

Semantic changes

Code will often depend on API behavior and semantics, even when such behavior is not explicitly supported or documented. Therefore, APIs must not change visible behavior or semantics in ways that are likely to break reasonable user code, as such changes will be seen as breaking by those users.

Note

This does involve some level of judgment; it is not always clear whether a proposed change is likely to break users, and an expansive reading of this guidance could ostensibly prevent any change (which is not the intent).

Default values must not change

Default values are the values set by servers for resources when they are not specified by the client. This section only applies to static default values within fields on resources and does not apply to dynamic defaults such as the default IP address of a resource.

Changing the default value is considered breaking and must not be done. The default behavior for a resource is determined by its default values, and this must not change across minor versions.

For example:

message Book {
    // google.api.resource and other annotations and fields

    // The genre of the book
    // If this is not set when the book is created, the field will be given a value of FICTION.
    enum Genre {
      UNSPECIFIED = 0;
      FICTION = 1;
      NONFICTION = 2;
    }
}

Changing to:

message Book {
    // google.api.resource and other annotations and fields

    // The genre of the book
    // If this is not set when the book is created, the field will be given a value of NONFICTION.
    enum Genre {
      UNSPECIFIED = 0;
      FICTION = 1;
      NONFICTION = 2;
    }
}

would constitute a breaking change.

Serializing defaults

APIs must not change the way a field with a default value is serialized. For example if a field does not appear in the response if the value is equal to the default, the serialization must not change to include the field with the default. Clients may depend on the presence or absence of a field in a resource as semantically meaningful, so a change to how serialization is done for absent values must not occur in a minor version.

Consider the following proto, where the default value of wheels is 2:

// A representation of an automobile
message Automobile {
    // google.api.resource and other annotations and fields

    // The number of wheels on the automobile.
    // The default value is 2, when no value is sent by the client.
    int wheels = 2;
}

First the proto serializes to JSON when the value of wheels is 2 as follows:

{
  "name": "my-car"
}

Then, the API service changes the serialization to include wheel even if the value is equal to the default value, 2 as follows:

{
  "name": "my-car",
  "wheels": 2
}

This constitutes a change that is not backwards compatible within a major version.

Further reading

• For compatibility around field behavior, see [AEP-203][].
• For compatibility around pagination, see [AEP-158][].
• For compatibility around long-running operations, see [AEP-151][].
• For understanding stability levels and expectations, see [AEP-181][].