Protocol Buffers 101


2.7k words • 16 min read
programming › development
, ,

Since we are now aware of the When & Why of gRPC, we can dive into Protocol Buffers. Protocol Buffers is a simple language-neutral and platform-neutral Interface Definition Language (IDL) for defining data structure schemas and programming interfaces. We will refer to Protocol Buffers as Protobuf

The ProtoBuf logo
The ProtoBuf logo

Intro to ProtoBuf

The story of Protobuf dates back to the 2000s! At that time, XML was the common format for API communication but XML was slow, verbose, and heavy. At the same time, Google was dealing with massive distributed systems, huge amounts of data & multiple services needing to talk to each other efficiently. Google’s scale demanded something which should be smaller, faster, schema-driven, and language-agnostic.

So Google engineers built an internal solution which was Protocol Buffers which was then known as Proto1. This internal version became the backbone of communication across Google’s infrastructure. In July 2008, Google open-sourced version 2 of ProtoBuf for everyone which was called Proto2.

Around 2015, with the release of gRPC, Google also released the latest version of Protos which is Proto3.

Protobuf have exploded in popularity and is gaining rapid adoption because it’s:

  • Compact and extremely fast
  • Schema-driven (no guessing about structure)
  • Has great backward/forward compatibility
  • A First-class fit for gRPC
  • Supported in nearly every language

Even though REST & JSON still dominates public APIs, Protobuf remains the go-to choice for high-performance, internal service communication.

Building the Mental Model

Switching from JSON or XML to Protocol Buffers isn’t just a syntax change, it’s a mindset change. Let’s first have a look at a simple Person object representation in XML, JSON and Protobuf. 

1
2
3
4
5
6
7
<!--  XML representation of the Person object -->
<user>
  <id>101</id>
  <name>Ashok Dey</name>
  <email>[email protected]</email>
  <is_active>true</is_active>
</user>
1
2
3
4
5
6
7
// JSON represent of the Person object
{ 
  "id": 101,
  "name": "Ashok Dey", 
  "email": "[email protected]", 
  "is_active": true 
}
1
2
3
4
5
6
7
8
9
10
11
// Protobuf schema definition of the Person object

syntax = "proto3"; // <-- the version of proto

message MessageFromJson {
    int32 id = 1;
    string name = 2;
    string email = 3;
    bool is_active = 4;
}

1
2
3
4
5
# Protobuf binary value for the JSON/XML Person representation
08 65 
12 09 41 73 68 6f 6b 20 44 65 79 
1a 0f 61 64 40 61 73 68 6f 6b 64 65 79 2e 63 6f 6d 
20 01

What can you deduce from the above representation? A few differences are prominent and few are not reflected directly. Let’s understand the differences one by one.

  • The foremost difference is that JSON/XML is human optimized whereas Protobuf is optimized for machines.
  • While in JSON/XML, you can add/remove fields as per required, the Protobuf schema must be versioned making it strict and not flexible.
  • For serialization & deserialization, built in parsers can be used for JSON/XML but for Protobuf, the protoc compiler generates strongly typed classes first.
  • From TagName to Tag Number: Field names are the identity of the data in JSON/XML while for Protobuf, field numbers are the true identity. The field names can change but numbers cannot.
  • For Protobuf, field names aren’t transmitted and the encoding is designed for speed & compactness while field names get repeated in every JSON/XML message.

JSON/XML treat data as self-describing text documents while ProtoBuf treats data as compact binary messages governed by a strict typed schema.

Now that we have some idea of using Protobuf, it’s always fun if we start learning by building something small but meaningful!

Elements of Protobuf

Ok, lets replicate a simple Todo App data modelling using the Protobuf. We will incrementally cover all the basic building blocks of Protobuf step by step.

Defining message

Whatever you want to represent in Protobuf has to be done as a message. You need to create a .proto file, todo.proto in this case. The first line of any protobuf file should contain the version of proto being used. We will use the latest version e.i. v3. 

1
2
3
4
5
6
7
8
9
10
11
12
// contents of todo.proto

// first line will help the compiler to identify the version of Protobuf.
syntax = "proto3";

// let's define the most basic todo any app can have 
message Todo {
  int64 id = 1;
  string title = 2;
  optional string description = 3;
  bool done = 4;
}

FieldName and FieldNumber

The properties of an object are what we called fields in a protobuf message. Every field name should be assigned a Field Number which should be unique and cannot change in future (for backwards compatibility, we will look at this later). The field number is what used for serialization of messages unlike the filed names used in JSON/XML.

1
2
3
4
5
6
7
8
syntax = "proto3";

message Todo {
  int64 id = 1;                       // <--- FieldName: "id", FieldNumber: 1
  string title = 2;                   // <--- Every field should have a data type for strict type checking
  optional string description = 3;    // <--- Fields can be `optional` or `required` 
  bool done = 4;
}

There are certain rules associated to the field names and field numbers:

  • The given number must be unique among all fields for that message.
  • Field numbers ranges from 1 to 2,147,483,647 but 19,000 to 19,999 are reserved for the internal implementations.
  • You cannot use any previously reserved field numbers
  • Field numbers should never be reused.
  • Never take a field number out of the reserved list for reuse with a new field definition
  • Unlike JSON/XML, the Protobuf use the field number (here 1) instead of the field name (i.e. id).

Scalar Types

Protocol Buffers supports a set of scalar types, similar to the basic data types in programming languages. These types define integers, floats, booleans, bytes, and strings. This facilitates Protobuf to strictly check and share data. More on data-types in Protobuf

Category Type Description
Signed Integers int32 32-bit signed integer
int64 64-bit signed integer
sint32 32-bit signed integer, optimized for negative values (ZigZag encoding)
sint64 64-bit signed integer, optimized for negative values (ZigZag encoding)
———————— ———- ———————————————————————-
Unsigned Integers uint32 32-bit unsigned integer
uint64 64-bit unsigned integer
———————— ———- ———————————————————————-
Fixed-Width Integers fixed32 Always 4 bytes; efficient for large unsigned 32-bit values
fixed64 Always 8 bytes; efficient for large unsigned 64-bit values
sfixed32 4-byte fixed signed 32-bit integer
sfixed64 8-byte fixed signed 64-bit integer
———————— ———- ———————————————————————-
Floating Point float 32-bit floating-point number
double 64-bit floating-point number
———————— ———- ———————————————————————-
Boolean bool Boolean value (true/false)
String & Bytes string UTF-8 encoded string
bytes Arbitrary raw byte sequence
———————— ———- ———————————————————————-

Enums

Enums in Protobuf allow us to define a set of named integer constants. They make messages more expressive, maintainable, and strongly typed which is ideal for representing states, categories, roles, or statuses. For example let’s suppose we want to add Priority for the Todo in our applications, here’s how we will use enums for it. 

1
2
3
4
5
6
enum Priority {
  PRIORITY_UNSPECIFIED = 0;  // 0 is required as the first value in proto3
  PRIORITY_LOW = 1;
  PRIORITY_MEDIUM = 2;
  PRIORITY_HIGH = 3;
}

There are a few key points we should keep in mind while using Enums in Protobuf:

  • Enum values map to integer numbers.
  • Names must be unique within the enum.
  • Values must be unique integers.
  • 0 is required as the first value in proto3.

Using Priority as a field in the Toto message:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
syntax = "proto3";

message Todo {
  int64 id = 1;
  string title = 2;
  optional string description = 3;
  bool done = 4;

  enum Priority {
    PRIORITY_UNSPECIFIED = 0;
    PRIORITY_LOW = 1;
    PRIORITY_MEDIUM = 2;
    PRIORITY_HIGH = 3;
  }
  Priority priority = 5;  // <-- using the enum as a property
}

Note: We just used Nested message definition for Enums in the Proto message above, i.e. we defined the enums inside the Todo message.

Arrays or Lists

Lists or arrays are frequently used data structures for any object modelling. In Protobuf, arrays are represented using the repeated keyword. repeated is the built-in way to define lists, collections, and sequences of values. Suppose we want to add multiple tags for a Todo in our applications, this is the perfect use case for repeated keyword. 

1
repeated string tags = 6; // <-- this will be a list of strings

In JSON, it will be like: 

1
"tags": ["gaming", "action", "single-player"],

There are a few characteristics of Protobuf lists that we should know:

  • Lists preserves the ordering of elements.
  • We can create lists for any available data types like scalars and custom message types.
  • Lists can be empty but they can never be nil or null.
  • Default value of list is empty.

Note: For numeric types, Protobuf v3 (proto3) automatically uses Packed Encoding. Packed means values are stored in a single continuous buffer instead of separate entries.

Let’s take another example where suppose we have posts and the posts will have list of comments associated with them. We can model that use case like: 

1
2
3
4
5
6
7
8
9
10
11
12
syntax = "proto3";

message Post {
  string title = 1;
  repeated string tags = 2;                // array of strings
  repeated Comment comments = 3;           // array of comments
}

message Comment {
  string user = 1;
  string message = 2;
}

Maps

Like arrays, Protobuf also allows us to use Maps. They behaves like an associative container (or a dictionary) and stores the key-value pairs. It allows us to store a dynamically sized collection of key-value pairs within your message structure.

Suppose we require a structure where for PostId we can have the collection of Comments for the posts. This is how we can model such use case using Maps: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
syntax = "proto3";

message User {
  int32 id = 1;
  string name = 2;
}

message Comment {
  int32 id = 1;
  string User = 2;
  string text = 3;
}

// we cannot directly create a list inside the map definition 
// hence we are creating a custom message 
message CommentList {
  repeated Comment comments = 1;   // array (list) of comments
}

message PostComments {
  map<int32, CommentList> comments_by_post = 1; // key: post_id → value: list of comments
}

You might be thinking, “Why not just use a map like map<Post, CommentList>?” I wondered the same at first. However, Protobuf impose specific limitations on map definitions, particularly on what can be used as a key. This is because of how the serialization format works.

Let’s have a look at the rules for map keys & values:

  • Key type must be scalar, like int32, int64, uint32, uint64, bool, string
  • Value type can be anything scalars, enums, messages
  • No duplicate keys
  • Order is not guaranteed
  • Maps cannot be repeated fields as it’s already a collections by design

OneOf

oneof is a special construct in Protobuf that lets us define a group of fields where only one field can be set at any given time. It works like a discriminated union or an XOR choice of fields. It’s useful when we want to save space, enforce exclusivity, or model variants of data.

Suppose your application supports multiple mode of Payments but at given time for a particular transition only one method can be used. In such scenarios we will use oneof construct to model the Proto. It will look something like: 

1
2
3
4
5
6
7
message Payment {
  oneof method {
    string card_number = 1;
    string upi_id = 2;
    string bank_account = 3;
  }
}

Well Known Types

Apart from the built-in types available in Protobuf, there’s something called as the Well-known types in Protobuf. They are a collection of predefined, commonly used message types provided by Google to solve problems that basic scalar types can’t handle, such as timestamps, durations, wrappers for primitives, structured data, and more.

Let’s have a look at a few essential WKT:

  • Timestamp: Represents a point in time (UTC), down to nanoseconds which is equivalent to an ISO 8601 datetime.
  • Duration: Represents an amount of time like “5s” or “150ms”.
  • Any: Allows storing any proto message dynamically.
  • Empty: Represents no data which is used for RPC endpoints that don’t need request or response content.

These WKT needs to be imported and used in the proto messages. Let’s have a look at the example: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
syntax = "proto3";

import "google/protobuf/any.proto";
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
import "google/protobuf/empty.proto";

message TaskEvent {
  int32 id = 1;      

  google.protobuf.Timestamp started_at = 2;         
  google.protobuf.Timestamp finished_at = 3;

  google.protobuf.Duration execution_time = 4;      

  google.protobuf.Any details = 5;              // <-- can be any meta data of the task              
}

Protobuf Metadata

Metadata describe how the schema should be interpreted, organized, and compiled. This metadata does not represent the actual data sent over the wire; instead, it provides essential context for the compiler, defining the proto version or feature set, establishing namespaces, pulling in external definitions, and configuring language-specific code generation behavior.

In a Protobuf file, the top-level statements—such as syntax, package, import, and option—serve as metadata. Together, these declarations form the structural and semantic foundation of the proto file, ensuring that messages, enums, and services are generated consistently and understood correctly across different platforms and languages.

  • Syntax: The syntax statement defines the version of the Protocol Buffers language that the file uses. It tells the compiler how to interpret fields, defaults, presence rules, and encoding behavior.
  • Package: Defines the namespace for all messages, enums, and services in the proto file. This prevents naming collisions across large codebases and ensures generated code is organized cleanly in each target language. While optional, it is considered best practice for all non-trivial schemas.
  • Import: It allows you to bring in definitions from other .proto files. This supports modular proto design, letting you reuse common messages, share well-known types like Timestamp, and avoid duplication across different proto modules.
  • Option: It provides configuration settings for the proto file, messages, fields, or generated code. Options can specify language-specific code generation behavior (e.g., Java/Go package names), optimization preferences, or feature toggles in edition-based protos. Options give you fine-grained control over how the compiler and code generators interpret your schema.

Revising the Message

Now we are well equipped with the constructs of Protobuf and it’s building blocks. Now let’s re-look at the Todo message we started initially. We will add the final version of it in this section and here’s how it will look: 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
syntax = "proto3";

package protos.todos;

import "google/protobuf/timestamp.proto";

option go_package = "ashokdey.com/protos/todos";
option java_package = "com.ashokdey.protos.todos";


message Todo {
  int64 id = 1;
  string title = 2;
  optional string description = 3;
  bool done = 4;

  enum Priority {
    PRIORITY_UNSPECIFIED = 0;
    PRIORITY_LOW = 1;
    PRIORITY_MEDIUM = 2;
    PRIORITY_HIGH = 3;
  }
  Priority priority = 5;  // <-- using the enum as a property

  google.protobuf.Timestamp created_at = 6;
}

Now we can head to the compilation of the proto file and play with serialization & deserialization. I will be using Golang, you can do the same using any of your preferred programming language.

Conclusion

Protocol Buffers offer a powerful, efficient, and language-agnostic way to define structured data and API contracts. Although there is a bit of a learning curve, especially when coming from JSON world, but the long-term benefits in performance, consistency, and maintainability are substantially rewarding.

Interestingly, my own journey with Protobuf began back in 2017, 5 years ago! In the last five years the Protobuf ecosystem has evolved rapidly and continues to grow stronger with each passing year.

As you explore Protocol Buffers on your own, I encourage you to get hands-on and deepen your understanding. Can you design a complete mini-application using Protocol Buffers by modeling all its core entities (users, posts, comments, authentication, and notifications)?

Next we will dive into the compilation process, handle the serialization and and how services are defined and used in real applications via Protobuf.

Stay healthy, stay blessed!