In the Go protobuf ecosystem there are two major implementations
to choose from. There’s the official
golang/protobuf, which uses
reflection to marshal and unmarshal structs, and there’s
gogo/protobuf, a third party
implementation that leverages type-specific marshalling code for
extra performance, and has many cool extensions you can use to
customize the generated code.
gogo/protobuf has been recommended
as the best choice of Go serialization library in a
large test of different implementations.
Unfortunately, the design of
golang/protobuf and the gRPC ecosystem
makes it hard to integrate third party implementations,
and there are certain situations where using
with gRPC can break unexpectedly, at runtime.
In this post I will try to cover best practices for
I made an example repo of using
gogo/protobuf with various parts of the
greater gRPC ecosystem, graciously hosted under the
gogo namespace by
Walter Schulze, complete with a
gRPC-Gateway and OpenAPI UI:
If you find anything that isn’t listed on there, or in this post, please submit an issue against this repo, and I will attempt to implement a workaround or raise a relevant issue upstream.
Still here? Lets move on to the details.
and golang/genproto repos
provide a large number of protofiles and pre-generated Go files, all
maintained by Google’s engineers. However, because they use
to compile the Go files, they are not strictly compatible with
gogo/protobuf, as they do not register with the correct backend.
Instead, if you find you need to reach for these pre-compiled files,
This contains a growing number of Go files pre-generated with
protoc-gen-gogo, and registering against the correct backend.
If there are any files missing from this repo, make sure to raise
an issue (or make a PR) and it’ll be added in no time.
Bonus: because the generated files are in the same folder as the proto files, including the files works with golang/dep, limitations on including non-go files notwithstanding.
Protobuf Any types
type is used in a wide variety of the GoogleAPIs proto messages,
but using it with
gogo/protobuf requires extra care. The
message types work by using the internal “registry” of the protobuf
package used, so you need to make sure any messages you stick in
Any container have been generated with
gogo/googleapis repo is a great start, but the general
rule of thumb is to ensure all protofiles are generated with
gRPC is designed to be payload agnostic, and will work out of the
gogo/protobuf, as while it imports
it only uses it to
type assert incoming interfaces
into interfaces that are equally supported by all
No changes necessary here.
gRPC has this cool thing called server reflection, which allows a client to use a gRPC server without having to use the servers protofile, dynamically, at runtime. Some tools such as grpc-ecosystem/polyglot, ktr0731/evans, kazegusuri/grpcurl and fullstorydev/grpcurl (popular pun) have support for dynamic reflection based requests today.
gogo/protobuf is currently not working perfectly
with server reflection, because the grpc-go implementation is
very tightly coupled
golang/protobuf. This presents a couple of different scenarios
gogo/protobuf may or may not work:
- If you use just the
protoc-gen-gofastgenerator, which simply generates type specific marshalling and unmarshalling code, you’ll be fine. Of course, using
protoc-gen-gofaststill comes with downsides, such as having to regenerate the whole proto dependency tree.
- If you use
protoc-gen-gogo*, unfortunately, reflection will not work on your server. This is because gogo.pb.go does not register itself with
golang/protobuf, and reflection recursively resolves all imports, and will complain of
gogo.protonot being found.
This is of course quite disappointing, but I’ve discussed
with Walter Schulze
(the maintainer of
gogo/protobuf) how best to solve this and
raised an issue against grpc-go.
If the maintainers of grpc-go do not want to make it easier
to use with
gogo/protobuf, there are other alternatives.
I’ll update this post once I know more.
The gRPC-Gateway is another popular project, and at first it
might seem completely compatible
it suffers from a number of incompatibilities, most of which
can be traced to its liberal use of
golang/protobuf packages directly:
- The gRPC-Gateway does not work with
- The default JSON marshaller used by the gRPC-Gateway is unable to marshal non-nullable non-scalar fields.
- The gRPC-Gateways logic for handling the
google.protobuf.FieldMaskWell Known Type in query parameters is hardcoded to use
- A bug in the generator means generated files with Well Known Types need post-generation corrections.
Fortunately, workarounds exist for most of these problems.
gogo/protobuf will ensure enum resolution works.
golang/protobuf/jsonpb with the gRPC-Gateway
fixes the scalar field marshalling issue.
Both of these workarounds are implemented in the gRPC-example repo.
Unfortunately, there is no workaround for the
as far as I am aware. See
the issue in the grpc-example repo
for more discussion on this topic. Thanks to
Timon Wong for bringing
this to my attention.
As for the incorrect import, a simple
will sort that out (adjust as necessary):
$ sed -i "s/empty.Empty/types.Empty/g" <file.pb.gw.go>
Note that the gRPC-Gateway makes use of
so make sure you include the correct file from
mentioned when compiling your proto files.
gogo/protobuf delivers awesome customization
options and faster marshalling, getting it working well with the
larger gRPC ecosystem is complicated.
gogo/protobuf has it as a
stated goal to be merged back into
have been positive, but it’s hard to say whether it’ll lead to anything.
There is also an open issue
discussing the possibility of type specific marshalling and unmarshalling code
golang/protobuf itself, which is what I think is the biggest reason
most users turn to
In a perfect future, we’d have some or all of the customizability and speed of
gogo/protobuf with the official backing of
I made a repo for experimenting with various go proto generators that you can check out if you want to make your own tests.
If you enjoyed this blog post, have any questions or input,
don’t hesitate to contact me on
jbrandhorst on the Gophers Slack. I’d love to hear