Automatic TLS certificate distribution with Vault

Introducing Certify, a purpose built library

It’s often recommended that microservice communications are encrypted, especially if they cross the public internet. Even when you control the network, whether virtually or physically, encrypting traffic can still be valuable. Lets Encrypt has revolutionized the distribution of certificates for publically facing servers; unfortunately, encrypting internal traffic often involves complicated certificate generation setups. Hashicorp Vault is a project for storing secrets and performing authentication. It has a PKI Secret Engine backend which allows you to use it as a certificate authority in an internal public key infrastructure deployment. [Read More]

Advanced gRPC Error Usage

Best practices for error metadata propagation with gRPC

How to best handle errors in Go is a divisive issue, leading to opinion pieces by illustrious bloggers such as Dave Cheney, the commander himself Rob Pike as well as the official Go blog. I’m not going to tackle those opinions here, instead I’m going to talk about best practices for errors when using gRPC and Go. The gRPC Status package The Go gRPC implementation has a status package which exposes a nice simple interface for creating rich gRPC errors. [Read More]

So you want to use GoGo Protobuf

Best practices for using GoGo Protobuf

Introduction In the Go protobuf ecosystem there are two major implementations to choose from. There’s the official golang/protobuf, which uses reflection to marshal and unmarshal structs, and there’s gogo/protobuf, a third party implementation that leverages type-specific marshalling code for extra performance, and has many cool extensions you can use to customize the generated code. gogo/protobuf has been recommended as the best choice of Go serialization library in a large test of different implementations. [Read More]

I was invited to a meetup

My fantastic experience in Budapest

Introduction After my presentation at the Go London meetup I was approached on Gophers slack by a gentleman from Hungary called Máté Gulyás. Máté is one of the organisers of the Budapest Go Meetup, and was reaching out to me to ask if I would like to come down to Budapest and deliver a talk for the meetup. I was delighted with this invitation and promptly accepted and we started planning. [Read More]

Client side streaming in gRPC-Web

Enhancing the GopherJS gRPC-Web bindings with Websockets

In a previous post I introduced my open source project to bring GopherJS bindings to Improbable’s gRPC-Web client. I’m happy to say that the initial goal of supporting all features of the gRPC-Web client has been completed. I was initially going to leave it at that and wait for client side streaming to land in the WHATWG Streams API Standard, and subsequently added to the official grpc-web spec and probably the gRPC-Web client, but then I was sitting at the GolangUK conference and I had a brain wave. [Read More]

Chunking large messages with gRPC

Using gRPC server-side streaming with binary blobs

One of the gotchas of using gRPC is that it was not designed to transport large messages in one chunk. The default max message size is slightly arbitrarily set at 4MB today, and while it is possible to configure, that kind of behaviour might lead to a slippery slope scenario of ever increasing max message sizes. So what do we do when the message size is too large? We chunk the data into smaller pieces and stream it, using the gRPC streaming methods, naturally. [Read More]

Throttling resource intensive requests

OOM errors hate this one weird trick

Sometimes when you’re writing a server, you’ve got a function that consumes a lot of memory while running, or some other resource, and you might be worrying that a sudden burst of requests could crash the server, since gRPC by default will just spawn another goroutine to handle any incoming requests, oblivious to the danger. In these situations, it can be useful to implement some custom request throttling. Here I’ll show an easy way to accomplish this with the use of a Go channel. [Read More]

GopherJS Integration Tests

With friends Qunit, Ginkgo and Agouti

Recently I found myself wondering how I was going to test my new GopherJS gRPC-Web bindings. Writing tests was something I had been waiting with until I had something working, mostly because I had no idea how I was going to meaningfully test GopherJS code that relies on interactions with JS and the reponses of a server. I have in the past made a small contribution to the GopherJS websocket repo, and found myself impressed with the extensive tests written for the repo. [Read More]

gRPC-Web with GopherJS

Introducing the GopherJS gRPC-Web bindings

In a previous blog series I’ve talked about how to work with a gRPC backend from the GopherJS world. It relies on the gRPC-gateway which is a great piece of tech, but unfortunately carries a couple of downsides: Clients don’t know what types are used - the interface is HTTP JSON. This can be somewhat mitigated with the use of swagger generated interfaces, but it’s still not perfect. The interface being JSON means marshalling and unmarshalling can become a significant part of the latency between the client and the server. [Read More]

Go Protobuf Tips

A couple of tips for working with proto files in golang

I’ve had my fair share of dealing with proto files in go (and to some extent JS), so I thought I’d share some stuff I’ve learnt the hard way by working with proto files. Protoc include paths The protoc include paths can be pretty confusing, so I’ll give a few examples of how to use it properly. Just include the current directory protoc requires that the files referenced are in the include path, so if you’re referencing files relative to the current directory, you’ll need to specify -I. [Read More]