Elizar Pepino

January 1, 1

Exploring Protocol Buffers

+++ author = “Elizar” categories = [“protobuf”] date = 2022-03-27T16:13:15Z excerpt = “Protocol buffers are binary data transmitted over the wire and get deserialized later on into a language-specific data structure. Think JSON, but faster and simpler.” image = “https://res.cloudinary.com/dyd19kbsp/image/upload/v1648449443/protobuf_uvoqgm.png” tags = [“grpc”, “golang”, “protocolbuffers”, “distributed systems”] title = “Exploring Protocol Buffers”

+++

What are protocol buffers?

Protocol buffers are binary data transmitted over the wire and get deserialized later on into a language-specific data structure. Think JSON, but faster and simpler.

Protocol buffers let you declare the minimal set of information to describe the format of any data structure you want. The compiler does the work of generating all the serialization and deserialization code in many different languages. Simultaneously, you get type safety, backward compatibility with old versions, and lots of free optimizations.

Setup

First, obtain the Protocol Compiler. The easiest way is to download a pre-built binary from https://github.com/protocolbuffers/protobuf/releases.

Once you have protoc installed, then you’re good to go.

Let’s create a simple echo server with Go and Javascript

Before we get started, make sure you have the following installed on your system first:

Folder Structure

. ├── client // Javascript ├── proto // Proto Buffers └── server // Go

Create hello.proto

Under the ./proto folder, create a new directory called hello; inside it, create a new file called hello.proto; then copy and paste the following:

syntax = "proto3";
package hello;

message Hello {
	string from = 1;
	string text = 2;
	int64 date = 3;
}

Compile and generate code for the client(javascript) and server(go) with the following command:

$ protoc
--js_out=import_style=commonjs,binary:./client
--go_out=./server
./proto/**/*.proto

Your folder and file structure should look something like this:

. ├── client │ └── proto │ └── hello │ └── hello_pb.js ├── server │ └── proto │ └── hello │ └── hello.pb.go └── proto └── hello └── hello.proto

Server

Under ./server dir, run go mod init pb. This command will generate go module related files inside the directory. And then run go get -u github.com/golang/protobuf/protoc-gen-go.

Create a new file called main.go under the ./server dir then copy and paste the following:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
	"os"
	"pb/proto/hello"
	"time"

	"github.com/golang/protobuf/proto"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		body, _ := ioutil.ReadAll(r.Body)

		// parse message
		h := &hello.Hello{}
		err := proto.Unmarshal(body, h)

		if err != nil {
			http.Error(w, err.Error(), http.StatusBadRequest)
			return
		}

		date := time.Unix(h.GetDate(), 0).Format("Jan 2 2006, 15:04:05")
		msg := fmt.Sprintf("%s says %s on %s", h.GetFrom(), h.GetText(), date)
		fmt.Fprintln(w, msg)
	})

	port := ":" + os.Getenv("PORT")
	log.Println("Server running on " + port)
	log.Fatal(http.ListenAndServe(port, nil))
}

Client

For the client, we need to install the following modules:

  • google-protobuf
  • request

Run npm install google-protobuf request; create client.js file under the same folder; and then copy and paste the following code:

const request = require("request");
const { Readable } = require("stream");
const { Hello } = require("./proto/hello/hello_pb");

// init
const hello = new Hello();

const [from, text] = process.argv.slice(2, process.argv.length);

// set vals
hello.setFrom(from);
hello.setText(text);
hello.setDate(Math.round(Date.now() / 1000));

// serialize
const bytes = hello.serializeBinary();
const rs = new Readable();
rs._read = () => {};
rs.push(Buffer.from(bytes));
rs.push(null);
rs.pipe(
  request
    .post("http://localhost:31337")
    .on("response", (r) => r.pipe(process.stdout))
);

Let’s run and test our program.

Start the server

  1. cd server
  2. PORT=31337 go run main.go

Run the client

  1. cd client
  2. node client penzur "what's up?"
// Server response: penzur says what's up? on Jul 8, 2019, 03:18:38

Resources