January 1, 1
Exploring Protocol Buffers
+++ author = “Elizar” categories = [“protobuf”] date = 2022-03-27T16:13:15Z excerpt = “Protocol buffers are binary data transmitted over the wire and get deserialized later on into a language-specific data structure. Think JSON, but faster and simpler.” image = “https://res.cloudinary.com/dyd19kbsp/image/upload/v1648449443/protobuf_uvoqgm.png” tags = [“grpc”, “golang”, “protocolbuffers”, “distributed systems”] title = “Exploring Protocol Buffers”
+++
What are protocol buffers?
Protocol buffers are binary data transmitted over the wire and get deserialized later on into a language-specific data structure. Think JSON, but faster and simpler.
Protocol buffers let you declare the minimal set of information to describe the format of any data structure you want. The compiler does the work of generating all the serialization and deserialization code in many different languages. Simultaneously, you get type safety, backward compatibility with old versions, and lots of free optimizations.
Setup
First, obtain the Protocol Compiler
. The easiest way is to download a pre-built binary from https://github.com/protocolbuffers/protobuf/releases.
Once you have protoc installed, then you’re good to go.
Let’s create a simple echo server with Go and Javascript
Before we get started, make sure you have the following installed on your system first:
Folder Structure
. ├── client // Javascript ├── proto // Proto Buffers └── server // Go
Create hello.proto
Under the ./proto
folder, create a new directory called hello
; inside it, create a new file called hello.proto
; then copy and paste the following:
syntax = "proto3";
package hello;
message Hello {
string from = 1;
string text = 2;
int64 date = 3;
}
Compile and generate code for the client(javascript)
and server(go)
with the following command:
$ protoc
--js_out=import_style=commonjs,binary:./client
--go_out=./server
./proto/**/*.proto
Your folder and file structure should look something like this:
. ├── client │ └── proto │ └── hello │ └── hello_pb.js ├── server │ └── proto │ └── hello │ └── hello.pb.go └── proto └── hello └── hello.proto
Server
Under ./server
dir, run go mod init pb
. This command will generate go module
related files inside the directory. And then run go get -u github.com/golang/protobuf/protoc-gen-go
.
Create a new file called main.go
under the ./server
dir then copy and paste the following:
package main
import (
"fmt"
"io/ioutil"
"log"
"net/http"
"os"
"pb/proto/hello"
"time"
"github.com/golang/protobuf/proto"
)
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
body, _ := ioutil.ReadAll(r.Body)
// parse message
h := &hello.Hello{}
err := proto.Unmarshal(body, h)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
date := time.Unix(h.GetDate(), 0).Format("Jan 2 2006, 15:04:05")
msg := fmt.Sprintf("%s says %s on %s", h.GetFrom(), h.GetText(), date)
fmt.Fprintln(w, msg)
})
port := ":" + os.Getenv("PORT")
log.Println("Server running on " + port)
log.Fatal(http.ListenAndServe(port, nil))
}
Client
For the client, we need to install the following modules:
- google-protobuf
- request
Run npm install google-protobuf request
; create client.js
file under the same folder; and then copy and paste the following code:
const request = require("request");
const { Readable } = require("stream");
const { Hello } = require("./proto/hello/hello_pb");
// init
const hello = new Hello();
const [from, text] = process.argv.slice(2, process.argv.length);
// set vals
hello.setFrom(from);
hello.setText(text);
hello.setDate(Math.round(Date.now() / 1000));
// serialize
const bytes = hello.serializeBinary();
const rs = new Readable();
rs._read = () => {};
rs.push(Buffer.from(bytes));
rs.push(null);
rs.pipe(
request
.post("http://localhost:31337")
.on("response", (r) => r.pipe(process.stdout))
);
Let’s run and test our program.
Start the server
cd server
PORT=31337 go run main.go
Run the client
cd client
node client penzur "what's up?"
// Server response: penzur says what's up? on Jul 8, 2019, 03:18:38