Graceful shutdowns should be an essential practice for all production APIs and their dependencies. Unfortunately, that is often not the case. In this blog post, we will explain what a graceful shutdown is, why it is necessary, and how to implement it. We will provide an example of a graceful shutdown in Go using Gin, Rabbit, Mongo, and Docker Compose.
Table Of Contents
What Are Graceful Shutdowns?
To gracefully shut down means to smoothly and safely terminate the process by allowing the remaining tasks to finish and connections to close. This not only improves the user experience as ongoing requests can be completed, but also prevents data corruption, which can occur when a process terminates abruptly. It is important to remember that dependencies need to be gracefully shut down too.
What Does The Code Look Like?
A channel is created that receives signals from the operating system. Typically, this includes the interrupt and terminate signals. The API is then served, and when a signal is received, the server’s graceful shutdown method is called. Each dependency can then sequentially shut down using their respective graceful shutdown methods.
There are other reasons why we may want to gracefully shut down. An example could be losing connection to the database. To handle this, a channel is created that receives the connection loss error. The channel can then be added to a select statement with the OS signal and server error channels.
Each shutdown must include a timeout in case a process becomes unresponsive. If there is an error during the shutdown, the remaining dependencies should still attempt to gracefully shut down.
Example Implementation
Let’s work through an example to clear up any remaining confusion. For this demonstration, we will use Gin as our web framework, Rabbit and Mongo as our dependencies, and Docker Compose to deploy.
The following steps will be taken:
- Analyse the graceful shutdown example provided by Gin and explain how it works.
- Discuss the limitations of Gin’s example and provide an improved revision.
- Deploy our API, Rabbit, and Mongo in containers using Docker Compose.
- Create a connection to Rabbit and Mongo.
- Provide a solution that will gracefully shut down the API and both dependencies.
Gin’s Example
The code below is an example of a graceful shutdown provided by Gin and was taken from their website on November 15th, 2023. This will be the starting point for our example.
The code works as follows:
- An API is served in a new goroutine with a single endpoint.
- The main goroutine is blocked until an OS signal is received.
SIGINTcan occur when the server is stopped by pressing CTRL + C in the terminal.SIGTERMcan occur when the container in Docker or Kubernetes is stopped.SIGKILLcannot be caught as it immediately kills the process.
- When a signal is received, the
Shutdownmethod is called and the the graceful shutdown begins. - If there is an error while gracefully shutting down, then exit immediately. Otherwise, wait for the five-second timeout to finish before closing the application.
Before continuing, it is highly recommended to experiment with the code and ensure a full understanding of what is happening.
package main
import (
"context"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/gin-gonic/gin"
)
func main() {
router := gin.Default()
router.GET("/", func(c *gin.Context) {
time.Sleep(5 * time.Second)
c.String(http.StatusOK, "Welcome Gin Server")
})
srv := &http.Server{
Addr: ":8080",
Handler: router,
}
go func() {
// service connections
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("listen: %s\n", err)
}
}()
// Wait for interrupt signal to gracefully shutdown the server with
// a timeout of 5 seconds.
quit := make(chan os.Signal)
// kill (no param) default send syscanll.SIGTERM
// kill -2 is syscall.SIGINT
// kill -9 is syscall. SIGKILL but can"t be catch, so don't need add it
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
log.Println("Shutdown Server ...")
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Fatal("Server Shutdown:", err)
}
// catching ctx.Done(). timeout of 5 seconds.
select {
case <-ctx.Done():
log.Println("timeout of 5 seconds.")
}
log.Println("Server exiting")
}Making Some Improvements
Gin’s solution does solve the problem, but it requires a reshuffle to accommodate dependencies. We need to use a select statement so that we can handle multiple channels.
If we create an error channel that receives data from ListenAndServe, we can use a select statement to handle the error and signal channels. This allows our API to attempt a graceful shutdown even if there is a server error and facilitates adding dependencies later.
We create a graceful shutdown method to avoid duplication between cases in the select statement. This will become more important as the number of dependencies increases.
It is unnecessary to wait for the timeout before closing the application. The shutdown method allows any in-flight requests to be completed and returns any errors, so we gain nothing by waiting.
The code below incorporates all the changes mentioned.
package main
import (
"context"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/gin-gonic/gin"
)
func main() {
router := gin.Default()
router.GET("/", func(c *gin.Context) {
time.Sleep(5 * time.Second)
c.String(http.StatusOK, "Welcome Gin Server")
})
srv := &http.Server{
Addr: ":8080",
Handler: router,
}
srvErrs := make(chan error, 1)
go func() {
srvErrs <- srv.ListenAndServe()
}()
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
shutdown := gracefulShutdown(srv)
select {
case err := <-srvErrs:
shutdown(err)
case sig := <-quit:
shutdown(sig)
}
log.Println("Server exiting")
}
func gracefulShutdown(srv *http.Server) func(reason interface{}) {
return func(reason interface{}) {
log.Println("Server Shutdown:", reason)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Println("Error Gracefully Shutting Down API:", err)
}
}
}Deploying With Docker Compose
Let’s introduce some dependencies to test the solution. We will be adding Rabbit and Mongo for our example. To get started, we will deploy the API and dependencies in containers using Docker Compose.
Create an image of our API using a Dockerfile.
FROM golang:1.20.10-alpine3.17
RUN mkdir /app
COPY .. /app
WORKDIR /app
RUN go build -o api
CMD ["./api"]Create a compose.yaml file that includes entries for the API, Rabbit, and Mongo. Add health checks to ensure that both dependencies are running before the API starts.
version: '3.8'
name: graceful-shutdown
services:
api:
container_name: api
build:
context: .
dockerfile: Dockerfile
ports:
- '8080:8080'
depends_on:
mongo:
condition: service_healthy
rabbit:
condition: service_healthy
rabbit:
container_name: rabbit
image: rabbitmq:3.12.10-alpine
ports:
- '5672:5672'
healthcheck:
test: rabbitmq-diagnostics check_port_connectivity
start_period: 20s
interval: 10s
timeout: 10s
retries: 3
mongo:
container_name: mongo
image: mongo:7-jammy
ports:
- '27017:27017'
healthcheck:
test: echo 'db.runCommand("ping").ok' | mongosh mongo:27017/test --quiet
interval: 10s
timeout: 10s
retries: 3
start_period: 20sCheck that the cluster starts without any errors.
docker-compose up --buildConnecting To Rabbit
Add the Rabbit package.
go get github.com/rabbitmq/amqp091-goIn the rabbit.go file below, we call the Dial function and pass in the Rabbit URL. The Connection struct is returned along with any errors. If there are any errors, we will return them and terminate the app in main.go.
The Connection struct contains a method named NotifyClose which sends an error to a channel if the connection is closed. We create a wrapper function Err that returns the channel and will be used in the select statement in main.go.
This struct also contains a method called CloseDeadline, which allows us to gracefully shut down with a timeout. We call this in our wrapper function Disconnect, which will be used in the main.go file to initiate the graceful shutdown of Rabbit.
package main
import (
"log"
"time"
amqp "github.com/rabbitmq/amqp091-go"
)
type Rabbit struct {
conn *amqp.Connection
errChan <-chan *amqp.Error
}
func newRabbit() (*Rabbit, error) {
log.Println("Connecting to Rabbit...")
conn, err := amqp.Dial("amqp://guest:guest@rabbit:5672")
if err != nil {
return nil, err
}
errs := make(chan *amqp.Error, 1)
errChan := conn.NotifyClose(errs)
// ...
log.Println("Successfully connected to Rabbit")
return &Rabbit{
conn: conn,
errChan: errChan,
}, nil
}
func (r *Rabbit) Err() <-chan *amqp.Error {
return r.errChan
}
func (r *Rabbit) Disconnect(timeout time.Duration) error {
if r.conn == nil {
return nil
}
return r.conn.CloseDeadline(time.Now().Add(timeout))
}Connecting To Mongo
Add the Mongo package.
go get go.mongodb.org/mongo-driver/mongoIn the mongo.go file below, we invoke the Connect function, providing the URI and a context with a timeout as parameters. The function returns a Client struct along with any errors. If there are any errors or if we fail to connect to Mongo, we will return an error and terminate the application in main.go.
Unfortunately, unlike Rabbit, there is no method on the Client struct that will inform us of a connection loss. While there are some event monitoring channels available, they do not meet all of our needs.
As a result, we created the function monitor, which will ping Mongo and then sleep for five seconds. If the ping fails, then an error is sent to a channel that will be accessed by the wrapper function Err. This will be used in the select statement in main.go.
The Client struct does have a graceful shutdown method called Disconnect, which takes a context with a timeout. We will call this method in our wrapper function Disconnect in main.go.
package main
import (
"context"
"errors"
"fmt"
"log"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
type Mongo struct {
client *mongo.Client
errChan <-chan error
}
func newMongo() (*Mongo, error) {
log.Println("Connecting to Mongo...")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
uri := fmt.Sprintf("mongodb://mongo:%d", 27017)
client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
if err != nil {
return nil, err
}
if err = client.Ping(context.Background(), readpref.Primary()); err != nil {
return nil, err
}
log.Println("Successfully connected to mongo")
errChan := make(chan error)
go monitor(client, errChan)
return &Mongo{
client: client,
errChan: errChan,
}, nil
}
func monitor(client *mongo.Client, errChan chan error) {
for {
if err := client.Ping(context.Background(), readpref.Primary()); err != nil {
errChan <- errors.New("Lost connection to mongo")
break
}
time.Sleep(5 * time.Second)
}
}
func (m *Mongo) Err() <-chan error {
return m.errChan
}
func (m *Mongo) Disconnect(ctx context.Context) error {
if m.client == nil {
return nil
}
return m.client.Disconnect(ctx)
}Before you start panicking, Ping will attempt to get a response for serverSelectTime (which defaults to 30 seconds) before returning an error. This should be enough to handle any latency. Do not fear for Mongo’s CPU as Mongo replica sets regularly ping each other anyway.
It is worth considering that this occupies a goroutine, but the chances are that this will not be a problem for your project.
Final Solution
Below is the updated main.go file that ties everything together. The code is mostly the same as our previous solution but with some new additions. We attempt to establish a connection to both Rabbit and Mongo, but the application will terminate if this is unsuccessful.
The select statement now includes additional cases, utilising the respective wrapper functions that return the error channels. The graceful shutdown wrapper functions are now invoked in the gracefulShutdown method. Each dependency is given five seconds to shut down gracefully before giving up and moving on to the next.
package main
import (
"context"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"github.com/gin-gonic/gin"
)
func main() {
rabbit, err := newRabbit()
if err != nil {
log.Fatal(err)
}
mongo, err := newMongo()
if err != nil {
log.Fatal(err)
}
router := gin.Default()
router.GET("/", func(c *gin.Context) {
time.Sleep(5 * time.Second)
c.String(http.StatusOK, "Welcome Gin Server")
})
srv := &http.Server{
Addr: ":8080",
Handler: router,
}
srvErrs := make(chan error, 1)
go func() {
srvErrs <- srv.ListenAndServe()
}()
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
shutdown := gracefulShutdown(srv, rabbit, mongo)
select {
case err := <-srvErrs:
shutdown(err)
case sig := <-quit:
shutdown(sig)
case err := <-rabbit.Err():
shutdown(err)
case err := <-mongo.Err():
shutdown(err)
}
log.Println("Server exiting")
}
func gracefulShutdown(srv *http.Server, rabbit *Rabbit, mongo *Mongo) func(reason interface{}) {
return func(reason interface{}) {
log.Println("Server Shutdown:", reason)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(ctx); err != nil {
log.Println("Error Gracefully Shutting Down API:", err)
}
if err := rabbit.Disconnect(5 * time.Second); err != nil {
log.Println("Error Gracefully Shutting Down Rabbit:", err)
}
ctx, cancel = context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := mongo.Disconnect(ctx); err != nil {
log.Println("Error Gracefully Shutting Down Mongo:", err)
}
}
}The source code can be found on our GitHub.
Conclusion
To summarise, it is crucial not to overlook the significance of graceful shutdowns. If you are working on an API destined for production, they must be included. Graceful Shutdowns offer an improved user experience and reduce the risk of data corruption. With that said, if you are only experimenting and creating small projects, you may not require them. Nonetheless, this is knowledge you want to have in your tool kit.
If you have any questions or comments, please leave them below. Also, don’t forget to subscribe to our newsletter to receive more informative posts like this directly to your inbox.
If you found this article helpful, please share it with your friends and colleagues.
Thank you for reading.
Leave a Reply