🍍 🐏 🚖 Golang backend server template - part 1 (HTTP server) 📹 🌝 ☔️

The Golang server template below was prepared to transfer knowledge within our team. The main goal of the template, in addition to training, is to reduce the time for prototyping small server tasks on Go.

The template includes:

Passing server parameters through the github.com/urfave/cli command line
Configuring server settings through the github.com/sasbury/mini configuration file
Configure TLS HTTP server settings
Configuring a router registering HTTP and prof-handlers github.com/gorilla/mux
Setting logging levels without stopping the github.com/hashicorp/logutils server
Configuring HTTP traffic logging without stopping the server
Setting error logging in HTTP response without stopping the server
HTTP Basic Authentication
MS AD Authentication gopkg.in/korylprince/go-ad-auth.v2
JSON Web Token github.com/dgrijalva/jwt-go
Starting the server with the expectation of a return to the error channel
Using context to properly stop the server and related services
Custom error handling setup github.com/pkg/errors
Custom Logging Settings
Build with version, build date, and commit

Link to the project repository .

HTTP (. HTTP Golang):

POST /echo — request HTTP body response
POST /signin — JSON Web Token Cookie
POST /refresh — JSON Web Token Cookie
POST /httplog — HTTP
POST /httperrlog — HTTP response
POST /loglevel — DEBUG, INFO, ERROR

2.2.
2.3.
,
3.1. daemon
3.2. daemon
3.3. daemon
4.1.
4.2.
4.3.
4.4.
5.1.
5.2.
5.3.
5.4. HTTP
7.1. go mod
7.2.

1.

1:ERP, — 1 IBM MQ.
IBM MQ :

IBM MQ (, , )
IBM MQ
IBM MQ (SYNCPOINT)

XML :

() RFC 3076
HMAC - -34.11.94
HMAC

1C IBM MQ . REST API IBM MQ .

1C IBM MQ Go IBM MQ. , , "" C . 2 [].

backend , .

1 IBM MQ .

REST_IBMMQ

2.

2.1.

, , . github.com/urfave/cli. :

   --httpconfig value, --httpcfg value    HTTP Config file name
   --listenstring value, -l value         Listen string in format <host>:<port>
   --httpuser value, --httpu value        User name for access to HTTP server
   --httppassword value, --httppwd value  User password for access to HTTP server
   --jwtkey value, --jwtk value           JSON web token secret key
   --debug value, -d value                Debug mode: DEBUG, INFO, ERROR
   --logfile value, --log value           Log file name

2.2.

github.com/sasbury/mini. , :

[HTTP_SERVER]
ReadTimeout = 6000          // HTTP read timeout duration in sec - default 60 sec
WriteTimeout = 6000         // HTTP write timeout duration in sec - default 60 sec
IdleTimeout = 6000          // HTTP idle timeout duration in sec - default 60 sec
MaxHeaderBytes = 262144     // HTTP max header bytes - default 1 MB
MaxBodyBytes = 1048576      // HTTP max body bytes - default 0 - unlimited
UseProfile = false          // use Go profiling
ShutdownTimeout = 30        // service shutdown timeout in sec - default 30 sec

[TLS]
UseTLS = false                  // use SSL
UseHSTS = false                 // use HTTP Strict Transport Security
TLSertFile = certs/server.pem  // TLS Certificate file name
TLSKeyFile = certs/server.key   // TLS Private key file name
TLSMinVersion = VersionTLS10    // TLS min version VersionTLS13, VersionTLS12, VersionTLS11, VersionTLS10, VersionSSL30
TLSMaxVersion = VersionTLS12    // TLS max version VersionTLS13, VersionTLS12, VersionTLS11, VersionTLS10, VersionSSL30

[JWT]
UseJWT = false                  // use JSON web token (JWT)
JWTExpiresAt = 20000            // JWT expiry time in seconds - 0 without restriction

[AUTHENTIFICATION]
AuthType = INTERNAL             // Autehtification type NONE | INTERNAL | MSAD
MSADServer = company.com        // MS Active Directory server
MSADPort = 389                  // MS Active Directory Port
MSADBaseDN = OU=, DC=, DC=      // MS Active Directory BaseDN
MSADSecurity = SecurityNone     // MS Active Directory Security: SecurityNone, SecurityTLS, SecurityStartTLS

[LOG]
HTTPLog = false                         // Log HTTP traffic
HTTPLogType = INREQ                     // HTTP trafic log mode INREQ | OUTREQ | INRESP | OUTRESP | BODY
HTTPLogFileName = ./httplog/http%s.log  // HTTP log file
HTTPErrLog = HEADER | BODY              // Log error into HTTP response header and body

3. ,

UML .

, daemon. :

context.Context
/

3.1. daemon

, daemon.
, daemon .
, daemon :

httpserverErrCh: make(chan error, 1), //   HTTP

, :

daemon —
— daemon

IBM MQ:
- IBM MQ, ,
- IBM MQ
PostgreSQL:
- , , SQL
JSON BoltDB:
- BoltDB
- ( PostgreSQL). , ( BoltDB 150 2 64 , BoltDB NVMe 0.03 ms).
HTTP :
- http.server
- TCP
- TLS
- HTTP
- pprof

3.2. daemon

daemon .
, — , , , .
, . , HTTP

go func() { httpserverErrCh <- d.httpserver.Run() }()

, daemon .
daemon , , - . , ( ) .
, IBM MQ , daemon IBM MQ .

syscalCh := make(chan os.Signal, 1) //   
signal.Notify(syscalCh, syscall.SIGINT, syscall.SIGTERM)

//       
select {
case s := <-syscalCh: //  
    mylog.PrintfInfoMsg("Exiting, got signal", s)
    d.Shutdown() //  daemon
    return nil
case err := <-d.httpserverErrCh: //   HTTP    
    mylog.PrintfErrorInfo(err) //  
    d.Shutdown() //  daemon
    return err
}

, , ( ). , daemon.

func (s *Server) Run() error {
    defer func() {
        var myerr error
        r := recover()
        if r != nil {
            msg := "Recover from panic"
            switch t := r.(type) {
            case string:
                myerr = myerror.New("8888", msg, t)
            case error:
                myerr = myerror.WithCause("8888", msg, t)
            default:
                myerr = myerror.New("8888", msg)
            }
            mylog.PrintfErrorInfo(myerr) //  
            s.errCh <- myerr             //       daemon
        }
    }()

    //  
}

3.3. daemon

daemon .

, , :

( ShutdownTimeout )
. IBM MQ :
- IBM MQ
stopCh

, :

. ,
, , , ,

for {
    select {
    case <-ctx.Done(): //    

        //  

        s.stopCh <- struct{}{} //     
        return
    default:
        //   
    }
}

HTTP :

//        ShutdownTimeout
cancelCtx, cancel := context.WithTimeout(s.ctx, time.Duration(s.cfg.ShutdownTimeout*int(time.Second)))
defer cancel()

//       ShutdownTimeout
if err := s.httpServer.Shutdown(cancelCtx); err != nil {
    return err
}

s.httpService.Shutdown() //   

//     HTTP 
s.stopCh <- struct{}{}

4.

4.1.

github.com/pkg/errors.
, , .

type Error struct {
    ID       uint64 //   
    Code     string //  
    Msg      string //  
    Caller   string // ,        
    Args     string //  
    CauseErr error  //  - 
    CauseMsg string //   - 
    Trace    string //      
}

, . , 1 IBM MQ, 4 . , "8" HTTP, "7" — IBM MQ.

Caller — , . , . :

httpserver.go:[209] - (*Server).Run()

Caller

func caller(depth int) string {
    pc := make([]uintptr, 15)
    n := runtime.Callers(depth+1, pc)
    frame, _ := runtime.CallersFrames(pc[:n]).Next()
    idxFile := strings.LastIndexByte(frame.File, '/')
    idx := strings.LastIndexByte(frame.Function, '/')
    idxName := strings.IndexByte(frame.Function[idx+1:], '.') + idx + 1

    return frame.File[idxFile+1:] + ":[" + strconv.Itoa(frame.Line) + "] - " + frame.Function[idxName+1:] + "()"
}

Args — , . .

CauseErr CauseMsg — . , .

Trace — . . github.com/pkg/errors '%+v'. .

fmt.Sprintf("'%+v'", pkgerr.New(""))

4.2.

Error, .

func (e *Error) Error() string {
    mes := fmt.Sprintf("ID=[%v], code=[%s], mes=[%s]", e.ID, e.Code, e.Msg)
    if e.Args != "" {
        mes = fmt.Sprintf("%s, args=[%s]", mes, e.Args)
    }
    if e.CauseMsg != "" {
        mes = fmt.Sprintf("%s, causemes=[%s]", mes, e.CauseMsg)
    }
    return mes
}

ID=[1], code=[8004], mes=[Error message], args=['arg1', 'arg2', 'arg3']

// %s    print the error code, message, arguments, and cause message.
// %v    in addition to %s, print caller
// %+v   extended format. Each Frame of the error's StackTrace will be printed in detail.
func (e *Error) Format(s fmt.State, verb rune) {
    switch verb {
    case 'v':
        fmt.Fprint(s, e.Error())
        fmt.Fprintf(s, ", caller=[%s]", e.Caller)
        if s.Flag('+') {
            fmt.Fprintf(s, ", trace=%s", e.Trace)
            return
        }
    case 's':
        fmt.Fprint(s, e.Error())
    case 'q':
        fmt.Fprint(s, e.Error())
    }
}

'%+v'

ID=[1], code=[8004], mes=[Error message], args=['arg1', 'arg2', 'arg3'], caller=[handler_echo.go:[31] - (*Service).EchoHandler.func1()], trace='
github.com/romapres2010/httpserver/error.New
        D:/golang/src/github.com/romapres2010/httpserver/error/error.go:72
github.com/romapres2010/httpserver/httpserver/httpservice.(*Service).EchoHandler.func1
        D:/golang/src/github.com/romapres2010/httpserver/httpserver/httpservice/handler_echo.go:31
        ...

. .
— Format.

4.3.

New(code string, msg string, args ...interface{}) error

WithCause(code string, msg string, causeErr error, args ...interface{}) error

, args ...interface{}.

. , :

myerr = myerror.WithCause("8001", "Failed to read HTTP body: reqID", err, reqID)

4.4.

, INFO trace. , , , ,
WithCause()
INFO. .
, ERROR , trace.

HTTP :

requestBuf, err := ioutil.ReadAll(r.Body)
if err != nil {
    myerr = myerror.WithCause("8001", "Failed to read HTTP body: reqID", err, reqID)
    mylog.PrintfErrorInfo(myerr) //    
    s.processError(myerr, w, http.StatusInternalServerError, reqID) //     HTTP response
    return myerr
}

processError HTTP / .
— HTTPErrLog, POST /httperrlog.

POST /httperrlog HTTP/1.1
HTTP-Err-Log: HEADER | BODY

HTTP header

// carriage return (CR, ASCII 0xd), line feed (LF, ASCII 0xa), and the zero character (NUL, ASCII 0x0)
headerReplacer := strings.NewReplacer("\x0a", " ", "\x0d", " ", "\x00", " ")

w.Header().Set("Err-Trace", headerReplacer.Replace(myerr.Trace))

5.

github.com/hashicorp/logutils.
RFC 5424 — The Syslog Protocol, :

debug —
info — , , /
error — ,

— 100

Dave Cheney Let’s talk about logging.
— A simple logging interface for Go.

5.1.

"log". github.com/hashicorp/logutils.

// logFilter represent a custom logger seting
var logFilter = &logutils.LevelFilter{
    Levels:   []logutils.LogLevel{"DEBUG", "INFO", "ERROR"},
    MinLevel: logutils.LogLevel("INFO"), // initial setting
    Writer:   os.Stderr,                 // initial setting
}

// InitLogger init custom logger
func InitLogger(wrt io.Writer) {
    logFilter.Writer = wrt   // custom logger
    log.SetOutput(logFilter) // set std logger to our custom
}

, , main MultiWriter

//     
if logFileFlag != "" {
    //        
    logFileFlag = strings.Replace(logFileFlag, "%s", time.Now().Format("2006_01_02_150405"), 1)

    //        APPEND
    logFile, err := os.OpenFile(logFileFlag, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
    if err != nil {
        myerr := myerror.WithCause("6020", "Error open log file: Filename", err, logFileFlag)
        mylog.PrintfErrorMsg(fmt.Sprintf("%+v", myerr))
        return myerr
    }
    if logFile != nil {
        defer logFile.Close()
    }

    wrt := io.MultiWriter(os.Stderr, logFile) //    os.Stderr  

    mylog.InitLogger(wrt) //     
} else {
    mylog.InitLogger(os.Stderr)
}

5.2.

INFO DEBUG .

2020/03/10 17:56:39 [INFO] - httpserver.go:[61] - New() - Creating new HTTP server
2020/03/10 17:56:39 [INFO] - httpserver.go:[141] - New() - Created new TCP listener: network = 'tcp', address ['127.0.0.1:3000']
2020/03/10 17:56:39 [INFO] - httpserver.go:[155] - New() - Handler is registered: Path, Method ['/echo', 'POST']
2020/03/10 17:56:39 [INFO] - httpserver.go:[155] - New() - Handler is registered: Path, Method ['/signin', 'POST']
2020/03/10 17:56:39 [INFO] - httpserver.go:[155] - New() - Handler is registered: Path, Method ['/refresh', 'POST']
2020/03/10 17:56:39 [INFO] - httpserver.go:[179] - New() - HTTP server is created
2020/03/10 17:56:39 [INFO] - daemon.go:[121] - New() - New daemon is created
2020/03/10 17:56:39 [INFO] - daemon.go:[128] - (*Daemon).Run() - Starting daemon
2020/03/10 17:56:39 [INFO] - daemon.go:[133] - (*Daemon).Run() - Daemon is running. For exit <CTRL-c>
2020/03/10 17:56:39 [INFO] - httpserver.go:[209] - (*Server).Run() - Starting HTTP server
2020/03/10 17:57:22 [INFO] - daemon.go:[142] - (*Daemon).Run() - Exiting, got signal ['interrupt']
2020/03/10 17:57:22 [INFO] - daemon.go:[155] - (*Daemon).Shutdown() - Shutting down daemon
2020/03/10 17:57:22 [INFO] - httpserver.go:[215] - (*Server).Shutdown() - Waiting for shutdown HTTP Server: sec ['30']
2020/03/10 17:57:22 [INFO] - httpserver.go:[231] - (*Server).Shutdown() - HTTP Server shutdown successfuly
2020/03/10 17:57:22 [INFO] - daemon.go:[167] - (*Daemon).Shutdown() - Daemon is shutdown
2020/03/10 17:57:22 [INFO] - main.go:[163] - main.func1() - Server is shutdown

ERROR .

5.3.

func PrintfInfoMsg(mes string, args ...interface{}) {
    printfMsg("[INFO]", 0, mes, args...)
}
func PrintfDebugMsg(mes string, args ...interface{}) {
    printfMsg("[DEBUG]", 0, mes, args...)
}
func PrintfErrorMsg(mes string, args ...interface{}) {
    printfMsg("[ERROR]", 0, mes, args...)
}

, .
— DEBUG .

INFO DEBUG, .

mylog.PrintfInfoMsg(fmt.Sprintf("Create new TCP listener network='tcp', address='%s'", serverCfg.ListenSpec))

mylog.PrintfInfoMsg("Created new TCP listener: network = 'tcp', address", cfg.ListenSpec)

, , DEBUG INFO 1%.
— INFO DEBUG POST /loglevel.

POST /loglevel HTTP/1.1
Log-Level-Filter: DEBUG | ERROR | INFO

5.4. HTTP

, HTTP .
httplog HTTP header body:

— HttpLog HTTPLogType, POST /httplog.

POST /httplog HTTP/1.1
Http-Log: TRUE
Http-Log-Type: INREQ | OUTREQ | INRESP | OUTRESP | BODY

6.

: HTTP Basic Authentication MS AD Authentication.
MS AD Authentication gopkg.in/korylprince/go-ad-auth.v2.
, MS AD.

[AUTHENTIFICATION]
AuthType = INTERNAL | MSAD | NONE
MSADServer = company.com
MSADPort = 389
MSADBaseDN = OU=company, DC=dc, DC=corp
MSADSecurity = SecurityNone

HTTP Basic Authentication .

JSON Web Token (JWT) . JWTExpiresAt (JWTExpiresAt=0 — ). JWT .

[JWT]
UseJWT = true
JWTExpiresAt = 20000

JWT github.com/dgrijalva/jwt-go.
JWT: , , cookie jwt

JWT :

JWT , HTTP Basic Authentication MS AD Authentication
JWT , (StatusUnauthorized), JWT
JWT POST /signin
- JSON Web Token Claim ( Claim ),
- Claims JSON Web Token, HS256
- Token http Cookie "token". Cookie Token
, http Cookie JWT . , StatusUnauthorized
JWT POST /refresh
logout

7.

7.1. go mod

golang/dep/cmd/dep go mod, vendor . "" . .

go get -u ./../...
go mod vendor

go mod , . commit, :

go get github.com/ibm-messaging/mq-golang/ibmmq@19b946c

, vendor,

go build -v -mod vendor

7.2.

make ( - ):

rebuild —
build —
check - check code using github.com/golangci/golangci-lint

When compiling, the version, build date, and commit are embedded in the executable file. This information is displayed in a log file - very useful for parsing errors.
To implement such an implementation, we add variables to main

var (
    version   = "0.0.2" //  ,  
    commit    = "unset" //  commit
    buildTime = "unset" //    
)

When compiling in a makefile, we request git for commit

COMMIT?=$(shell git rev-parse --short HEAD)
BUILD_TIME?=$(shell date -u '+%Y-%m-%d_%H:%M:%S')

When assembling, we bind these values to previously declared variables (option -ldflags "-X" ), at the same time indicate the OSes under which we collect and the storage location of the binary

GOOS=${GOOS} go build -v -a -mod vendor \
-ldflags "-X main.commit=${COMMIT} -X main.buildTime=${BUILD_TIME}" \
-o bin/${GOOS}/${APP}

In main.main, I write the final version, commit and build date information into a variable

app.Version = fmt.Sprintf("%s, commit '%s', build time '%s'", version, commit, buildTime)

Golang backend server template - part 1 (HTTP server)

1.

2.