Get best of the week

Managing Packages with Go Modules: A Pragmatic Guide

Hello everyone. In anticipation of the start of the Golang Developer course , we have prepared another interesting translation for you.


Modules are a way to deal with dependencies in Go. Initially presented as an experiment, the modules are supposed to be introduced onto the field as a new standard for managing packages from version 1.13.

I find this topic unusual enough for beginners coming from other languages, and so I decided to collect some thoughts and tips here to help others like me get an idea of ​​package management in Go. We will start with a general introduction, and then move on to the less obvious aspects, including using the vendor folder, using modules with Docker in development, tool dependencies, etc.

If you are already familiar with Go modules and know the Wiki, like the back of your hand, this article probably won’t be very helpful to you. But for the rest, however, it can save several hours of trial and error.

So if you are on the way, jump in and enjoy the ride.



Quick start


If version control is already integrated in your project, you can simply run

go mod init

Or specify the path to the module manually. This is something like a name, URL, and import path for your package:

go mod init github.com/you/hello

This will create a file go.mod, which also defines the project requirements and lochit depending on their correct version (as an analogy for you, it's like package.json, and package-lock.jsoncombined into a single file):

module github.com/you/hello
go 1.12

Run go getto add a new dependency to your project:

Note that although you cannot specify a range of versions with go get, what you define here is not a specific version, but a minimum version. As we will see later, there is a way to gracefully update dependencies according to semver.

# use Git tags
go get github.com/go-chi/chi@v4.0.1
# or Git branch name
go get github.com/go-chi/chi@master
# or Git commit hash
go get github.com/go-chi/chi@08c92af

Now our file is go.modas follows:

module github.com/you/hello
go 1.12
require github.com/go-chi/chi v4.0.2+incompatible // indirect

The suffix is +incompatibleadded to all packages that are not yet configured for Go modules or violate their version control rules.

Since we have not imported this package anywhere in our project, it was marked as // indirect. We can tidy this up with the following command:

go mod tidy

Depending on the current state of your repository, it will either delete an unused module or delete a comment // indirect.

If any dependency by itself does not have go.mod(for example, it is not yet configured for modules), then all its dependencies will be written to the parent file go.mod(as an option, your file go.mod)along with a comment // indirectto indicate that they are not from direct import In your

global plan, the goal go mod tidyis also to add any dependencies needed for other combinations of OS, architectures and build tags. Be sure to run it before each release.

Also make sure that a file is created after adding the dependencygo.sum. You might think that this is a lock file. But in fact it go.modalready provides enough information for 100% reproducible builds. The file go.sumis created for verification purposes: it contains the expected cryptographic checksums of the contents of individual versions of the module.

Partly because it is go.sumnot a lock file, it saves the written checksums for the module version even after you stop using this module. This allows you to check the checksums if you resume using it later, which provides additional security.


Mkcert just migrated to the modules (with vendor / for backward compatibility) and everything went smoothly
https://github.com/FiloSottile/mkcert/commit/26ac5f35395fb9cba3805faf1a5a04d260271291

$ GO111MODULE=on go1.11rc1 mod init
$ GO111MODULE=on go1.11rc1 mod vendor
$ git add go.mod go.sum vendor
$ git rm Gopkg.lock Gopkg.toml Makefile



FAQ: Should I commit go.sumin git?
A: Definitely yes. With it, the owners of your sources do not need to trust other GitHub repositories and owners of custom import paths. Already on the way to us, something better, but for now this is the same model as the hashes in the lock files.

The go buildand commands go testwill automatically load all missing dependencies, although you can do this explicitly with the help of go mod downloadpre-populating local caches that may be useful for CI.

By default, all of our packages from all projects are loaded into the directory $GOPATH/pkg/mod. We will discuss this in more detail later.

Upgrading Package Versions


You can use go get -ueither go get -u=patchto update dependencies to the latest minor version or patch, respectively.

But you cannot upgrade to major versions like that. The code included in Go modules must technically comply with the following rules:

  • Match semver (example tag VCS v1.2.3).
  • If the module is version v2 or higher, the major version of the module should be included both /vNat the end of the module path used in the file go.modand in the package import path:

import "github.com/you/hello/v2"

Apparently, this is done so that different versions of packages can be imported in one assembly (see diamond dependency problem ).

In a nutshell, Go expects you to be very careful when introducing major versions.

Replacing imported modules


You can specify the necessary module for your own fork or even the local path to the file using the directive replace:

go mod edit -replace github.com/go-chi/chi=./packages/chi

Result:

module github.com/you/hello
go 1.12
require github.com/go-chi/chi v4.0.2+incompatible
replace github.com/go-chi/chi => ./packages/chi

You can delete the line manually or run:

go mod edit -dropreplace github.com/go-chi/chi

Project Dependency Management


Historically, all Go code was stored in one giant mono-repository, because that's how Google organizes its code base, and this affects the language design.

Go modules are a departure from this approach. You no longer need to keep all your projects in $GOPATH.

However, technically all of your downloaded dependencies are still placed in $GOPATH/pkg/mod. If you use Docker containers for local development, this can be a problem, since dependencies are stored outside the project. By default, they are simply not visible in your IDE.



This is usually not a problem for other languages, but this is what I first encountered when working with the Go codebase.

Fortunately, there are several (undocumented) ways to solve this problem.

Option 1. Install GOPATH inside your project directory.


At first glance, this may seem counterintuitive, but if you run Go from a container , you can override GOPATH so that it points to the project directory so that packages are accessible from the host:

version: '3.7'

services:
  app:
    command: tail -f /dev/null
    image: golang:1.12.6-stretch
    environment:
      #        - /code/.go/pkg/mod
      - GOPATH=/code/.go
    ports:
      - 8000:8000
    volumes:
      - ./:/code:cached
    working_dir: /code

Popular IDEs should be able to install GOPATH at the project (workspace) level:



The only drawback of this approach is the lack of interaction with the Go runtime on the host computer. You must execute all Go commands inside the container.

Option 2: Vending Your Dependencies


Another way is to copy the dependencies of your project to a folder vendor:

go mod vendor

It should be noted right away: we DO NOT allow Go to directly upload materials to the vendor folder: this is not possible with modules. We simply copy already downloaded packages.

In addition, if you unfasten your dependencies, as in the example above, then clear $GOPATH/pkg/modand then try to add several new dependencies to your project, you will see the following:

  1. Go will rebuild the download cache for all software packages $GOPATH/pkg/mod/cache.
  2. All loaded modules will be copied to $GOPATH/pkg/mod.
  3. And finally, Go will copy these modules to a vendorfolder, deleting examples, tests, and some other files that you don’t directly depend on.

Moreover, there are a lot of things missing in this newly created vendor folder:



A typical Docker Compose file looks like this (note the volume bindings):

version: '3.7'

services:
  app:
    command: tail -f /dev/null
    image: golang:1.12.6-stretch
    ports:
      - 8000:8000
    volumes:
     #    go,           
      - modules:/go/pkg/mod/cache
      - ./:/code:cached
    working_dir: /code 

volumes:
  modules:
    driver: local

Please note that I do NOT comic this vendor folder in the version control system or am not going to use it in production. This is a strictly local development script, which can usually be found in some other languages.

However, when I read comments from some Go maintainers and some offers related to partial vending (?), I get the impression that this feature was originally intended not for this user case.

One of the commentators on reddit helped me shed some light on this:

Usually people vendor their dependencies for reasons such as the desire to have tight assemblies without access to the network, as well as the availability of a copy of ready-made dependencies in the event of a github failure or the repository disappearing, and the possibility of an easier audit of changes in dependencies using standard VCS tools, etc. .

Yeah, it does not look like anything from the fact that I might be interested.

According to the Go command, you can easily enable vending by setting an environment variable GOFLAGS=-mod=vendor. I do not recommend doing this. Using flags will simply break go getwithout providing any other benefits to your daily workflow:



In fact, the only place you need to enable vending is your IDE:



After some trial and error, I came up with the following procedure for adding vendor dependencies in this approach.

Step 1. Requirement


You can require a dependency with go get:

go get github.com/rs/zerolog@v1.14.3

Step 2. Import


Then import it somewhere in your code:

import (
   _ "github.com/rs/zerolog"
)

Step 3. Vending


Finally, re-open your dependencies:

go mod vendor

There is a pending proposal to allow go mod vendor to accept certain module templates that may (or may not) resolve some of the problems associated with this workflow.

go mod vendoralready automatically requires missed imports, so step 1 is optional in this workflow (if you do not want to specify version restrictions). However, without step 2, it will not pick up the downloaded package.

This approach works better with the host system, but it is rather confusing when it comes to editing your dependencies.


Personally, I think redefining GOPATH is a cleaner approach as it does not sacrifice functionality go get. Nevertheless, I wanted to show both strategies, because the vendor folder may be more familiar to people coming from other languages, such as PHP, Ruby, Javascript, etc. As you can see from the fraud described in this article, this not a particularly good choice for Go.

Tool dependencies


We may need to install some Go-based tools that are not imported, but used as part of the project development environment. A simple example of such a tool is CompileDaemon , which can monitor your code for changes and restart your application.

The officially recommended approach is to add a tools.gofile (the name does not matter) with the following content:

// +build tools
package tools
import (
_ "github.com/githubnemo/CompileDaemon"
)

  • The limitation // +build toolsprevents your regular assemblies from actually importing your tool.
  • The import expression allows go commands to accurately write version information of your tools to go.modyour module file .

So that is all. I hope you won’t be as confused as I was when I first started using Go modules. You can visit the Go Modules wiki for more details.


Get on the course.

More articles:


All Articles