Intro

Athens Logo

Athens is a Server for Your Go Packages

Welcome, Gophers! Athens is an open source enterprise ready Go Module proxy with extensive configuration for a variety of online and offline usecases. It is in use for anti-censorship, compliance, data privacy, and data continuity usecases in homes and corporations across the globe today.

How To Get Started?

Run docker run -p '3000:3000' gomods/athens:latest

Then, set up your GOPROXY and go get going!

export GOPROXY=http://localhost:3000 && go get module@v1

When you’re ready to run something more production ready, Athens can run on on a variety of platforms including AWS, Azure, GCP, Digital Ocean, Alibaba, and bare metal.

What Does Athens Do?

Athens is an implementation of the Go Module proxy. Go clients talk to Athens to retrieve packages at its most basic level. Athens supports many usecases on top of that basic premise.

Not Ready to try Athens Yet?

Here are some other ways to get involved:

  • Read the full walkthrough with setting up, running and testing the Athens proxy explores this in greater depth.
  • Join our office hours! It’s a great way to meet folks working on the project, ask questions or just hang out. All are welcome to join and participate.
  • Check out our issue queue for good first issues
  • Join us in the #athens channel on the Gophers Slack

Athens banner attributed to Golda Manuel

Subsections of Intro

Introduction to Athens

Welcome to Athens, Gophers! We gave a very brief overview of Athens on the home page, so if you want to know more, you’ve come to the right place!

This section gives you all the details you need to understand what Athens does, why it exists, and how it fits into your workflow.

Where to Go From Here

We recommend you read this section from front to back:

Subsections of Introduction to Athens

Athens 101

What is Athens?

Shortly: Athens is a project building on top of vgo (or go1.11+) trying to bring dependencies closer to you so you can count on repeatable builds even at a time when VCS is down.

The big goal of Athens is to provide a new place where dependencies — not code — live. Dependencies are immutable blobs of code and associated metadata that come from Github. They live in storage that Athens controls.

You probably already know what “immutable” means, but let me just point it out again because it’s really important for this whole system. When folks change their packages, iterate, experiment, or whatever else, code on Athens won’t change. If the package author releases a new version, Athens will pull that down and it’ll show up. So if you depend on package M version v1.2.3, it will never change on Athens. Not even after force push, not even after repo cease to exist.

Download protocol

Athens builds on top of Go CLI which specifies a set of endpoints with which it communicates with external proxies providing modules. This set of endpoints we call Download Protocol

The original vgo research paper on Download protocol can be found here: https://research.swtch.com/vgo-module

Each of these endpoints sits on top of a module. Let’s assume module htp authored by acidburn.

So for each of the endpoints mentioned below we will assume address acidburn/htp/@v/{endpoint} (e.g acidburn/htp/@v/list)

In the examples below, $HOST and $PORT are placeholders for the host and port of your Athens server.

List of versions

This endpoint returns a list of versions that Athens knows about for acidburn/htp. The list is just separated by newlines:

GET $HOST:$PORT/github.com/acidburn/htp/@v/list
v0.1.0
v0.1.1
v1.0.0
v1.0.1
v1.2.0

Version info

GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.info

This returns JSON with information about v1.0.0. It looks like this:

{
    "Name": "v1.0.0",
    "Short": "v1.0.0",
    "Version": "v1.0.0",
    "Time": "1972-07-18T12:34:56Z"
}

Go.mod file

GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.mod

This returns the go.mod file for version v1.0.0. If $HOST:$PORT/github.com/acidburn/htp version v1.0.0 has no dependencies, the response body would look like this:

module github.com/acidburn/htp

Module sources

GET $HOST:$PORT/github.com/acidburn/htp/@v/v1.0.0.zip

This is what it sounds like — it sends back a zip file with the source code for the module in version v1.0.0.

Latest

GET $HOST:$PORT/github.com/acidburn/htp/@latest

This endpoint returns the latest version of the module. If the version does not exist it should retrieve the hash of latest commit.

Why Does It Matter?

Immutability

The Go community has had lots of problems with libraries disappearing or changing without warning. It’s easy for package maintainers to make changes to their code that can break yours - and much of the time it’s an accident! Could your build break if one of your dependencies did this?

  • Commit abdef was deleted
  • Tag v0.1.0 was force pushed
  • The repository was deleted altogether

Since your app’s dependencies come directly from a VCS (Version Control System, such as GitHub), any of those above cases can happen to you and your builds can break when they do - oh no! Athens solves these problems by copying code from VCS’s into immutable storage.

This way, you don’t need to upload anything manually to Athens storage. The first time Go asks Athens for a dependency, Athens will go get it from VCS (github, bitbucket etc). But once that module has been retrieved, it will be forever persisted in its storage backend and the proxy will never go back to VCS for that same version again. This is how Athens achieves module immutability. Keep in mind, you are in charge of that storage backend.

Logic

The fact that the Go command line can now ping your own server to download dependencies, that means you can program whatever logic you want around providing such dependencies. Things like Access Control (discussed below), adding custom versions, custom forks, and custom packages. For example, Athens provides a Validation Hook that will get called for every module download to determine whether a module should be downloaded or not. Therefore, you can extend Athens with your own logic such as scanning a module path or code for red flags etc.

Performance

Downloading stored dependencies from Athens is significantly faster than downloading dependencies from Version Control Systems. This is because go get by default uses VCS to download modules such as git clone, while go get with GOPROXY enabled will use HTTP to download zip archives. Therefore, depending on your computer and internet connection speed, it takes 10 seconds to download the CockroachDB source tree as a zip file from GitHub but almost four minutes to git clone it.

Access Control

Worse than packages disappearing, packages can be malicious. To make sure no such malicious package is ever installed by your team or company, you can have your proxy server return a 500 when the Go command line asks for an excluded module. This will cause the build to fail because Go expects a 200 HTTP response code. With Athens, you can achieve this through the filter file.

Vendor Directory Becomes Optional

With immutability, performance, and a robust proxy server, there’s no longer an absolute need for each repository to have its vendor directory checked in to its version control. The go.sum file ensures that no package is manipulated after the first install. Furthermore, your CI/CD can install all of your dependencies on every build with little time.

Components

From a very high-level view, there are 3 major components of the system.

Client

The client is a user, powered by go binary with module support. At the moment of writing this document, it is Go v1.12+.

VCS

VCS is an external source of data for Athens. Athens scans various VCSs such as github.com and fetches sources from there.

Proxy

We intend proxies to be deployed primarily inside of enterprises to:

  • Host private modules
  • Exclude access to public modules
  • Store public modules

Importantly, a proxy is not intended to be a complete mirror of an upstream proxy. For public modules, its role is to store and provide access control.

Installing Athens

The Go ecosystem has always been federated and completely open. Anyone with a GitHub or GitLab (or any other supported VCS) account can effortlessly provide a library with just a git push (or similar). No extra accounts to create or credentials to set up.

A Federated Ecosystem

We feel that Athens should keep the community federated and open, and nobody should have to change their workflow when they’re building apps and libraries. So, to make sure the community can stay federated and open, we’ve made it easy to install Athens for everyone so that:

  • Anyone can run their own full-featured mirror, public or private
  • Any organization can run their own private mirror, so they can manage their private code just as they would their public code

Immutability

As you know, go get and go mod download will fetch packages directly from version control systems like GitHub. This system has been mostly great for both package developers and the dependent apps, but at the same we’ve suffered from a fundamental problem for a long time.

Code in version control systems can always change even after it’s been committed. For example, a package developer can run git push -f and overwrite a commit or tag that you depend on in your project. In these cases, you’ll often see checksum verification errors (for example, see here).

Athens prevents these issues by storing code in its own, immutable database. Here’s what happens when you run go get:

  1. go get requests a module from Athens
  2. Athens accepts the request and begins looking for the module
  3. First, it looks in its storage. If it finds the module, Athens immediately sends it back to the go get client from (1)
  4. If it doesn’t find the module, it fetches the module from the version control system, saves in storage, and returns to the client from (1)

Athens never changes anything once it saves a module to storage, so the system has the following two important properties:

  • Athens will only ever call go mod download once per module version. In other words, Athens will only hit step (4) once for any given module & version
  • Athens treats storage as append-only, so once a module is saved, it never changes, even if a developer changes it in GitHub

Release Scheme

We follow semver. Our Docker images are tagged to indicate stability:

  • latest = the most recent stable release
  • canary = the most recent build of master

We strongly recommend using a tagged release, e.g. gomods/athens:v0.3.0, instead of the latest or canary tags.

Where to Go from Here

To make sure it’s easy to install, we try to provide as many ways as possible to install and run Athens:

Subsections of Installing Athens

Building a versioned Athens binary from source

You can do that easily with just few commands:

Bash

git clone https://github.com/gomods/athens
cd athens
make build-ver VERSION="0.2.0"

PowerShell

git clone https://github.com/gomods/athens
cd athens
$env:GO111MODULE="on"
$env:GOPROXY="https://proxy.golang.org"
$version = "0.2.0"
$date = (Get-Date).ToUniversalTime()
go build -ldflags "-X github.com/gomods/athens/pkg/build.version=$version -X github.com/gomods/athens/pkg/build.buildDate=$date" -o athens ./cmd/proxy

This will give you a binary named athens. You can print the version and time information by running:

 ./athens -version

which should return something like:

Build Details:
        Version:        0.2.0
        Date:           2018-12-13-20:51:06-UTC

Install Athens on Kubernetes

When you follow the instructions in the Walkthrough, you end up with an Athens Proxy that uses in-memory storage. This is only suitable for trying out the Athens proxy for a short period of time, as you will quickly run out of memory and Athens won’t persist modules between restarts. In order to run a more production-like proxy, you may want to run Athens on a Kubernetes cluster. To aid in deployment of the Athens proxy on Kubernetes, a Helm chart has been provided. This guide will walk you through installing Athens on a Kubernetes cluster using Helm.


Prerequisites

In order to install Athens on your Kubernetes cluster, there are a few prerequisites that you must satisfy. If you already have completed the following steps, please continue to configuring helm. This guide assumes you have already created a Kubernetes cluster.

Install the Kubernetes CLI

In order to interact with your Kubernetes Cluster, you will need to install kubectl.

Install The Helm CLI

Helm is a tool for installing pre-configured applications on Kubernetes. Install helm by running the following command:

MacOS

brew install kubernetes-helm

Windows

  1. Download the latest Helm release.
  2. Decompress the tar file.
  3. Copy helm.exe to a directory on your PATH.

Linux

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash

Configure Helm

If your cluster has already been configured to use Helm, please continue to deploy Athens.

If not, please read on.

RBAC Cluster

If your cluster has RBAC enabled, you will need to create a ServiceAccount, ClusterRole and ClusterRoleBinding for Helm to use. The following command will create these and initialize Helm.

kubectl create -f https://raw.githubusercontent.com/Azure/helm-charts/master/docs/prerequisities/helm-rbac-config.yaml
helm init --service-account tiller

Non RBAC Cluster

If your cluster has does not have RBAC enabled, you can simply initialize Helm.

helm init

Before deploying Athens, you will need to wait for the Tiller pod to become Ready. You can check the status by watching the pods in kube-system:

$ kubectl get pods -n kube-system -w
NAME                                    READY     STATUS    RESTARTS   AGE
tiller-deploy-5456568744-76c6s          1/1       Running   0          5s

Deploy Athens

The fastest way to install Athens using Helm is to deploy it from our public Helm chart repository. First, add the repository with this command:

$ helm repo add gomods https://gomods.github.io/athens-charts
$ helm repo update

Next, install the chart with default values to athens namespace:

$ helm install athens gomods/athens-proxy --namespace athens

This will deploy a single Athens instance in the athens namespace with disk storage enabled. Additionally, a ClusterIP service will be created.

Advanced Configuration

Replicas

By default, the chart will install Athens with a replica count of 1. To change this, change the replicaCount value:

helm install athens gomods/athens-proxy --namespace athens --set replicaCount=3

Resources

By default, the chart will install Athens without specific resource requests or limits. To change this, change the resources value:

helm install athens gomods/athens-proxy --namespace athens \
  --set resources.requests.cpu=100m \
  --set resources.requests.memory=64Mi \
  --set resources.limits.cpu=100m \
  --set resources.limits.memory=64Mi

For more information, see Managing Compute Resources for Containers in the Kubernetes documentation.

Give Athens access to private repositories via Github Token (Optional)

  1. Create a token at https://github.com/settings/tokens
  2. Provide the token to the Athens proxy either through the config.toml file (the GithubToken field) or by setting the ATHENS_GITHUB_TOKEN environment variable.

Storage Providers

The Helm chart currently supports running Athens with two different storage providers: disk and mongo. The default behavior is to use the disk storage provider.

Disk Storage Configuration

When using the disk storage provider, you can configure a number of options regarding data persistence. By default, Athens will deploy using an emptyDir volume. This probably isn’t sufficient for production use cases, so the chart also allows you to configure persistence via a PersistentVolumeClaim. The chart currently allows you to set the following values:

persistence:
  enabled: false
  accessMode: ReadWriteOnce
  size: 4Gi
  storageClass:

Add it to override-values.yaml file and run:

helm install gomods/athens-proxy -n athens --namespace athens -f override-values.yaml

enabled is used to turn on the PVC feature of the chart, while the other values relate directly to the values defined in the PersistentVolumeClaim documentation.

Mongo DB Configuration

To use the Mongo DB storage provider, you will first need a MongoDB instance. Once you have deployed MongoDB, you can configure Athens using the connection string via storage.mongo.url. You will also need to set storage.type to “mongo”.

helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=mongo --set storage.mongo.url=<some-mongodb-connection-string>

S3 Configuration

To use S3 storage with Athens, set storage.type to s3 and set storage.s3.region and storage.s3.bucket to the desired AWS region and S3 bucket name, respectively. By default, Athens will attempt to load AWS credentials using the AWS SDK from the chain of environment variables, shared credentials files, and EC2 instance credentials. To manually specify AWS credentials, set storage.s3.access_key_id, storage.s3.secret_access_key, and change storage.s3.useDefaultConfiguration to false.

helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=s3 --set storage.s3.region=<your-aws-region> --set storage.s3.bucket=<your-bucket>

Minio Configuration

To use S3 storage with Athens, set storage.type to minio. You need to set storage.minio.endpoint as the URL of your minio-installation. This URL can also be an kubernetes-internal one (e.g. something like minio-service.default.svc). You need to create a bucket inside your minio-installation or use an existing one. The bucket needs to be referenced in storage.minio.bucket. Last athens need authentication credentials for your minio in storage.minio.accessKey and storage.minio.secretKey.

helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=minio --set storage.minio.endpoint=<your-minio-endpoint> --set storage.minio.bucket=<your-bucket> --set storage.minio.accessKey=<your-minio-access-key> --set storage.minio.secretKey=<your-minio-secret-key>

Google Cloud Storage

To use Google Cloud Storage storage with Athens, set storage.type to gcp. You need to set storage.gcp.projectID and storage.gcp.bucket to the desired GCP project and bucket name, respectively.

Depending on your deployment environment you will also need to set storage.gcp.serviceAccount to a key which has read/write access to the GCS bucket. If you are running Athens inside GCP, you will most likely not need this as GCP figures out internal authentication between products for you.

helm install gomods/athens-proxy -n athens --namespace athens --set storage.type=gcp --set storage.gcp.projectID=<your-gcp-project> --set storage.gcp.bucket=<your-bucket>

Kubernetes Service

By default, a Kubernetes ClusterIP service is created for the Athens proxy. “ClusterIP” is sufficient in the case when the Athens proxy will be used from within the cluster. To expose Athens outside of the cluster, consider using a “NodePort” or “LoadBalancer” service. This can be changed by setting the service.type value when installing the chart. For example, to deploy Athens using a NodePort service, the following command could be used:

helm install gomods/athens-proxy -n athens --namespace athens --set service.type=NodePort

Ingress Resource

The chart can optionally create a Kubernetes Ingress Resource for you as well. To enable this feature, set the ingress.enabled resource to true.

helm install gomods/athens-proxy -n athens --namespace athens --set ingress.enabled=true

Further configuration values are available in the values.yaml file:

ingress:
  enabled: true
  annotations:
    certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/tls-acme: "true"
    ingress.kubernetes.io/force-ssl-redirect: "true"
    kubernetes.io/ingress.class: nginx
  hosts:
    - athens.mydomain.com
  tls:
    - secretName: athens.mydomain.com
      hosts:
        - "athens.mydomain.com

Example above sets automatic creation/retrieval of TLS certificates from Let’s Encrypt with cert-manager and uses nginx-ingress controller to expose Athens externally to the Internet.

Add it to override-values.yaml file and run:

helm install gomods/athens-proxy -n athens --namespace athens -f override-values.yaml

Upstream module repository

You can set the URL for the upstream module repository then Athens will try to download modules from the upstream when it doesn’t find them in its own storage.

You have a few good options for what you can set as an upstream:

  • https://gocenter.io to use JFrog’s GoCenter
  • https://proxy.golang.org to use the Go Module Mirror
  • The URL to any other Athens server

The example below shows you how to set GoCenter up as upstream module repository:

upstreamProxy:
  enabled: true
  url: "https://gocenter.io"

Add it to override-values.yaml file and run:

helm install gomods/athens-proxy -n athens --namespace athens -f override-values.yaml

.netrc file support

A .netrc file can be shared as a secret to allow the access to private modules. The secret must be created from a netrc file using the following command (the name of the file must be netrc):

kubectl create secret generic netrcsecret --from-file=./netrc

In order to instruct athens to fetch and use the secret, netrc.enabled flag must be set to true:

helm install athens gomods/athens-proxy --namespace athens --set netrc.enabled=true

gitconfig support

A gitconfig file can be shared as a secret to allow the access to modules in private git repositories. For example, you can configure access to private repositories via HTTPS using personal access tokens on GitHub, GitLab and other git services.

First of all, prepare your gitconfig file:

cat << EOF > /tmp/gitconfig
[url "https://user:token@git.example.com/"]
    insteadOf = ssh://git@git.example.com/
    insteadOf = https://git.example.com/
EOF

Next, create the secret using the file created above:

kubectl create secret generic athens-proxy-gitconfig --from-file=gitconfig=/tmp/gitconfig

In order to instruct athens to use the secret, set appropriate flags (or parameters in values.yaml):

helm install athens gomods/athens-proxy --namespace athens \
    --set gitconfig.enabled=true \
    --set gitconfig.secretName=athens-proxy-gitconfig \
    --set gitconfig.secretKey=gitconfig

Using the Athens Docker images

Whether setting Athens up using Kubernetes or using the Walkthrough, you’ll most likely be using one of the images that the Athens project produces. This document details what images are available, and has a recap from the Walkthrough of how to use them on their own.


Available Docker images

The Athens project produces two docker images, available via Docker Hub

  1. A release version as gomods/athens, each tag corresponds with an Athens release, e.g. v0.7.1. Additionally, a canary tag is available and tracks each commit to main
  2. A tip version, as gomods/athens-dev, tagged with every commit to main, e.g. 1573339

For a detailed tags list, check each image’s Docker Hub

Running Athens as a Docker image

This is a quick recap of the Walkthrough

Using the docker cli

In order to run the Athens Proxy using docker, we need first to create a directory that will store the persitant modules. In the example below, the new directory is named athens-storage and is located in our userspace (i.e. $HOME). Then we need to set the ATHENS_STORAGE_TYPE and ATHENS_DISK_STORAGE_ROOT environment variables when we run the Docker container.

Bash

export ATHENS_STORAGE=$HOME/athens-storage
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   --name athens-proxy \
   --restart always \
   -p 3000:3000 \
   gomods/athens:latest

PowerShell

$env:ATHENS_STORAGE = "$(Join-Path $HOME athens-storage)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
   -e ATHENS_STORAGE_TYPE=disk `
   --name athens-proxy `
   --restart always `
   -p 3000:3000 `
   gomods/athens:latest

Non-Root User

The Athens docker images comes with a non-root user athens with uid: 1000, gid: 1000 and home directory /home/athens. In situations where running as root is not permitted, this user can be used instead. In all other instuctions replace /root/ with /home/athens/ and set the user and group ids in the run environment to 1000.

docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   -v "$PWD/gitconfig/.gitconfig:/home/athens/.gitconfig" \
   --name athens-proxy \
   --restart always \
   -p 3000:3000 \
   -u 1000:1000 \
   gomods/athens:latest

Troubleshooting Athens in Docker

init issues

The Athens docker image uses tini so that defunct processes get reaped. Docker 1.13 and greater includes tini and lets you enable it by passing the --init flag to docker run or by configuring the docker deamon with "init": true. When running in this mode. you may see a warning like this:

[WARN  tini (6)] Tini is not running as PID 1 and isn't registered as a child subreaper.
 Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
 To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.

This is the “Athens-tini” complaining that it’s not running as PID 1. There is no harm in that, since the zombie processes will be reaped by the tini included in Docker.

Shared Team Instance

When you follow the instructions in the Walkthrough, you end up with an Athens Proxy that uses in-memory storage. This is only suitable for trying out the Athens proxy for a short period of time, as you will quickly run out of memory and Athens won’t persist modules between restarts. This guide will help you get Athens running in a more suitable manner for scenarios like providing an instance for your development team to share.

We will use Docker to run the Athens proxy, so first make sure you have Docker installed.

Selecting a Storage Provider

Athens currently supports a number of storage drivers. For local, use we recommend starting with the local disk provider. For other providers, please see the Storage Provider documentation.

Running Athens with Local Disk Storage

In order to run Athens with disk storage, you will next need to identify where you would like to persist modules. In the example below, we will create a new directory named athens-storage in our current directory. Now you are ready to run Athens with disk storage enabled. To enable disk storage, you need to set the ATHENS_STORAGE_TYPE and ATHENS_DISK_STORAGE_ROOT environment variables when you run the Docker container.

The examples below use the :latest Docker tags for simplicity, however we strongly recommend that after your environment is up and running that you switch to using an explicit version (for example :v0.3.0).

Bash

export ATHENS_STORAGE=~/athens-storage
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   --name athens-proxy \
   --restart always \
   -p 3000:3000 \
   gomods/athens:latest

PowerShell

$env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
   -e ATHENS_STORAGE_TYPE=disk `
   --name athens-proxy `
   --restart always `
   -p 3000:3000 `
   gomods/athens:latest

Note: if you have not previously mounted this drive with Docker for Windows, you may be prompted to allow access

Athens should now be running as a Docker container with the local directory, athens-storage mounted as a volume. When Athens retrieves the modules, they will be will be stored in the directory previously created. First, let’s verify that Athens is running:

$ docker ps
CONTAINER ID        IMAGE                               COMMAND           PORTS                    NAMES
f0429b81a4f9        gomods/athens:latest   "/bin/app"        0.0.0.0:3000->3000/tcp   athens-proxy

Now, we can use Athens from any development machine that has Go v1.12+ installed. To verify this, try the following example:

Bash

$ export GO111MODULE=on
$ export GOPROXY=http://127.0.0.1:3000
$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!

PowerShell

$env:GO111MODULE = "on"
$env:GOPROXY = "http://127.0.0.1:3000"
git clone https://github.com/athens-artifacts/walkthrough.git
cd walkthrough
$ go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!

We can verify that Athens handled this request by examining the Docker logs:

$ docker logs -f athens-proxy
time="2018-08-21T17:28:53Z" level=warning msg="Unless you set SESSION_SECRET env variable, your session storage is not protected!"
time="2018-08-21T17:28:53Z" level=info msg="Starting application at 0.0.0.0:3000"
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.info [200]
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.mod [200]
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.zip [200]

Now, if you view the contents of the athens_storage directory, you will see that you now have additional files representing the samplelib module.

Bash

$ ls -lr $ATHENS_STORAGE/github.com/athens-artifacts/samplelib/v1.0.0/
total 24
-rwxr-xr-x  1 jeremyrickard  wheel    50 Aug 21 10:52 v1.0.0.info
-rwxr-xr-x  1 jeremyrickard  wheel  2391 Aug 21 10:52 source.zip
-rwxr-xr-x  1 jeremyrickard  wheel    45 Aug 21 10:52 go.mod

PowerShell

$ dir $env:ATHENS_STORAGE\github.com\athens-artifacts\samplelib\v1.0.0\


    Directory: C:\athens-storage\github.com\athens-artifacts\samplelib\v1.0.0


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        8/21/2018   3:31 PM             45 go.mod
-a----        8/21/2018   3:31 PM           2391 source.zip
-a----        8/21/2018   3:31 PM             50 v1.0.0.info

When Athens is restarted, it will serve the module from this location without re-downloading it. To verify that, we need to first remove the Athens container.

docker rm -f athens-proxy

Now, we need to clear the local Go modules storage. This is needed so that your local Go command line tool will re-download the module from Athens. The following commands will clear the local module storage:

Bash

sudo rm -fr "$(go env GOPATH)/pkg/mod"

PowerShell

rm -recurse -force $(go env GOPATH)\pkg\mod

Now, we can re-run the Athens container:

Bash

docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   --name athens-proxy \
   --restart always \
   -p 3000:3000 \
   gomods/athens:latest

PowerShell

docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
   -e ATHENS_STORAGE_TYPE=disk `
   --name athens-proxy `
   --restart always `
   -p 3000:3000 `
   gomods/athens:latest

When we re-run our Go example, the Go cli will again download module from Athens. Athens, however, will not need to retrieve the module. It will be served from the Athens on-disk storage.

Bash

$ ls -lr $ATHENS_STORAGE/github.com/athens-artifacts/samplelib/v1.0.0/
total 24
-rwxr-xr-x  1 jeremyrickard  wheel    50 Aug 21 10:52 v1.0.0.info
-rwxr-xr-x  1 jeremyrickard  wheel  2391 Aug 21 10:52 source.zip
-rwxr-xr-x  1 jeremyrickard  wheel    45 Aug 21 10:52 go.mod

PowerShell

$ dir $env:ATHENS_STORAGE\github.com\athens-artifacts\samplelib\v1.0.0\


    Directory: C:\athens-storage\github.com\athens-artifacts\samplelib\v1.0.0


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        8/21/2018   3:31 PM             45 go.mod
-a----        8/21/2018   3:31 PM           2391 source.zip
-a----        8/21/2018   3:31 PM             50 v1.0.0.info

Notice that the timestamps given have not changed.

Next Steps:

Install on AWS Fargate (ECS)

In this document, we’ll show how to use AWS Fargate (ECS) to run the Athens proxy.


Selecting a Storage Provider

There is documentation about how to use environment variables to configure the various storage providers. However, for this particular example we will use Amazon S3 Storage (s3).

Before You Begin

This guide assumes you already have an AWS account as well as the necessary authentication and permissions to create resources in the account.

Whether you choose to create your resources using the awscli or use something like Terraform, the resources required are the same.

S3 Bucket

In order to persist modules, we will create a s3 bucket for storage.

Below are two examples of creating the s3 bucket using the awscli and Terraform.

awscli:

$ aws s3api create-bucket --bucket athens-proxy-us-east-1-123456789012 --region us-east-1

terraform:

resource "aws_s3_bucket" "cache" {
  bucket = "athens-proxy-us-east-1-123456789012"
}

note: it is a good idea to use environment, region, and/or account ID as components to the bucket name due to their globally unique naming rules.

ECS Task IAM Role

In order for the ECS container instances to use the s3 bucket, we will need to configure the task IAM role to include the proper allow rules.

Below is a least-privileged policy document in both JSON and Terraform to enable ECS containers s3 bucket access to store and retrieve cache assets.

json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::athens-proxy-us-east-1-123456789012"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::athens-proxy-us-east-1-123456789012/*"
        }
    ]
}

terraform:

resource "aws_iam_policy" "task_role" {
  name   = "athens-proxy-task-role"
  path   = "/"
  policy = data.aws_iam_policy_document.task_role_policy.json
}

data "aws_iam_policy_document" "task_role_policy" {
  statement {
    effect = "Allow"
    actions = [
      "s3:ListBucket",
      "s3:GetBucketLocation"
    ]
    resources = [aws_s3_bucket.cache.arn]
  }
  statement {
    effect = "Allow"
    actions = [
      "s3:PutObject",
      "s3:GetObject",
      "s3:DeleteObject"
    ]
    resources = ["${aws_s3_bucket.cache.arn}/*"]
  }
  statement {
    effect = "Allow"
    actions = [
      "sts:AssumeRole",
      "sts:TagSession"
    ]
    resources = ["*"]
  }
}

ECS Task Definition

In order for Athens to be able to authenticate to the s3 bucket, we will need to configure the storage variables associated with s3.

Below is an excerpt from a task definition that shows the minimum environment variables needed.

"environment": [
  {"name": "AWS_REGION", "value": "us-east-1"},
  {"name": "AWS_USE_DEFAULT_CONFIGURATION", "value": "true"},
  {"name": "ATHENS_STORAGE_TYPE", "value": "s3"},
  {"name": "ATHENS_S3_BUCKET_NAME", "value": "athens-proxy-us-east-1-123456789012"},
]

Install on Azure Container Instances

When you follow the instructions in the Walkthrough, you end up with an Athens Proxy that uses in-memory storage. This is only suitable for trying out the Athens proxy for a short period of time, as you will quickly run out of memory and Athens won’t persist modules between restarts. This guide will help you get Athens running in a more suitable manner for scenarios like providing an instance for your development team to share.

In this document, we’ll show how to use Azure Container Instances (ACI) to run the Athens proxy.

Selecting a Storage Provider

Athens currently supports a number of storage drivers. For quick and easy use on ACI, we recommend using the local disk provider. For more permanent use, we recommend using MongoDB or other more persistent infrastructure. For other providers, please see the storage provider documentation.

Required Environment Variables

Before executing any of the commands below, make sure you have the following environment variables set up on your system:

  • AZURE_ATHENS_RESOURCE_GROUP - The Azure Resource Group to install the container in. You need to already have one of these before installing Athens
    • See here for details on how to create a resource group
  • AZURE_ATHENS_CONTAINER_NAME - The name of the container. This should be alphanumeric and you can have - and _ characters
  • LOCATION - The Azure region to install the container in. See the previous link for an exhaustive list, but here’s a useful cheat sheet that you can use immediately, without reading any docs:
    • North America: eastus2
    • Europe: westeurope
    • Asia: southeastasia
  • AZURE_ATHENS_DNS_NAME - The DNS name to assign to the container. It has to be globally unique inside of the region you set (LOCATION)

Installing with the Disk Storage Driver

az container create \
-g "${AZURE_ATHENS_RESOURCE_GROUP}" \
-n "${AZURE_ATHENS_CONTAINER_NAME}-${LOCATION}" \
--image gomods/athens:v0.3.0 \
-e "ATHENS_STORAGE_TYPE=disk" "ATHENS_DISK_STORAGE_ROOT=/var/lib/athens" \
--ip-address=Public \
--dns-name="${AZURE_ATHENS_DNS_NAME}" \
--ports="3000" \
--location=${LOCATION}

Once you’ve created the ACI container, you’ll see a JSON blob that includes the public IP address of the container. You’ll also see the fully qualified domain name (FQDN) of the running container (it will be prefixed by AZURE_ATHENS_DNS_NAME).

Installing with the MongoDB Storage Driver

First, make sure you have the following environment variable set up:

  • AZURE_ATHENS_MONGO_URL - The MongoDB connection string. For example: mongodb://username:password@mongo.server.com/?ssl=true

Then run the create command:

az container create \
-g "${AZURE_ATHENS_RESOURCE_GROUP}" \
-n "${AZURE_ATHENS_CONTAINER_NAME}-${LOCATION}" \
--image gomods/athens:v0.3.0 \
-e "ATHENS_STORAGE_TYPE=mongo" "ATHENS_MONGO_STORAGE_URL=${AZURE_ATHENS_MONGO_URL}" \
--ip-address=Public \
--dns-name="${AZURE_ATHENS_DNS_NAME}" \
--ports="3000" \
--location=${LOCATION}

Once you’ve created the ACI container, you’ll see a JSON blob that includes the public IP address of the container. You’ll also see the fully qualified domain name (FQDN) of the running container (it will be prefixed by AZURE_ATHENS_DNS_NAME).

Install on Google Cloud Run

Google Cloud Run is a service that aims to bridge the gap between the maintainance benefits of serverless architecture and the flexibility of Kubernetes. It is built on top of the opensource Knative project. Deploying using Cloud Run is similar to deploying using Google App Engine with the benefits of a free tier and a simpler build process.

Selecting a Storage Provider

There is documentaion about how to use environment variables to configure a large number of storage providers; however, for this prarticular example we will use Google Cloud Storage(GCS) because it fits nicely with Cloud Run.

Before You Begin

This guide assumes you have completed the following tasks:

  • Signed up for Google Cloud
  • Installed the gcloud command line tool
  • Installed the beta plugin for the gcloud command line tool (this is how to set it up)
  • Created a (GCS) bucket for your go modules

Setup a GCS Bucket

If you do not already have GCS bucket you can set one up using the gsutil tool.

First select a region you would like to have your storage in. You can then create a bucket in that region using the following command substituting your in your region and bucket name.

$ gsutil mb -l europe-west-4 gs://some-bucket

Setup

Change the values of these environment variables to be appropriate for your application. For GOOGLE_CLOUD_PROJECT, this needs to be the name of the project that has your cloud run deployment in it. ATHENS_REGION should be the region that your cloud run instance will be in, and GCS_BUCKET should be the Google Cloud Storage bucket that Athens will store module code and metadata in..

$ export GOOGLE_CLOUD_PROJECT=your-project
$ export ATHENS_REGION=asia-northeast1
$ export GCS_BUCKET=your-bucket-name
$ gcloud auth login
$ gcloud auth configure-docker

You will then need to push a copy of the Athens docker image to your google cloud container registry.

Below is an example using v0.11.0, for the latest version, check out the latest Athens release

$ docker pull gomods/athens:v0.11.0

$ docker tag gomods/athens:v0.11.0 gcr.io/$GOOGLE_CLOUD_PROJECT/athens:v0.11.0

$ docker push gcr.io/$GOOGLE_CLOUD_PROJECT/athens:v0.11.0

Once you have the container image in your registry you can use gcloud to provision your Athens instance.

$ gcloud beta run deploy \
    --image gcr.io/$GOOGLE_CLOUD_PROJECT/athens:v0.11.0 \
    --platform managed \
    --region $ATHENS_REGION \
    --allow-unauthenticated \
    --set-env-vars=ATHENS_STORAGE_TYPE=gcp \
    --set-env-vars=GOOGLE_CLOUD_PROJECT=$GOOGLE_CLOUD_PROJECT \
    --set-env-vars=ATHENS_STORAGE_GCP_BUCKET=$GCS_BUCKET \
    athens

Once this command finishes is will provide a url to your instance, but you can always find this through the cli:

$ gcloud beta run services describe athens --platform managed --region $ATHENS_REGION | grep hostname

Install on Google App Engine

Google App Engine (GAE) is a Google service allows applications to be deployed without provisioning the underlying hardware. It is similar to Azure Container Engine which is covered in a previous section. This guide will demonstrate how you can get Athens running on GAE.

Selecting a Storage Provider

There is documentaion about how to use environment variables to configure a large number of storage providers; however, for this prarticular example we will use Google Cloud Storage(GCS) because it fits nicely with Cloud Run.

Before You Begin

This guide assumes you have completed the following tasks:

  • Signed up for Google Cloud
  • Installed the gcloud command line tool

Setup a GCS Bucket

If you do not already have GCS bucket you can set one up using the gsutil tool.

First select a global region you would like to have your storage in. You can then create a bucket in that region using the following command substituting your in your region and bucket name.

$ gsutil mb -l europe-west-4 gs://some-bucket

Setup

First clone the Athens repository

$ git clone https://github.com/gomods/athens.git

There is already a Google Application Engine scaffold set up for you. Copy it into a new file and make changes to the environment variables.

$ cd athens
$ cp scripts/gae/app.sample.yaml scripts/gae/app.yaml
$ code scripts/gae/app.yaml

Once you have configured the environment variables you can deploy Athens as a GAE service.

$ make deploy-gae

Install Athens with BOSH

Athens can be deployed in many ways. The following guide explains how to use BOSH, a deployment and lifecycle tool, in order to deploy an Athens server on virtual machines (VMs) on any infrastructure as a service (IaaS) that is supported by BOSH.


Prerequisites

Install BOSH

Make sure to have the BOSH CLI installed and set up a BOSH Director on an infrastructure of your choice.

Setup the Infrastructure

If you choose to deploy on a IaaS provider, there are a few prerequisites that need to be set up before starting with the deployment. Depending on which IaaS you will be deploying, you may need to create:

  • Public IP: a public IP address for association with the Athens VM.

  • Firewall Rules: the following ingress ports must be allowed

    • 3000/tcp - Athens proxy port (if you specify a different port than the default port 3000 in the job properties, adapt this rule accordingly).

    Egress traffic should be restricted depending on your requirements.

Amazon Web Services (AWS)

AWS requires additional settings that should be added to a credentials.yml file using the following template:

# security group IDs to apply to the VM
athens_security_groups: [sg-0123456abcdefgh]

# VPC subnet to deploy Athens to
athens_subnet_id: subnet-0123456789abcdefgh

# a specific, elastic IP address for the VM
external_ip: 3.123.200.100

The credentials need to be added to the deploy command, i.e.

-o manifests/operations/aws-ops.yml

VirtualBox

The fastest way to install Athens using BOSH is probably a Director VM running on VirtualBox which is sufficient for development or testing purposes. If you follow the bosh-lite installation guide, no further preparation is required to deploy Athens.

Deployment

A deployment manifest contains all the information for managing and updating a BOSH deployment. To aid in the deployment of Athens on BOSH, the athens-bosh-release repository provides manifests for basic deployment configurations inside the manifests directory. For quickly creating a standalone Athens server, clone the release repository and cd into it:

git clone --recursive https://github.com/s4heid/athens-bosh-release.git
cd athens-bosh-release

Once the infrastructure has been prepared and the BOSH Director is running, make sure that a stemcell has been uploaded. If this has not been done yet, choose a stemcell from the stemcells section of bosh.io, and upload it via the command line. Additionally, a cloud config is required for IaaS specific configuration used by the Director and the Athens deployment. The manifests directory also contains an example cloud config, which can be uploaded to the Director via

bosh update-config --type=cloud --name=athens \
    --vars-file=credentials.yml manifests/cloud-config.yml

Execute the deploy command which can be extended with ops/vars files depending on which IaaS you will be deploying to.

bosh -d athens deploy manifests/athens.yml  # add extra arguments

For example, when using AWS the deploy command for an Athens Proxy with disk storage would look like

bosh -d athens deploy \
    -o manifests/operations/aws-ops.yml \
    -o manifests/operations/with-persistent-disk.yml \
    -v disk_size=1024 \
    --vars-file=credentials.yml manifests/athens.yml

This will deploy a single Athens instance in the athens deployment with a persistent disk of 1024MB. The IP address of that instance can be obtained with

bosh -d athens instances

which is useful for targeting Athens, e.g. with the GOPROXY variable. You can follow this quickstart guide for more information.

Managing private repos with .netrc files

Authenticate private repositories via .netrc

  1. Create a .netrc file that looks like the following:

    machine <ip or fqdn>

    login <username>

    password <user password>

  2. Tell Athens through an environment variable the location of that file

    ATHENS_NETRC_PATH=<location/to/.netrc>

  3. Athens will copy the file into the home directory and override whatever .netrc file is in home directory. Alternatively, if the host of the Athens server already has a .netrc file in the home directory, then authentication should work out of the box.

Authenticate Mercurial private repositories via .hgrc

  1. Create a .hgrc file with authentication data

  2. Tell Athens through an environment variable the location of that file

    ATHENS_HGRC_PATH=<location/to/.hgrc>

  3. Athens will copy the file into the home directory and override whatever .hgrc file is in home directory. Alternatively, if the host of the Athens server already has a .hgrc file in the home directory, then authentication should work out of the box.

Configuring Athens

Configuring Athens

Here we’ll cover how to configure the Athens application utilizing various configuration scenarios.

This section covers some of the more commonly used configuration variables, but there are more! If you want to see all the configuration variables you can set, we’ve documented them all in this configuration file.

Authentication

There are numerous version control systems available to us as developers. In this section we’ll outline how they can be used by supplying required credentials in various formats for the Athens project.

Storage

In Athens we support many storage options. In this section we’ll describe how they can be configured

Upstream proxy

In this section we’ll describe how the upstream proxy can be configured to fetch all modules from a Go Modules Repository such as GoCenter, The Go Module Mirror, or another Athens Server.

Proxying A Checksum DB

In this section we’ll describe how to proxy a Checksum DB as per https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md

Subsections of Configuring Athens

The download mode file

Athens accepts an HCL formatted file that has instructions for how it should behave when a module@version isn’t found in its storage. This functionality gives Athens the flexibility configure Athens to fit your organization’s needs. The most popular uses of this download file are:

  • Configure Athens to never download or serve a module or group of modules
  • Redirect to a different module proxy for a module or group of modules

This document will outline how to use this file - called the download mode file - to accomplish these tasks and more.

Please see the “Use cases” section below for more details on how to enable these behaviors and more.

Configuration

First, once you’ve created your download mode file, you tell Athens to use it by setting the DownloadMode configuration parameter in the config.toml file, or setting the ATHENS_DOWNLOAD_MODE environment variable. You can set this configuration value to one of two values to tell Athens to use your file:

  1. Set its value to file:$FILE_PATH, where $FILE_PATH is the path to the HCL file
  2. Set its value to custom:$BASE_64 where $BASE_64 is the base64 encoded HCL file

Instead of one of the above two values, you can set this configuration to sync, async, none, redirect, or async_redirect. If you do, the download mode will be set globally rather than for specific sub-groups of modules. See below for what each of these values mean.

Download mode keywords

If Athens receives a request for the module github.com/pkg/errors at version v0.8.1, and it doesn’t have that module and version in its storage, it will consult the download mode file for specific instructions on what action to take:

  1. sync: Synchronously download the module from VCS via go mod download, persist it to the Athens storage, and serve it back to the user immediately. Note that this is the default behavior.
  2. async: Return a 404 to the client, and asynchronously download and persist the module@version to storage.
  3. none: Return a 404 and do nothing.
  4. redirect: Redirect to an upstream proxy (such as proxy.golang.org) and do nothing after.
  5. async_redirect: Redirect to an upstream proxy (such as proxy.golang.org) and asynchronously download and persist the module@version to storage.

Athens expects these keywords to be used in conjunction with module patterns (github.com/pkg/*, for example). You combine the keyword and the pattern to specify behavior for a specific group of modules.

Athens uses the Go path.Match function to parse module patterns.

Below is an example download mode file.

downloadURL = "https://proxy.golang.org"

mode = "async_redirect"

download "github.com/gomods/*" {
    mode = "sync"
}

download "golang.org/x/*" {
    mode = "none"
}

download "github.com/pkg/*" {
    mode = "redirect"
    downloadURL = "https://gocenter.io"
}

The first two lines describe the default behavior for all modules. This behavior is overridden for select module groups below. In this case, the default behavior is:

  • Immediatley redirect all requests to https://proxy.golang.org
  • In the background, download the module from the version control system (VCS) and store it

The rest of the file contains download blocks. These override the default behavior for specific groups of modules.

The first block specifies that any module matching github.com/gomods/* (such as github.com/gomods/athens) will be downloaded from GitHub, stored, and then returned to the user.

The second block specifies that any module matching golang.org/x/* (such as golang.org/x/text) will always return a HTTP 404 response code. This behavior ensures that Athens will never store or serve any module names starting with golang.org/x.

If a user has their GOPROXY environment variable set with a comma separated list, their go command line tool will always try the option next in the list. For example, if a user has their GOPROXY environment variable set to https://athens.azurefd.net,direct, and then runs go get golang.org/x/text, they will still download golang.org/x/text to their machine. The module just won’t come from Athens.

The last block specifies that any module matching github.com/pkg/* (such as github.com/pkg/errors) will always redirect the go tool to https://gocenter.io. In this case, Athens will never persist the given module to its storage.

Use cases

The download mode file is versatile and allows you to configure Athens in a large variety of different ways. Below are some of the mode common.

Blocking certain modules

If you’re running Athens to serve a team of Go developers, it might be useful to ensure that the team doesn’t use a specific group or groups of modules (for example, because of licensing or security issues).

In this case, you would write this in your file:

download "bad/module/repo/*" {
    mode = "none"
}

Preventing storage overflow

If you are running Athens using a storage backend that has limited space, you may want to prevent Athens from storing certain groups of modules that take up a lot of space. To avoid exhausting Athens storage, while still ensuring that the users of your Athens server still get access to the modules you can’t store, you would use a redirect directive, as shown below:

download "very/large/*" {
    mode = "redirect"
    url = "https://reliable.proxy.com"
}

If you use the redirect mode, make sure that you specify a url value that points to a reliable proxy.

Authentication to private repositories

Authentication

SVN private repositories

  1. Subversion creates an authentication structure in

    ~/.subversion/auth/svn.simple/<hash>
    
  2. In order to properly create the authentication file for your SVN servers you will need to authenticate to them and let svn build out the proper hashed files.

    $ svn list http://<domain:port>/svn/<somerepo>
    Authentication realm: <http://<domain> Subversion Repository
    Username: test
    Password for 'test':
    
  3. Once we’ve properly authenticated we want to share the .subversion directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.

    Bash

    export ATHENS_STORAGE=~/athens-storage
    export ATHENS_SVN=~/.subversion
    mkdir -p $ATHENS_STORAGE
    docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
      -v $ATHENS_SVN:/root/.subversion \
      -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
      -e ATHENS_STORAGE_TYPE=disk \
      --name athens-proxy \
      --restart always \
      -p 3000:3000 \
      gomods/athens:latest

    PowerShell

    $env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
    $env:ATHENS_SVN = "$(Join-Path $pwd .subversion)"
    md -Path $env:ATHENS_STORAGE
    docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
      -v "$($env:ATHENS_SVN):/root/.subversion" `
      -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
      -e ATHENS_STORAGE_TYPE=disk `
      --name athens-proxy `
      --restart always `
      -p 3000:3000 `
      gomods/athens:latest

Bazaar(bzr) private repositories

  • Bazaar is not supported with the Dockerfile provided by Athens. but the instructions are valid for custom Athens build with bazaar.*
  1. Bazaaar config files are located in
  • Unix

    ~/.bazaar/
    
  • Windows

    C:\Documents and Settings\<username>\Application Data\Bazaar\2.0
    
  • You can check your location using

    bzr version
    
  1. There are 3 typical configuration files
  • bazaar.conf
    • default config options
  • locations.conf
    • branch specific overrides and/or settings
  • authentication.conf
    • credential information for remote servers
  1. Configuration file syntax
  • # this is a comment

  • [header] this denotes a section header

  • section options reside in a header section and contain an option name an equals sign and a value

    • EXAMPLE:

      [DEFAULT]
      email = John Doe <jdoe@isp.com>
      
  1. Authentication Configuration

    Allows one to specify credentials for remote servers. This can be used for all the supported transports and any part of bzr that requires authentication(smtp for example). The syntax obeys the same rules as the others except for the option policies which don’t apply.

    Example:

    [myprojects] scheme=ftp host=host.com user=joe password=secret

    Pet projects on hobby.net

    [hobby] host=r.hobby.net user=jim password=obvious1234

    Home server

    [home] scheme=https host=home.net user=joe password=lessobV10us

    [DEFAULT]

    Our local user is barbaz, on all remote sites we’re known as foobar

    user=foobar

    NOTE: when using sftp the scheme is ssh and a password isn’t supported you should use PPK

    [reference code] scheme=https host=dev.company.com path=/dev user=user1 password=pass1

    development branches on dev server

    [dev] scheme=ssh # bzr+ssh and sftp are availablehere host=dev.company.com path=/dev/integration user=user2

    #proxy [proxy] scheme=http host=proxy.company.com port=3128 user=proxyuser1 password=proxypass1

  2. Once we’ve properly setup our authentication we want to share the bazaar configuration directory with the Athens proxy server in order to reuse those credentials. Below we’re setting it as a volume on our proxy container.

    Bash

    export ATHENS_STORAGE=~/athens-storage
    export ATHENS_BZR=~/.bazaar
    mkdir -p $ATHENS_STORAGE
    docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
      -v $ATHENS_BZR:/root/.bazaar \
      -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
      -e ATHENS_STORAGE_TYPE=disk \
      --name athens-proxy \
      --restart always \
      -p 3000:3000 \
      gomods/athens:latest

    PowerShell

    $env:ATHENS_STORAGE = "$(Join-Path $pwd athens-storage)"
    $env:ATHENS_BZR = "$(Join-Path $pwd .bazaar)"
    md -Path $env:ATHENS_STORAGE
    docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
      -v "$($env:ATHENS_BZR):/root/.bazaar" `
      -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
      -e ATHENS_STORAGE_TYPE=disk `
      --name athens-proxy `
      --restart always `
      -p 3000:3000 `
      gomods/athens:latest

Atlassian Bitbucket and SSH-secured git VCS’s

This section was originally written to describe configuring the Athens git client to fetch specific Go imports over SSH instead of HTTP against an on-prem instance of Atlassian Bitbucket. With some adjustment it may point the way to configuring the Athens proxy for authenticated access to hosted Bitbucket and other SSH-secured VCS’s. If your developer workflow requires that you clone, push, and pull Git repositories over SSH and you want Athens to perform the same way, please read on.

As a developer at example.com, assume your application has a dependency described by this import which is hosted on Bitbucket

import "git.example.com/golibs/logo"

Further, assume that you would manually clone this import like this

$ git clone ssh://git@git.example.com:7999/golibs/logo.git

A go-get client, such as that called by Athens, would begin resolving this dependency by looking for a go-import meta tag in this output

$ curl -s https://git.example.com/golibs/logo?go-get=1
<?xml version="1.0"?>
<!DOCTYPE html>
<html lang="en">
   <head>
      <meta charset="utf-8">
         <meta name="go-import" content="git.example.com/golibs/logo git https://git.example.com/scm/golibs/logo.git"/>
         <body/>
      </meta>
   </head>
</html>

which says the content of the Go import actually resides at https://git.example.com/scm/golibs/logo.git. Comparing this URL to what we would normally use to clone this project over SSH (above) suggests this global Git config http to ssh rewrite rule

[url "ssh://git@git.example.com:7999"]
	insteadOf = https://git.example.com/scm

So to fetch the git.example.com/golibs/logo dependency over SSH to populate its storage cache, Athens ultimately calls git, which, given that rewrite rule, in turn needs an SSH private key matching a public key bound to the cloning developer or service account on Bitbucket. This is essentially the github.com SSH model. At a bare minimum, we need to provide Athens with an SSH private key and the http to ssh git rewrite rule, mounted inside the Athens container for use by the root user

$ mkdir -p storage
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
    -v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
    -v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
    -v "$PWD/ssh-keys:/root/.ssh" \
    -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary

$PWD/gitconfig/.gitconfig contains the http to ssh rewrite rule

[url "ssh://git@git.example.com:7999"]
	insteadOf = https://git.example.com/scm

$PWD/ssh-keys contains the aforementioned private key and a minimal ssh-config

$ ls ssh-keys/
config		id_rsa

We also provide an ssh config to bypass host SSH key verification and to show how to bind different hosts to different SSH keys

$PWD/ssh-keys/config contains

Host git.example.com
Hostname git.example.com
StrictHostKeyChecking no
IdentityFile /root/.ssh/id_rsa

Now, builds executed through the Athens proxy should be able to clone the git.example.com/golibs/logo dependency over authenticated SSH.

SSH_AUTH_SOCK and ssh-agent Support

As an alternative to passwordless SSH keys, one can use an ssh-agent. The ssh-agent-set SSH_AUTH_SOCK environment variable will propagate to go mod download if it contains a path to a valid unix socket (after following symlinks).

As a result, if running with a working ssh agent (and a shell with SSH_AUTH_SOCK set), after setting up a gitconfig as mentioned in the previous section, one can run athens in docker as such:

$ mkdir -p storage
$ ssh-add .ssh/id_rsa_something
$ ATHENS_STORAGE=storage
$ docker run --rm -d \
    -v "$PWD/$ATHENS_STORAGE:/var/lib/athens" \
    -v "$PWD/gitconfig/.gitconfig:/root/.gitconfig" \
    -v "${SSH_AUTH_SOCK}:/.ssh_agent_sock" \
    -e "SSH_AUTH_SOCK=/.ssh_agent_sock" \
    -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens -e ATHENS_STORAGE_TYPE=disk --name athens-proxy -p 3000:3000 gomods/athens:canary

GitHub Apps

Instead of using a Machine User on GitHub, it is possible to create a GitHub App and authenticate via it.

Create a GitHub App in Settings > Developer settings > GitHub Apps and install it. The AppID/ClientID, Installation ID and Private Key are required from the App.

Install the GitHub App Git Credential Helper in your $PATH. The Athens Docker image comes with this pre-installed.

Configure your global Git config as follows:

[credential "https://github.com/your-org"]
    helper = "github-app -username <app-name> -appId <app-id> -privateKeyFile <path-to-private-key> -installationId <installation-id>"
    useHttpPath = true

[credential "https://github.com"]
    helper = "cache --timeout=3600"

[url "https://github.com"]
    insteadOf = ssh://git@github.com

This instructs Git to authenticate with the GitHub App and cache the results for 3600s (the authentication token is valid for 1 hour).

Now, builds executed through the Athens proxy should be able to clone the github.com/your-org/your-repo dependency over GitHub Apps.

GitHub Enterprise Self-hosted

To authenticate against a self-hosted GitHub Enterprise, the instructions are the same for GitHub hosted Apps with the exception for the Git config, which should include your domain, as follows:

[credential "https://github.example.com/your-org"]
    helper = "github-app -username <app-name> -appId <app-id> -privateKeyFile <path-to-private-key> -installationId <installation-id> -domain github.example.com"
    useHttpPath = true

[credential "https://github.example.com"]
    helper = "cache --timeout=3600"

[url "https://github.example.com"]
    insteadOf = ssh://git@github.com

Configuring Storage

Storage

The Athens proxy supports many storage types:

All of them can be configured using config.toml file. You need to set a valid driver in StorageType value or you can set it in environment variable ATHENS_STORAGE_TYPE on your server. Also for most of the drivers you need to provide additional configuration data which will be described below.

Memory

This storage doesn’t need any specific configuration and it’s also used by default in the Athens project. It writes all of data into local disk into tmp dir.

This storage type should only be used for development purposes!

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "memory"

Disk

Disk storage allows modules to be stored on a file system. The location on disk where modules will be stored can be configured.

You can pre-fill disk-based storage to enable Athens deployments that have no access to the internet. See here for instructions on how to do that.

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "disk"

[Storage]
    [Storage.Disk]
        RootPath = "/path/on/disk"

where /path/on/disk is your desired location. Also it can be set using ATHENS_DISK_STORAGE_ROOT env

Mongo

This driver uses a Mongo server as data storage. On start this driver will create an athens database and module collection on your Mongo server.

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "mongo"

[Storage]
    [Storage.Mongo]
        # Full URL for mongo storage
        # Env override: ATHENS_MONGO_STORAGE_URL
        URL = "mongodb://127.0.0.1:27017"

        # Not required parameter
        # Path to certificate to use for the mongo connection
        # Env override: ATHENS_MONGO_CERT_PATH
        CertPath = "/path/to/cert/file"

        # Not required parameter
        # Allows for insecure SSL / http connections to mongo storage
        # Should be used for testing or development only
        # Env override: ATHENS_MONGO_INSECURE
        Insecure = false

        # Not required parameter
        # Allows for use of custom database 
        # Env override: ATHENS_MONGO_DEFAULT_DATABASE
        DefaultDBName = athens

        # Not required parameter
        # Allows for use of custom collection 
        # Env override: ATHENS_MONGO_DEFAULT_COLLECTION
        DefaultCollectionName = modules

Google Cloud Storage

This driver uses Google Cloud Storage and assumes that you already have an account and bucket in it. If you never used Google Cloud Storage there is quick guide how to create bucket inside it.

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "gcp"

[Storage]
    [Storage.GCP]
        # ProjectID to use for GCP Storage
        # Env overide: GOOGLE_CLOUD_PROJECT
        ProjectID = "YOUR_GCP_PROJECT_ID"

        # Bucket to use for GCP Storage
        # Env override: ATHENS_STORAGE_GCP_BUCKET
        Bucket = "YOUR_GCP_BUCKET"

AWS S3

This driver is using the AWS S3 and assumes that you already have account and bucket created in it. If you never used Amazon Web Services there is quick guide how to create bucket inside it. After this you can pass your credentials inside config.toml file. If the access key ID and secret access key are not specified in config.toml, the driver will attempt to load credentials for the default profile from the AWS CLI configuration file created during installation.

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "s3"

[Storage]
        [Storage.S3]
        ### The authentication model is as below for S3 in the following order
        ### If AWS_CREDENTIALS_ENDPOINT is specified and it returns valid results, then it is used
        ### If config variables are specified and they are valid, then they return valid results, then it is used
        ### Otherwise, it will default to default configurations which is as follows
        # attempt to find credentials in the environment, in the shared
        # configuration (~/.aws/credentials) and from ec2 instance role
        # credentials. See
        # https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
        # and
        # https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
        # for environment variables that will affect the aws configuration.
        # Setting UseDefaultConfiguration would only use default configuration. It will be deprecated in future releases 
        # and is recommended not to use it.

        # Region for S3 storage
        # Env override: AWS_REGION
        Region = "MY_AWS_REGION"

        # Access Key for S3 storage
        # Env override: AWS_ACCESS_KEY_ID
        Key = "MY_AWS_ACCESS_KEY_ID"

        # Secret Key for S3 storage
        # Env override: AWS_SECRET_ACCESS_KEY
        Secret = "MY_AWS_SECRET_ACCESS_KEY"

        # Session Token for S3 storage
        # Not required parameter
        # Env override: AWS_SESSION_TOKEN
        Token = ""

        # S3 Bucket to use for storage
        # Env override: ATHENS_S3_BUCKET_NAME
        Bucket = "MY_S3_BUCKET_NAME"
        
        # If true then path style url for s3 endpoint will be used
        # Env override: AWS_FORCE_PATH_STYLE
        ForcePathStyle = false

        # If true then the default aws configuration will be used. This will
        # attempt to find credentials in the environment, in the shared
        # configuration (~/.aws/credentials) and from ec2 instance role
        # credentials. See
        # https://godoc.org/github.com/aws/aws-sdk-go#hdr-Configuring_Credentials
        # and
        # https://godoc.org/github.com/aws/aws-sdk-go/aws/session#hdr-Environment_Variables
        # for environment variables that will affect the aws configuration.
        # Env override: AWS_USE_DEFAULT_CONFIGURATION
        UseDefaultConfiguration = false

        # https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/
        # Note that this the URI should not end with / when AwsContainerCredentialsRelativeURI is set
        # Env override: AWS_CREDENTIALS_ENDPOINT
        CredentialsEndpoint = ""

        # Env override: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
        # If you are planning to use AWS Fargate, please use http://169.254.170.2 for CredentialsEndpoint
        # Ref: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v2.html
        AwsContainerCredentialsRelativeURI = ""

        # An optional endpoint URL (hostname only or fully qualified URI)
        # that overrides the default generated endpoint for S3 storage client.
        #
        # You must still provide a `Region` value when specifying an endpoint.
        # Env override: AWS_ENDPOINT
        Endpoint = ""

Minio

Minio is an open source object storage server that provides an interface for S3 compatible block storages. If you have never used minio, you can read this quick start guide. Any S3 compatible object storage is supported by Athens through the minio interface. Below, you can find different configuration options we provide for Minio. Example configuration for Digital Ocean and Alibaba OSS block storages are provided below.

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"

[Storage]
    [Storage.Minio]
        # Endpoint for Minio storage
        # Env override: ATHENS_MINIO_ENDPOINT
        Endpoint = "127.0.0.1:9001"

        # Access Key for Minio storage
        # Env override: ATHENS_MINIO_ACCESS_KEY_ID
        Key = "YOUR_MINIO_SECRET_KEY"

        # Secret Key for Minio storage
        # Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
        Secret = "YOUR_MINIO_SECRET_KEY"

        # Enable SSL for Minio connections
        # Defaults to true
        # Env override: ATHENS_MINIO_USE_SSL
        EnableSSL = false

        # Minio Bucket to use for storage
        # Env override: ATHENS_MINIO_BUCKET_NAME
        Bucket = "gomods"

DigitalOcean Spaces

For Athens to communicate with DigitalOcean Spaces, we are using Minio driver because DO Spaces tries to be fully compatible with it. Also configuration for this storage looks almost the same in our proxy as for Minio.

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"

[Storage]
    [Storage.Minio]
        # Address of DO Spaces storage
        # Env override: ATHENS_MINIO_ENDPOINT
        Endpoint = "YOUR_ADDRESS.digitaloceanspaces.com"

        # Access Key for DO Spaces storage
        # Env override: ATHENS_MINIO_ACCESS_KEY_ID
        Key = "YOUR_DO_SPACE_KEY_ID"

        # Secret Key for DO Spaces storage
        # Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
        Secret = "YOUR_DO_SPACE_SECRET_KEY"

        # Enable SSL
        # Env override: ATHENS_MINIO_USE_SSL
        EnableSSL = true

        # Space name in your DO Spaces storage
        # Env override: ATHENS_MINIO_BUCKET_NAME
        Bucket = "YOUR_DO_SPACE_NAME"

        # Region for DO Spaces storage
        # Env override: ATHENS_MINIO_REGION
        Region = "YOUR_DO_SPACE_REGION"

Alibaba OSS

For Athens to communicate with Alibaba Cloud Object Storage Service, we are using Minio driver. Also configuration for this storage looks almost the same in our proxy as for Minio.

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "minio"

[Storage]
    [Storage.Minio]
        # Address of Alibaba OSS storage
        # Env override: ATHENS_MINIO_ENDPOINT
        Endpoint = "YOUR_ADDRESS.aliyuncs.com"

        # Access Key for Minio storage
        # Env override: ATHENS_MINIO_ACCESS_KEY_ID
        Key = "YOUR_OSS_KEY_ID"

        # Secret Key for Alibaba OSS storage
        # Env override: ATHENS_MINIO_SECRET_ACCESS_KEY
        Secret = "YOUR_OSS_SECRET_KEY"

        # Enable SSL
        # Env override: ATHENS_MINIO_USE_SSL
        EnableSSL = true

        # Parent folder in your Alibaba OSS storage
        # Env override: ATHENS_MINIO_BUCKET_NAME
        Bucket = "YOUR_OSS_FOLDER_PREFIX"

Azure Blob Storage

This driver uses Azure Blob Storage

If you never used Azure Blog Storage, here is a quickstart

It assumes that you already have the following:

Configuration:
# StorageType sets the type of storage backend the proxy will use.
# Env override: ATHENS_STORAGE_TYPE
StorageType = "azureblob"

[Storage]
    [Storage.AzureBlob]
        # Storage Account name for Azure Blob
        # Env override: ATHENS_AZURE_ACCOUNT_NAME
        AccountName = "MY_AZURE_BLOB_ACCOUNT_NAME"

        # Account Key to use with the storage account
        # Env override: ATHENS_AZURE_ACCOUNT_KEY
        AccountKey = "MY_AZURE_BLOB_ACCOUNT_KEY"

        # Managed Identity Resource Id to use with the storage account
        # Env override: ATHENS_AZURE_MANAGED_IDENTITY_RESOURCE_ID
        ManagedIdentityResourceId = "MY_AZURE_MANAGED_IDENTITY_RESOURCE_ID"

        # Storage Resource to use with the storage account
        # Env override: ATHENS_AZURE_STORAGE_RESOURCE
        StorageResource = "MY_AZURE_STORAGE_RESOURCE"

        # Name of container in the blob storage
        # Env override: ATHENS_AZURE_CONTAINER_NAME
        ContainerName = "MY_AZURE_BLOB_CONTAINER_NAME"

External Storage

External storage lets Athens connect to your own implementation of a storage backend. All you have to do is implement the storage.Backend interface and run it behind an http server.

Once you implement the backend server, you must then configure Athens to use that storage backend as such:

Configuration:
# Env override: ATHENS_STORAGE_TYPE
StorageType = "external"

[Storage]
    [Storage.External]
        # Env override: ATHENS_EXTERNAL_STORAGE_URL
        URL = "http://localhost:9090"

Athens provides a convenience wrapper that lets you implement a storage backend with ease. See the following example:

package main

import (
    "github.com/gomods/athens/pkg/storage"
    "github.com/gomods/athens/pkg/storage/external"
)

// TODO: implement storage.Backend
type myCustomStorage struct {
    storage.Backend
}

func main() {
    handler := external.NewServer(&myCustomStorage{})
    http.ListenAndServe(":9090", handler)
}

Running multiple Athens pointed at the same storage

Athens has the ability to run concurrently pointed at the same storage medium, using a distributed locking mechanism called “single flight”.

By default, Athens is configured to use the memory single flight, which stores locks in local memory. This works when running a single Athens instance, given the process has access to it’s own memory. However, when running multiple Athens instances pointed at the same storage, a distributed locking mechansism is required.

Athens supports several distributed locking mechanisms:

  • etcd
  • redis
  • redis-sentinel
  • gcp (available when using the gcp storage type)
  • azureblob (available when using the azureblob storage type)

Setting the SingleFlightType (or ATHENS_SINGLE_FLIGHT TYPE in the environment) configuration value will enable usage of one of the above mechanisms. The azureblob and gcp types require no extra configuration.

Using etcd as the single flight mechanism

Using the etcd mechanism is very simple, just a comma separated list of etcd endpoints. The recommend configuration is 3 endpoints, however, more can be used.

SingleFlightType = "etcd"

[SingleFlight]
    [SingleFlight.Etcd]
        # Env override: ATHENS_ETCD_ENDPOINTS
        Endpoints = "localhost:2379,localhost:22379,localhost:32379"

Using redis as the single flight mechanism

Athens supports two mechanisms of communicating with redis: direct connection, and connecting via redis sentinels.

Direct connection to redis

Using a direct connection to redis is simple, and only requires a single redis-server. You can also optionally specify a password to connect to the redis server with

SingleFlightType = "redis"

[SingleFlight]
    [SingleFlight.Redis]
        # Endpoint is the redis endpoint for the single flight mechanism
        # Env override: ATHENS_REDIS_ENDPOINT
        Endpoint = "127.0.0.1:6379"

        # Password is the password for the redis instance
        # Env override: ATHENS_REDIS_PASSWORD
        Password = ""

Connecting to Redis via a redis url is also supported:

SingleFlightType = "redis"

[SingleFlight]
    [SingleFlight.Redis]
        # Endpoint is the redis endpoint for the single flight mechanism
        # Env override: ATHENS_REDIS_ENDPOINT
        # Note, if TLS is required use rediss:// instead.
        Endpoint = "redis://user:password@127.0.0.1:6379:6379/0?protocol=3"

If the redis url is invalid or cannot be parsed, Athens will fall back to treating Endpoint as if it were a normal host:port pair. If a password is supplied in the redis url, in addition to being provided in the Password configuration option, the two values must match otherwise Athens will fail to start.

Customizing lock configurations:

If you would like to customize the distributed lock options then you can optionally override the default lock config to better suit your use-case:

[SingleFlight.Redis]
    ...
    [SingleFlight.Redis.LockConfig]
        # TTL for the lock in seconds. Defaults to 900 seconds (15 minutes).
        # Env override: ATHENS_REDIS_LOCK_TTL
        TTL = 900
        # Timeout for acquiring the lock in seconds. Defaults to 15 seconds.
        # Env override: ATHENS_REDIS_LOCK_TIMEOUT
        Timeout = 15
        # Max retries while acquiring the lock. Defaults to 10.
        # Env override: ATHENS_REDIS_LOCK_MAX_RETRIES
        MaxRetries = 10

Customizations may be required in some cases for eg, you can set a higher TTL if it usually takes longer than 5 mins to fetch the modules in your case.

Connecting to redis via redis sentinel

NOTE: redis-sentinel requires a working knowledge of redis and is not recommended for everyone.

redis sentinel is a high-availability set up for redis, it provides automated monitoring, replication, failover and configuration of multiple redis servers in a leader-follower setup. It is more complex than running a single redis server and requires multiple disperate instances of redis running distributed across nodes.

For more details on redis-sentinel, check out the documentation

As redis-sentinel is a more complex set up of redis, it requires more configuration than standard redis.

Required configuration:

  • Endpoints is a list of redis-sentinel endpoints to connect to, typically 3, but more can be used
  • MasterName is the named master instance, as configured in the redis-sentinel configuration

Optionally, like redis, you can also specify a password to connect to the redis-sentinel endpoints with

SingleFlightType = "redis-sentinel"

[SingleFlight]
  [SingleFlight.RedisSentinel]
      # Endpoints is the redis sentinel endpoints to discover a redis
      # master for a SingleFlight lock.
      # Env override: ATHENS_REDIS_SENTINEL_ENDPOINTS
      Endpoints = ["127.0.0.1:26379"]
      # MasterName is the redis sentinel master name to use to discover
      # the master for a SingleFlight lock
      MasterName = "redis-1"
      # SentinelPassword is an optional password for authenticating with
      # redis sentinel
      SentinelPassword = "sekret"

Distributed lock options can be customised for redis sentinal as well, in a similar manner as described above for redis.

Using GCP as a singleflight mechanism

The GCP singleflight mechanism does not required configuration, and works out of the box. It has a single option with which it can be customized:

[SingleFlight.GCP]
    # Threshold for how long to wait in seconds for an in-progress GCP upload to
    # be considered to have failed to unlock.
    StaleThreshold = 120

Proxying a checksum database API

If you run go get github.com/mycompany/secret-repo@v1.0.0 and that module version is not yet in your go.sum file, Go will by default send a request to https://sum.golang.org/lookup/github.com/mycompany/secret-repo@v1.0.0. That request will fail because the Go tool requires a checksum, but sum.golang.org doesn’t have access to your private code.

The result is that (1) your build will fail, and (2) your private module names have been sent over the internet to an opaque public server that you don’t control.

You can read more about this sum.golang.org service here

Proxying a checksum DB

Many companies use Athens to host their private code, but Athens is not only a module proxy. It’s also a checksum database proxy. That means that anyone inside of your company can configure go to send these checksum requests to Athens instead of the public sum.golang.org server.

If the Athens server is configured with checksum filters, then you can prevent these problems.

If you run the below command using Go 1.13 or later:

$ GOPROXY=<athens-url> go build .

… then the Go tool will automatically send all checksum requests to <athens-url>/sumdb/sum.golang.org instead of https://sum.golang.org.

By default, when Athens receives a /sumdb/... request, it automatically proxies it to https://sum.golang.org, even if it’s a private module that sum.golang.org doesn’t and can’t know about. So if you are working with private modules, you’ll want to change the default behavior.

If you want Athens to not send some module names up to the global checksum database, set those module names in the NoSumPatterns value in config.toml or using the ATHENS_GONOSUM_PATTERNS environment variable.

The following sections will go into more detail on how checksum databases work, how Athens fits in, and how this all impacts your workflow.

How to set this all up

Before you begin, you’ll need to run Athens with configuration values that tell it to not proxy certain modules. If you’re using config.toml, use this configuration:

NoSumPatterns = ["github.com/mycompany/*", "github.com/secret/*"]

And if you’re using an environment variable, use this configuration:

$ export ATHENS_GONOSUM_PATTERNS="github.com/mycompany/*,github.com/secret/*"

You can use any string compatible with path.Match in these environment variables

After you start Athens up with this configuration, all checksum requests for modules that start with github.com/mycompany or github.com/secret will not be forwarded, and Athens will return an error to the go CLI tool.

This behavior will ensure that none of your private module names leak to the public internet, but your builds will still fail. To fix that problem, set another environment variable on your machine (that you run your go commands)

$ export GONOSUMDB="github.com/mycompany/*,github.com/secret/*"

Now, your builds will work and you won’t be sending information about your private codebase to the internet.

I’m confused, why is this hard?

When the Go tool has to download new code that isn’t currently in the project’s go.sum file, it tries its hardest to get a checksum from a server it trusts, and compare it to the checksum in the actual code it downloads. It does all of this to ensure provenance. That is, to ensure that the code you just downloaded wasn’t tampered with.

The trusted checksums are all stored in sum.golang.org, and that server is centrally controlled.

These build failures and potential privacy leaks can only happen when you try to get a module version that is not already in your go.sum file.

Athens does its best to respect and use the trusted checksums while also ensuring that your private names don’t get leaked to the public server. In some cases, it has to choose whether to fail your build or leak information, so it chooses to fail your build. That’s why everybody using that Athens server needs to set up their GONOSUMDB environment variable.

We believe that along with good documentation - which we hope this is! - we have struck the right balance between convenience and privacy.

Pre-filling disk storage

One of the popular features of Athens is that it can be run completely cut off from the internet. In this case, though, it can’t reach out to an upstream (e.g. a VCS or another module proxy) to fetch modules that it doesn’t have in storage. So, we need to manually fill up the disk partition that Athens uses with the dependencies that we need.

This document will guide you through packaging up a single module called github.com/my/module, and inserting it into the Athens disk storage.

First, get the tools

You’ll need to produce the following assets from module source code:

  • source.zip - just the Go source code, packaged in a zip file
  • go.mod - just the go.mod file from the module
  • $VERSION.info - metadata about the module

The source.zip file has a specific directory structure and the $VERSION.info has a JSON structure, both of which you’ll need to get right in order for Athens to serve up the right dependency formats that the Go toolchain will accept.

We don’t recommend that you create these assets yourself. Instead, use pacmod or gopack

Using pacmod

To install the pacmod tool, run go get like this:

$ go get github.com/plexsystems/pacmod@v0.4.0

This command will install the pacmod binary to your $GOPATH/bin/pacmod directory, so make sure that is in your $PATH.

Next, run pacmod to create assets

After you have pacmod, you’ll need the module source code that you want to package. Before you run the command, set the VERSION variable in your environment to the version of the module you want to generate assets for.

Below is an example for how to configure it.

$ export VERSION="v1.0.0"

Note: make sure your VERSION variable starts with a v

Next, navigate to the top-level directory of the module source code, and run pacmod like this:

$ pacmod pack github.com/my/module $VERSION .

Once this command is done, you’ll notice three new files in the same same directory you ran the command from:

  • go.mod
  • $VERSION.info
  • $VERSION.zip

Using gopack

To use this method you need docker-compose installed.

Fork gopack project and clone it to your local machine (or just download files to your computer)

Edit goget.sh with a list of go modules you want to download:

#!/bin/bash
go get github.com/my/module1;
go get github.com/my/module2;

Run

docker-compose up --abort-on-container-exit

Once this command is done, you’ll notice in the ATHENS_STORAGE folder all modules ready to be moved to your Athens disk storage.

Next, move assets into Athens storage directory

Now that you have assets built, you need to move them into the location of the Athens disk storage. In the below commands, we’ll assume $STORAGE_ROOT is the environment variable that points to the top-level directory that Athens uses for its on-disk.

If you set up Athens with the $ATHENS_DISK_STORAGE_ROOT environment variable, the root of this storage location is the value of this environment variable. Use export STORAGE_ROOT=$ATHENS_DISK_STORAGE_ROOT to prepare your environment for the below commands.

First create the subdirectory into which you’ll move the assets you created:

$ mkdir -p $STORAGE_ROOT/github.com/my/module/$VERSION

Finally, make sure that you’re still in the module source repository root directory (the same as you were in when you ran the pacmod command), and move your three new files into the new directory you just created:

$ mv go.mod $STORAGE_ROOT/github.com/my/module/$VERSION/go.mod
$ mv $VERSION.info $STORAGE_ROOT/github.com/my/module/$VERSION/$VERSION.info
$ mv $VERSION.zip $STORAGE_ROOT/github.com/my/module/$VERSION/source.zip

Note that we’ve changed the name of the .zip file

Finally, test your setup

At this point, your Athens server should have its disk-based cache filled with the github.com/my/module module at version $VERSION. Next time you request this module, Athens will find it in its disk storage and will not try to fetch it from an upstream source.

You can quickly test this behavior by running below curl command, assuming your Athens server is running on http://localhost:3000 and is already configured to use the same disk storage that you pre-filled above.

$ curl localhost:3000/github.com/my/module/@v/$VERSION.info

When you run this command, Athens should immediately return, without contacting any other network services.

Filtering modules (deprecated)

Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to filter modules in Athens.

The proxy supports the following three use cases

  1. Fetches a module directly from the source (upstream proxy)
  2. Exclude a particular module
  3. Include a module in the local proxy.

These settings can be done by creating a configuration file which can be pointed by setting either FilterFile in config.dev.toml or setting ATHENS_FILTER_FILE as an environment variable.

Writing the configuration file

Every line of the configuration can start either with a

  • + denoting that the module has to be included by the proxy
  • D denoting that the module has to be fetched directly from an upstream proxy and not stored locally
  • - denoting that the module is excluded and will not be fetched into the proxy or from the upstream proxy

It allows for # to add comments and new lines are skipped. Anything else would result in an error

Sample configuration file

# This is a comment


- github.com/azure
+ github.com/azure/azure-sdk-for-go

# get golang tools directly
D golang.org/x/tools

In the above example, golang.org/x/tools is fetched directly from the upstream proxy. All the modules from github.com/azure are excluded except github.com/azure/azure-sdk-for-go

Adding a default mode

The list of modules can grow quickly in size and sometimes may want to specify configuration for a handful of modules. In this case, they can set a default mode for all the modules and add specific rules to certain modules that they want to apply to. The default rule is specified at the beginning of the file. It can be an either +, - or D

An example default mode is

D
- github.com/manugupt1/athens
+ github.com/gomods/athens

In the above example, all the modules are fetched directly from the source. github.com/manugupt1/athens is excluded and github.com/gomods/athens is stored in the proxy storage.

Adding versions to the filter

Using an “approved list” is a common practice that requires each minor or patch version to be approved before it is allowed in the codebase. This is accomplished by adding a list of version patterns to the rule. These version patterns are comma-separated and prefix-matching, so v2 and v2.3.* both match the requested version 2.3.5.

An example version filter is

-
# use internal github enterprise server directly
D enterprise.github.com/company

# external dependency approved list
+ github.com/gomods/athens v0.1,v0.2,v0.4.1

In the above example, any module not in the rules will be excluded. All modules from enterprise.github.com/company are fetched directly from the source. The github.com/gomods/athens module will be stored in the proxy storage, but only for version v0.4.1 and any patch versions under v0.1 and v0.2 minor versions.

Versions Filter Modifiers

Athens provides advanced filter modifiers to cover cases such as API compatibility or when a given dependency changes its license from a given versions. The modifiers are intended to be used in the pattern list of the filter file.

-
# external dependency approved list
+ github.com/gomods/athens 

The currently supported modifiers are

  • ~1.2.3 will enable all patch versions from 1.2.3 and above (e.g. 1.2.3, 1.2.4, 1.2.5)

    • Formally, 1.2.x where x >= 3
  • ^1.2.3 will enable all patch and minor versions from 1.2.3 and above (e.g. 1.2.4, 1.3.0 and 1.4.5)

    • Formally, 1.x.y where x >= 2 and y >= 3
  • <1.2.3 will enable all versions lower than 1.2.3 (e.g. 1.2.2, 1.0.0 and 0.58.9)

    • Formally, x.y.z where x <= 1, y < = 2 and z < 3

This kind of modifiers will work only if a three parts semantic version is specified. For example, ~4.5.6 will work while ~4.5 won’t.

Using an upstream Go modules repository (deprecated)

Note: the filter file that this page documents is deprecated. Please instead see “Filtering with the download mode file” for updated instructions on how to set upstream repositories in Athens.

By default, Athens fetches module code from an upstream version control system (VCS) like github.com, but this can be configured to use a Go modules repository like GoCenter or another Athens Server.

  1. Create a filter file (e.g /usr/local/lib/FilterForGoCenter) with letter D (stands for “direct access”) in first line. For more details, please refer to documentation on - Filtering Modules

    # FilterFile for fetching modules directly from upstream
    D
  2. If you are not using a config file, create a new config file (based on the sample config.dev.toml) and edit values to match your environment). Additionally in the current or new config file, set the following parameters as suggested:

    FilterFile = "/usr/local/lib/FilterForGoCenter"
    GlobalEndpoint = "https://<url_to_upstream>"
    # To use GoCenter for example, replace <url_to_upstream> with gocenter.io
    # You can also use https://proxy.golang.org to use the Go Module mirror
  3. Restart Athens specifying the updated current or new config file.

     <path_to_athens>/proxy  -config_file <path-to updated  current or new configfile>
  4. Verify the new configuration using the steps mentioned in “Try out Athens” document, and go through the same walkthrough example.

Home template configuration

As of v0.14.0 Athens ships with a default, minimal HTML home page that advises users on how to connect to the proxy. It factors in whether GoNoSumPatterns is configured, and attempts to build configuration for GO_PROXY. It relies on the users request Host header (on HTTP 1.1) or the Authority header (on HTTP 2) as well as whether the request was over TLS to advise on configuring GO_PROXY. Lastly, the homepage provides a quick guide on how users can leverage the Athens API.

Of course, not all instructions will be this simple. Some installations may be reachable at different addresses in CI than for desktop users. In this case, and others where the default home page does not make sense it is possible to override the template.

Do so by configuring HomeTemplatePath via the config or ATHENS_HOME_TEMPLATE_PATH to a location on disk with a Go HTML template or placing a template file at /var/lib/athens/home.html.

Athens automatically injects the following variables in templates:

Setting Source
Host Built from the request Host (HTTP1) or Authority (HTTP2) header and presence of TLS. Includes ports.
NoSumPatterns Comes directly from the configuration.

Using these values is done by wrapping them in bracers with a dot prepended. Example: {{ .Host }}

For more advanced formatting read more about Go HTML templates.

<!DOCTYPE html>
<html>
<head>
	<title>Athens</title>
	<style>
		body {
			font-family: Arial, sans-serif;
			margin: 20px;
		}

		pre {
				background-color: #f4f4f4;
				padding: 5px;
				border-radius: 5px;
				width: fit-content;
  				padding: 10px;
		}


		code {
			background-color: #f4f4f4;
			padding: 5px;
			border-radius: 5px;
		}

	</style>
</head>
<body>
	
	<h1>Welcome to Athens</h1>

	<h2>Configuring your client</h2>
	<pre>GOPROXY={{ .Host }},direct</pre>
	{{ if .NoSumPatterns }}
	<h3>Excluding checksum database</h3>
	<p>Use the following GONOSUM environment variable to exclude checksum database:</p>
	<pre>GONOSUM={{ .NoSumPatterns }}</pre>
	{{ end }}

	<h2>How to use the Athens API</h2>
	<p>Use the <a href="/catalog">catalog</a> endpoint to get a list of all modules in the proxy</p>

	<h3>List of versions</h3>
	<p>This endpoint returns a list of versions that Athens knows about for <code>acidburn/htp</code>:</p>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/list</pre>

	<h3>Version info</h3>
	<p>This endpoint returns information about a specific version of a module:</p>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.info</pre>
	<p>This returns JSON with information about v1.0.0. It looks like this:
	<pre>{
	"Name": "v1.0.0",
	"Short": "v1.0.0",
	"Version": "v1.0.0",
	"Time": "1972-07-18T12:34:56Z"
}</pre>

	<h3>go.mod file</h3>
	<p>This endpoint returns the go.mod file for a specific version of a module:</p>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.mod</pre>
	<p>This returns the go.mod file for version v1.0.0. If {{ .Host }}/github.com/acidburn/htp version v1.0.0 has no dependencies, the response body would look like this:</p>
	<pre>module github.com/acidburn/htp</pre>

	<h3>Module sources</h3>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@v/v1.0.0.zip</pre>
	<p>This is what it sounds like — it sends back a zip file with the source code for the module in version v1.0.0.</p>

	<h3>Latest</h3>
	<pre>GET {{ .Host }}/github.com/acidburn/htp/@latest</pre>
	<p>This endpoint returns the latest version of the module. If the version does not exist it should retrieve the hash of latest commit.</p>

</body>
</html>

Logging

Athens is designed to support a myriad of logging scenarios.

Standard

The standard structured logger can be configured in plain or json formatting via LogFormat or ATHENS_LOG_FORMAT. Additionally, verbosity can be controlled by setting LogLevel or ATHENS_LOG_LEVEL. In order for the standard structured logger to work, CloudRuntime and ATHENS_CLOUD_RUNTIME should not be set to a valid value.

The logging is via Logrus, so the allowed values for logging config options are determined by that project. For example, ATHENS_LOG_LEVEL can be debug, info, warn/warning, error, etc.

Runtimes

Athens can be configured according to certain cloud provider specific runtimes. The GCP runtime configures Athens to rename certain logging fields that could be dropped or overriden when running in a GCP logging environment. This runtime can be used with LogLevel or ATHENS_LOG_LEVEL to control the verbosity of logs.

The Design of Athens

This section of the documentation details the design of Athens. You can read the code and ask plenty of questions (which we’re always happy to answer!), but we want to take some time here to give you a head start by describing how Athens is designed in words and diagrams, rather than code.

What You’ll Find Here

We’ve split this section into two major sections:

  1. Proxy internals - basics of the Athens proxy architecture and major features
  2. Communication flow - how the Athens proxy interacts with the outside world to fetch and store code, respond to user requests, and so on

How to Read this Section

We’ve designed the documentation in this section as a reference, which contrasts some of the other sections. That means that you don’t need to read everything here from front to back to get value from it. We anticipate that you’ll get the most out of this section while you’re contributing to the project.

Subsections of The Design of Athens

Proxy

The Athens Proxy

The Athens proxy has two primary use cases:

  • Internal deployments
  • Public mirror deployments

This document details features of the Athens proxy that you can use to achieve either use case.

The Role of the Athens proxy

We intend proxies to be deployed primarily inside of enterprises to:

  • Host private modules
  • Exclude access to public modules
  • Store public modules

Importantly, a proxy is not intended to be a complete mirror of an upstream proxy. For public modules, its role is to store the modules locally and provide access control.

What happens when a public module is not stored?

When a user requests a module MxV1 from a proxy and the Athens proxy doesn’t have MxV1 in its store, it first determines whether MxV1 is private or not private.

If it’s private, it immediately stores the module into the proxy storage from the internal VCS.

If it’s not private, the Athens proxy consults its exclude list for non-private modules (see below). If MxV1 is on the exclude list, the Athens proxy returns 404 and does nothing else. If MxV1 is not on the exclude list, the Athens proxy executes the following algorithm:

upstreamDetails := lookUpstream(MxV1)
if upstreamDetails == nil {
	return 404 // if the upstream doesn't have the thing, just bail out
}
return upstreamDetails.baseURL

The important part of this algorithm is lookUpstream. That function queries an endpoint on the upstream proxy that either:

  • Returns 404 if it doesn’t have MxV1 in its storage
  • Returns the base URL for MxV1 if it has MxV1 in its storage

In a later version of the project, we may implement an event stream on proxies that any other proxy can subscribe to and listen for deletions/deprecations on modules that it cares about

Exclude Lists and Private Module Filters

To accommodate private (i.e. enterprise) deployments, the Athens proxy maintains two important access control mechanisms:

  • Private module filters
  • Exclude lists for public modules

Private Module Filters

Private module filters are string globs that tell the Athens proxy what is a private module. For example, the string github.internal.com/** tells the Athens proxy:

  • To never make requests to the public internet (i.e. to upstream proxies) regarding this module
  • To download module code (in its store mechanism) from the VCS at github.internal.com

Exclude Lists for Public Modules

Exclude lists for public modules are also globs that tell the Athens proxy what modules it should never download from any upstream proxy. For example, the string github.com/arschles/** tells the Athens proxy to always return 404 Not Found to clients.

Catalog Endpoint

The proxy provides a /catalog service endpoint to fetch all the modules and their versions contained in the local storage. The endpoint accepts a continuation token and a page size parameter in order to provide paginated results.

A query is of the form

https://proxyurl/catalog?token=foo&pagesize=47

Where token is an optional continuation token and pagesize is the desired size of the returned page. The token parameter is not required for the first call and it’s needed for handling paginated results.

The result is a json with the following structure:

{"modules": [{"module":"github.com/athens-artifacts/no-tags","version":"v1.0.0"}],
 "next":""}'

If a next token is not returned, then it means that no more pages are available. The default page size is 1000.

Communication

Communication flow

This is the story of a time long ago. A time of myth and legend when the ancient Gods were petty and cruel and they plagued build with irreproducibility. Only one project dared to challenge their power…Athens. Athens possessed a strength the world had never seen. Filling its storage.

Clean plate

At the beginning, there’s theoretical state when a storage of the Athens proxy is empty.

When User makes a request at this ancient time, it works as described on the flow below.

  • User runs go get to acquire new module.
  • Go CLI contacts the Athens proxy asking for module M, version v1.0
  • The Athens proxy checks whether or not it has this module in its storage. It does not.
  • The Athens proxy downloads code from the underlying VCS and converts it into the Go Module format.
  • After it receives all the bits, it stores it into its own storage and serves it to the User.
  • User receives module and is happy.

The process from the user using go get all the way to the user downloading a module is synchronous.

Communication flow for clear state Communication flow for clear state

Happy path

Now that the Athens proxy is aware of module M at version v1.0, it can serve that module immediately to the user, without fetching it from the VCS.

Communication flow for new proxy Communication flow for new proxy

From VCS to the User

You read about proxy, communication and then opened a codebase and thought to yourself: This is not as simple as described in the docs.

Athens has a set of architectural components that handle Go modules from their journey from the VCS into storage and down to the user. If you feel lost on how all these pieces work, read on my friend!

From communication, you know that when a module is not backed up in the storage it gets downloaded from VCS (such as github.com) and then it is served to the user. You also know that this whole process is synchronous. But when you read a code you see module fetchers and download protocol stashers and you struggle to figure out what’s what and how they differ. It might seem complicated, but this document will help explain everything that’s going on.

Components

Let’s start with describing all the components you will see along the way. There’s no better way to get a clear picture of everything than with a diagram.

Architecture chart of components Architecture chart of components

As you can see, there are a lot of layers and wrappers. The first two pieces you’ll see in the code are the Storage and the Fetcher. We’ll start our journey there.

Storage

Storage is what it sounds like. Storage instance created in proxy/storage.go’s GetStorage function. Based on storage type passed as an ENV variable it will create in-memory, filesystem, mongo… storage. This is where modules live. Once there, always there.

Fetcher

Fetcher is the first component on our way. As we can guess from the name, Fetcher (pkg/module/fetcher.go) is responsible for fetching the sources from VCS. For this, it needs two things: the go binary and afero.FileSystem. Path to binary and filesystems are passed to Fetcher during initialization.

mf, err := module.NewGoGetFetcher(goBin, fs)
if err != nil {
    return err
}

app_proxy.go

When a request for a new module comes, the Fetch function is invoked.

Fetch(ctx context.Context, mod, ver string) (*storage.Version, error)

fetch function

Then the Fetcher:

  • creates a temp directory using an injected FileSystem
  • in this temp dir, it constructs dummy go project consisting of simple main.go and go.mod so the go CLI can be used.
  • invokes go mod download -json {module}

This command downloads the module into the storage. Once the download is completed,

  • the Fetch function reads the module bits from storage and returns them to the caller.
  • The exact path of module files is returned by go mod as part of JSON response.

Stash

As it is important for us to keep components small and readable, we did not want to bloat Fetcher with storing functionality. For storing modules into a storage we use Stasher. This is the single responsibility of a simple Stasher.

We think it’s important to keep components small and orthogonal, so the Fetcher and the storage.Backend don’t interact. Instead, the Stasher composes them together and orchestrates the process of fetching code and then storing it.

The New method accepts the Fetcher and the storage.Backend with a set of wrappers (explained later).

New(f module.Fetcher, s storage.Backend, wrappers ...Wrapper) Stasher

stasher.go

The code in pkg/stash/stasher.go isn’t complex, but it’s important:

I think this does two things:

  • invokes Fetcher to get module bits
  • stores the bits using a storage

If you read carefully you noticed wrappers passed into a basic Stasher implementation. These wrappers add more advanced logic and help to keep components clean.

The new method then returns a Stasher which is a result of wrapping basic Stasher with wrappers.

for _, w := range wrappers {
    st = w(st)
}

stasher.go

Stash wrapper - Pool

As downloading a module is resource heavy (memory) operation, Pool (pkg/stash/with_pool.go) helps us to control simultaneous downloads.

It uses N-worker pattern which spins up the specified number of workers which then waits for a job to complete. Once they complete their job, they return the result and are ready for the next one.

A job, in this case, is a call to Stash function on a backing Stasher.

Stash wrapper - SingleFlight

We know that module fetching is a resource-heavy operation and we just put a limit on a number of parallel downloads. To help us save more resources we wanted to avoid processing the same module multiple times.

SingleFlight wrapper (pkg/stash/with_singleflight.go) takes care of that. Internally it keeps track of currently running downloads using a map. If a job arrives and map[moduleVersion] is empty, it initiates it with a callback channel and invokes a job on a backing Stasher.

s.subs[mv] = []chan error{subCh}
go s.process(ctx, mod, ver)

if there is an entry for the requested module, SingleFlight will subscribe for a result

s.subs[mv] = append(s.subs[mv], subCh)

and once the job is complete, the module is served one level up to the download protocol (or to wrapping stasher possibly)

Download protocol

The outer most level is the download protocol.

dpOpts := &download.Opts{
    Storage: s,
    Stasher: st,
    Lister: lister,
}
dp := download.New(dpOpts, addons.WithPool(protocolWorkers))

It contains two components we already mentioned: Storage, Stasher and one more additional: Lister.

Lister is used in List and Latest funcs to look upstream for the list of versions available.

Storage is here again. We saw it in a Stasher before, used for saving. In Download protocol it is used to check whether or not the module is already present. If it is, it is served directly from storage.

Otherwise, Download protocol uses Stasher to download module, store it into a storage and then it serves it back to the user.

You can also see addons.WithPool in a code snippet above. This addon is something similar to Stash wrapper - Pool. It controls the number of concurrent requests proxy can handle.

Contributing to Athens

Welcome, Gopher! We’re really glad you’re considering contributing to Athens. We’d like to briefly introduce you to our community before you get started.

We have some hard-and-fast rules in our community, like our Code of Conduct, but instead of making rules pre-emptively, we try to keep in mind a shared philosophy to help us all make decisions and make new rules when we need to.

Our Philosophy Document

The philosophy section of these docs details our philosophy, and if you get involved with our project we encourage you to read it. If you’re just browsing for now, here’s a brief summary:

  • Be Nice to Each Other - people > code all the time, every time. We think that if we focus on our relationships with each other, we’ll end up with better technology, better code, and a better community.
  • Make Development & Testing Easy - Well, as easy as possible. This helps us reduce cognitive load so we can focus on Athens, not setting up our dev/test environment.
  • Focus on the Commmunity - “If you want to go fast, go alone. If you want to go far, go together.” We all try to follow this proverb. We try to get as many people involved as we can, we do (almost) everything 100% transparently, and we trust each other to do the right thing.

Where to Go from Here

We hope you like what you see, and we’d love for you to get involved.

If you’re familiar with Git and GitHub basics, read how to participate in the Athens community to learn about how we organize ourselves and how to get started.

If you’re new to Git and GitHub, check out our guide for new contributors to open source, and then go on to that how to participate document I mentioned above.

Finally, if you are a new maintainer of the project, we have some documentation written for you at ./maintainers.

Subsections of Contributing to Athens

Chapter 1

The Athens Community

Welcome, Athenian! We’ve put together this section to help you get involved with the Athens community.

Before we go further, we want you to know two things:

  1. You can contribute in many ways, including documentation, testing, writing code, reporting bugs, reviewing code, technical writing, and more
  2. Absolutely everybody is welcome. You are welcome in our community, regardless of your level of programming experience, number of years writing Go, race, religion, sexual orientation, gender identity, or anything else. If you want to be here, we will do everything we can to help you feel welcome and involved

Ready to join us? Head over to the guide on how to participate to get started!

Subsections of The Athens Community

Participating in the Community

Absolutely everyone is welcome to join our community at any time! We are a friendly and inclusive group and we’d love to have you. We have three roles in the Athens community:

  • Community member
  • Contributor
  • Maintainer

Read on to find out more!

Community Members

Community members are folks who decide they want to get involved with our community. Absolutely anyone can do that whenever they want. If you want to get involved, that doesn’t mean you have to commit to being involved, but we hope our community is welcoming and the work is interesting enough to convince you to stay :)

We’ll provide all the support we can possibly provide to help you contribute in any way you’d like. If you’re considering joining us, here are some ideas for how you can get involved:

  • Comment on an issue that you’re interested in
  • Submit a pull request (PR) to fix an issue, or to improve something that doesn’t have an issue
  • Review a PR that you’re interested in
  • Join us at office hours (or more than one!)
    • See here for recordings of all our past meetings
  • Come chat with us in the gophers slack in the #athens channel
  • … and anything else that’s appropriate for you!

Contributors

As you participate in the community more and more, you’ll have the opportunity to become a contributor. Here’s what being a contributor means, and what you should do to become one.

What Being a Contributor Means

Contributors have read access to the Athens repository on Github. This means that as a contributor, you’re able to have issues assigned to you and you’ll be requested to review pull requests (PRs) via the Github pull request review system.

We rely heavily on the Github PR review system, which means that if you review a PR as a contributor, you can help decide when that PR is ready to be merged. Don’t worry that you don’t know enough, the final approval and merge will be by one or more maintainers.

How to Become a Contributor

To become a contributor, the core maintainers of the project would like to see you:

  • Attend our development meetings regularly1
  • Comment on issues with your experiences and opinions
  • Add your comments and reviews on pull requests (anyone can do this as a community member)
  • Contribute PRs to fix issues
  • Open issues as you find them

Contributors and maintainers will do their best to watch for community members who may make good contributors. But don’t be shy, if you feel that this is you, please reach out to one or more of the contributors or maintainers.

Maintainers

After you become a contributor, you’ll have the opportunity to become a maintainer. If you’re happy with your contributions but want more of a role in steering the project then read more on being a maintainer.

The End

The above descriptions lay out roughly what each role is and how you can move into each of them. Folks all have different strengths, live in different places, and so on. We’re a diverse group, and we want to keep it that way!

So, everything in this document is a guideline, not a hard-and-fast rule. If you are really good at something, or can’t do something else, talk to one of the maintainers and let us know what’s up. We will accommodate everyone the best we can.


1 Athens development meetings are during the day in US Pacific Time. We know that this time can be problematic for some folks due to work commitments, different time zones, and so on. If you can't come to meetings, that's totally ok and doesn't mean you can't become a contributor! Just let one of the maintainers know about it, or leave a message in #athens in the gophers slack.

2 Anyone and everyone is of course welcome to do this too!

Our Philosophy

This document lays out generally how we want to work with each other. It’s hard to make a rule or set a guideline for each and every situation that might come up in our community. That’s basically predicting the future!

We do of course set some boundaries like the code of conduct, but we want to fall back to this document for guidance when we encounter a new situation or question that we need to address.

Guiding Principles

This is the TL;DR of the whole document! The Athens project has a few guiding principles:

  • Be nice to each other
  • Make development & testing easy
  • Focus on the Community
  • Ask Questions

In the rest of this document, we’re going to go into detail for each of these items.

Be Nice to Each Other

We firmly believe that a nice, welcoming and constructive community comes first, and code and technology second. If the folks in the community around this project aren’t nice to each other, it doesn’t matter how cool our technology is.

  • Let’s try to make newcomers feel welcome
  • Let’s put our debates in Github issues, and be civil and constructive in them
  • Let’s be empathetic
  • Let’s listen more than we talk
  • We’re in different time zones, so let’s respect that
  • Let’s encourage each other to learn new aspects of our project

Most importantly, let’s be inclusive. Not everyone will share your point of view, communication style, and many other things. Try to consider their point of view and treat them with respect.

Make Development & Testing Easy

Cognitive load is bad when you’re writing code, so let’s try to minimize it.

  • Getting started should be a one-liner
  • Let’s make the hard things easier
  • Docs are good, let’s keep them up to date
  • Let’s make it pleasant to work with our code

Focus on the Community

There’s an African proverb that goes like this:

If you want to go fast, go alone. If you want to go far, go together.

We want to apply that wisdom to our community.

  • It’s easy and sometimes tempting to do everything yourself
  • But if we want to keep the project growing, it’s hard to have just a few people doing everything
  • Related: the bus factor
  • So let’s focus on bringing new people into the community, and getting them started (see: Be Nice to Each Other, above)

Ask Questions

Questions are a great way for us to share ideas and make the project and community better.

  • If you don’t know something, try not to be afraid to ask
  • If you think your question is stupid, ask it anyway
  • … And if you’re still uncomfortable asking in public, ask a maintainer in private
  • If someone asks you a question, it’s ok to answer it later
  • Put answers on paper where everyone can read them, if you can
  • … And FAQs are great to have - let’s do them!
  • Newcomers have the best perspectives, so listen well to their questions

Office Hours

Athens developers get together for an hour or so approximately every week on a video chat to learn and discuss a part of the codebase. The hour is relatively unstructured, but generally it starts out with a core maintainer (usually @arschles) going into the details of a part of the codebase. From there, we just go where the questions take us.

This is a great opportunity for new community members to get more involved with the project in a low-pressure setting. It’s also a great setting for existing community members to gain a deeper understanding into the codebase.

We meet at this zoom video chat URL: https://arschles.com/zoom

Absolutely everybody is welcome to attend these meetings. You’re free to suggest topics to talk about, share your perspective, and participate as little or much as you’d like.

You don’t have to be a contributor to attend.

Logistics

All office hours happen using the zoom software. Before you come to the meeting, please make sure you have installed the zoom client using this link: https://zoom.us/support/download.

If you’d like to attend an office hours, please join the #athens channel in the Gophers Slack Group. An announcement will be made on Tuesday mornings to remind folks before the office hours start. You can also follow @arschles and/or @gomods on Twitter to get these announcements.

Usually, office hours are at 2pm (14:00) US Pacific time on Tuesdays.

The Zoom Software

Also, if you are using a Mac, there is a vulnerability that can allow arbitrary websites to activate your camera without your permission. Please see here for more information.

If you haven’t updated your Zoom software since July 10, 2019, please do so to fix this issue.

Triaging Pull Requests

Hi, Gopher! We’re glad you’re interested in getting into PR triaging. This page details how to do that. Let’s get started!

TL;DR

We’re trying to all work together to make sure all of our pull requests (PRs) get reviewed and merged efficiently. So, we set up an easy way for anyone to “triage” pull requests on any Monday, Wednesday or Friday.

PR triaging means looking at older PRs and do either or both of these things, as appropriate:

  • Prompting reviewers to come back and re-review
  • Prompting submitters to come back and address reviews

Absolutely anyone can do triaging, and this is a great way to get involved with the community.

Sign up for triaging here.

Intro

The Athens community all works together to keep up to date on issues and pull requests. For issues, we take some time each week to review issues in the next milestone and others that folks are interested in.

We try to keep PR reviews moving a little bit faster and more efficiently, so we look at those 3 times a week.

PR reviews are asynchronous:

  • A PR gets submitted
  • You leave feedback in your review
  • The submitter reads and addresses it (e.g. change code or respond to the comment) your feedback sometime later
  • You come back and re-review sometime after that

I personally love the asynchronous workflow, but life happens - people forget, people get busy, go on vacation, etc… - as they should! We’re all human and we need breaks like that.

The problem is that PR reviews can get stalled. So, it’s important to make sure that a PR doesn’t sit idle for too long.

We’re getting a person to come check in three times a week to make sure that older PRs are still getting attention.

Triage Schedule

Since we don’t have a super huge volume of PRs, we’re looking for folks to do the following on a triage day:

  • Look at PRs not updated in the last 3 days
  • Add a comment to prompt reviewers and the submitter to come back to the PR:
    • If new commits have been added to the PR since 1 or more reviewer has done a review, please prompt those reviewers to come back and re-review
    • If there are still comments pending and the submitter hasn’t addressed them, please prompt the submitter to look at the new comments
  • If you see a PR that hasn’t been updated in more than 10 days, write this in the PR comments and we’ll come figure out what’s going on (probably contact someone directly or close the PR): @gomods/maintainers this PR is really old!

If you need to prompt someone in your triage, do it by mentioning someone on GitHub like this: @arschles can you look at this again?. If you notice that someone has been @mentioned already, you can try pinging them on Slack. If you ping them, be nice and remember that they might be busy with other things though :)

How do I Sign Up?

Anyone, regardless of background, experience, familiarity with the project, time zone, or pretty much anything else. This is a wonderful way to get involved with the project. Triages generally take 15 minutes or less if you’ve done a few before (see the bottom of this section if you haven’t).

If you’d like to do triaging on a particular day, please add your name to the triaging spreadsheet.

If you haven’t done a triage before and would like to get started, please submit an issue.

If any of this doesn’t make sense, please contact us in the #athens channel in the Gophers Slack and we’ll clear it up and get you started.

Can this be Automated?

Probably, yes! But we don’t know if there are exact criteria on when PRs should be “prompted” and how a bot should do that. Maybe we’ll learn those criteria here.

Even still, it’s nice to have a human touch as a submitter and reviewer. It matches our philosophy very well.

Chapter 2

New Contributors

Welcome, New Contributors!

This section is all about helping you get started with open source and Athens.

Let’s get started with learning how to use git

Subsections of New Contributors

Using Git

What is git?

Git is a free and open source distributed version control system. What does that really mean? It is a way to track changes to files on your computer. This is like keeping a detailed log of every time you change a file, what lines and characters were changed. So you could look at the log and see what changed, or undo those changes if you wanted or if you are working with others you can merge your changes together.

It’s a lot to take in, so don’t worry if you aren’t following yet.

If you want a more detailed walk through than we provide here, have a look at the Git Book.

Installing

Let’s start by getting git installed on you machine. You can check if it’s already installed by running git --version from the command line.

Follow your operating system specific instructions in Chapter 1.5 of the Git Book.

Basic concepts

Repository
This is a file structure on disk, like a database, it contains all files and the log of changes.
Staging
When you make changes inside a repository, they are untracked. You decide which changes to track, as you add changes they are added to the staging area. This let's you see all current changes before committing them.
Commit
After you are happy with the changes tracked in staging, you can commit them to the log we mentioned. You have a few options for writing a message that will be stored with the commit in the log, more on that later.
Branch
When you are in the repository the default out of the box is usually called `master`. This is the main branch of the repository (NOTE: if you are creating a new repository of your own, please change the default branch to `main`. `master` is an inappropriate and offensive), the main branch of the repository. Typically you will want to do your work on a new branch for each feature or bug. This allows you to see and work on different versions of the same code in one repository.
Checkout
To check out a branch, is to switch to view that branches version of the files in the repository.
Merge
When you want to incorporate another branch, `main` or someone else's feature for example, into your current branch you will merge the changes. This will apply the other changes on top of yours.
Remote
This is just a repository, that is accessible remotely. You can use the git command to push and pull changes to.
Push
Pushing to a remote will synchronize your locally committed changes to the remote.
Pull
Pulling from a remote will both fetch and merge the changes on the remote with the branch you have currently checked out.
Fetch
When you want to get some remote branch or changes, but not merge them yet, you can fetch them. Just ask the remote for the data and store it locally but not incorporate it into anything. You could then checkout the feature branch and run the code, or read over the changes.

Try it out

There is a great interactive tutorial, for free, available at Code Academy. Take some time to play with it and try out some of the commands.

Docs

Contributing To Docs

Contributing to docs is just as important, if not more important than, writing code. We use GoHugo to run this website. So if you’d like to improve it, here’s how you can run it locally:

  1. Install the Hugo binary: https://github.com/gohugoio/hugo#choose-how-to-install
  2. cd ./docs && hugo server

The Hugo server will run on http://localhost:1313 and it will automatically watch your files and update the website every time you make a change.

Alternatively you can run our custom docker image as outlined here

Github

Using GitHub

We use GitHub to host the remote copy of our repository and both track our issues and manage our pull requests. If you haven’t signed up before, take a minute to go do that now. We’ll be right here when you get back.

Issues

On GitHub we use the concept of an issue to record every change needed in our code base. This means ideas, bugs, features, support, even just discussions.

Never hesitate to open an issue if you have question about the code or think you may have found a bug. Sometimes people will find part of the documentation or instructions difficult to follow, let us know if this happens.

In particular for our project on GitHub, we have a template for Bug Reports and Feature Requests with one for Proposals .. proposed. When you click on ‘New Issue’ you will be given a choice of which template to start with, if they do not seem to fit your need there is an option to ‘Open a regular issue’ - which is blank.

GitHub Issue Tracker GitHub Issue Tracker GitHub Issue Template Selection GitHub Issue Template Selection

Forks

In order to keep things a bit tidy in our main repository, we all do work on what is called a fork before creating a pull request. A fork is essentially a copy of a repository that is owned by your GitHub account. You can create as many branches as you like there and don’t be shy with creating commits. We will squash them all into one before we merge, more on that below.

But first take a minute to read this awesome post on creating and maintaining a fork of the project. It even goes into adding other collaborators forks as remotes which you will find useful as you start working more and more with other contributors.

Pull requests

Brian Ketelsen created an awesome video on making your first open source pull request, for an overview of the process go watch that now. There’s also a great video series on contributing to an Open Source project by Kent C. Dodds.

When we receive a new pull request, it may take time for a maintainer or other contributor to get to it, but when we do a few things will happen.

You can expect at least a few comments, don’t let them discourage you though. We are all trying to help one another write the best code and documentation possible, all criticism should be considered constructive.

If you feel like it is not, please do not hesitate to reach out to one of the maintainers. We take our code of conduct very seriously.

Most likely one or more contributors and maintainers will leave a review on your pull request. You can discuss the changes requested in the pull request itself, or if you need help with something in particular you can reach out to us in the #athens channel on Gophers Slack.

After all requested changes are resolved a maintainer will give it a final look and as long as our continuous integration passed they will merge it with the main branch.

Project process

Let’s just go over a quick general guideline on contributing to Athens.

Find an issue

When you see an issue you would like to work on, if no one else has expressed interest please comment on the issue with something like:

  • I will take this
  • I would like to work on this

This let’s other contributors know someone is working on it, we all know free time is hard to get and don’t want anyone wasting theirs.

If you don’t see an issue for your bug or feature, please open one to discuss with the community. Some things may not be in the best interest of the project, or may have already been discussed.

If your issue is something very small, like a typo or broken link, you may skip straight the pull request.

Open a pull request

After you have created a branch on your fork, and made the changes. Please make sure all tests still pass, see DEVELOPMENT.md for details. Then after you push all changes up to your fork, head over to Athens to open a pull request. Usually, right after you have pushed a new branch and you visit the original repository, GitHub will prompt you to open a new pull request. Otherwise you can do so from the Pull Requests tab on the repository.

How To Contribute

Development Guide for Athens

The proxy is written in idiomatic Go and uses standard tools. If you know Go, you’ll be able to read the code and run the server.

Athens uses Go Modules for dependency management. You will need Go v1.12+ to get started on Athens.

See our Contributing Guide for tips on how to submit a pull request when you are ready.

Go version

Athens is developed on Go v1.12+.

To point Athens to a different version of Go set the following environment variable

GO_BINARY_PATH=go1.12.X
# or whichever binary you want to use with athens

Run the Proxy

If you’re inside GOPATH, make sure GO111MODULE=on, if you’re outside GOPATH, then Go Modules are on by default. The main package is inside cmd/proxy and is run like any go project as follows:

cd cmd/proxy
go build
./proxy

After the server starts, you’ll see some console output like:

Starting application at 127.0.0.1:3000

Dependencies

Services that Athens Needs

Athens relies on several services (i.e. databases, etc…) to function properly. We use Docker images to configure and run those services. However, Athens does not require any storage dependencies by default. The default storage is in memory, you can opt-in to using the fs which would also require no dependencies. But if you’d like to test out Athens against a real storage backend (such as MongoDB, Minio, S3 etc), continue reading this section:

If you’re not familiar with Docker, that’s ok. We’ve tried to make it easy to get up and running:

  1. Download and install docker-compose (docker-compose is a tool for easily starting and stopping lots of services at once)
  2. Run make dev from the root of this repository

That’s it! After the make dev command is done, everything will be up and running and you can move on to the next step.

If you want to stop everything at any time, run make down.

Note that make dev only runs the minimum amount of dependencies needed for things to work. If you’d like to run all the possible dependencies run make alldeps or directly the services available in the docker-compose.yml file. Keep in mind, though, that make alldeps does not start up Athens or Oympus, but only their dependencies.

Run unit tests

In order to run unit tests, services they depend on must be running first:

make alldeps

then you can run the unit tests:

make test-unit

Run the docs

To get started with developing the docs we provide a docker image, which runs Hugo to render the docs. Using the docker image, we mount the /docs directory into the container. To get it up and running, from the project root run:

make docs
docker run -it --rm \
        --name hugo-server \
        -p 1313:1313 \
        -v ${PWD}/docs:/src:cached \
        gomods/hugo

Then open http://localhost:1313.

Linting

In our CI/CD pass, we use govet, so feel free to run it locally beforehand:

go vet ./...

Maintainers

There are a lot of ways to contribute to the Athens project and being a maintainer is only one of those paths. Maintainers on the Athens Project are expected to devote roughly five hours a week to the development and maintenance of the project. Their responsibilities include:

  • Help organize our development meetings (i.e. help organize the agenda)
  • Promote the project and build community (e.g. present on it where possible, write about it, …) when possible2
  • Triage issues (e.g. adding labels, promoting discussions, finalizing decisions)
  • Organize and promote PR reviews (e.g. prompting community members, contributors, and other maintainers to review)
  • Help foster a safe and welcoming environment for all project participants. This will include enforcing our code of conduct. We adhere to the Contributor Covenant, if you haven’t read it yet you can do so here (english version).

We’d love to have your partnership in building Athens, but it is a statement to the commitment that a project of Athens’ size and complexity demands. If that appeals to you join us on Slack.

Benefits of being a maintainer

  • Free access to GitHub CoPilot
  • A rewarding sense of accomplishment
  • Fun, complex work in a safe and collaborative environment
  • Access to very smart people

Contributing without maintaining

If maintaining Athens isn’t something you can do, we’re always looking for people to:

  • Develop features
  • Patch bugs
  • Triage issues
  • Provide support

You don’t need to do anything other than visit our GitHub page to do all of the above!

Good First Issues

The following is intended for project maintainers when triaging issues and deciding to label them as a good first issue. It is meant to serve as a guide on good practices only as each issue is different. This designation will be at the discretion of a maintainer.

In this template we will assume this may be the contributor’s first pull request in The Athens Project.

NOTE: Although this is written with maintainers in mind, anyone writing an issue with the goal of helping first time contributors is welcome to use this template as their guide.


Generally we will still want to try and follow the issue template for bugs or features. If you are performing issue triage you may need to add more information to fulfill the below template.

Template

New to Athens?

If you are new to the project, and you haven’t already, please take some time to read our contribution docs.

What needs to be done?

The issue should contain a detailed outline of the bug or feature.

This should include links to any relevant references both within and outside of the project. Other issues, pull requests or comments. It could also include relevant links to our docs or maybe blogs posts.

Try to include ideas of what good a starting place might be, or anything that has been experimented with already.

New To Open Source

If you are new to open source, using git or GitHub, or just want some workflow tips, head over to our new contributor guide.

FAQ

Is Athens Just a Proxy? A Registry?

TL;DR “Registry” doesn’t describe what Athens is trying to do here. That implies that there’s only one service in the world that can serve Go modules to everyone. Athens isn’t trying to be that. Instead, Athens is trying to be part of a federated group of module proxies.

A registry is generally run by one entity, is one logical server that provides authentication (and provenance sometimes), and is pretty much the de-facto only source of dependencies. Sometimes it’s run by a for-profit company.

That’s most definitely not what we in the Athens community are going for, and that would harm our community if we did go down that path.

First and foremost, Athens is an implementation of the Go Modules download API. Not only does the standard Go toolchain support any implementation of that API, the Athens proxy is designed to talk to any other server that implements that API as well. That allows Athens to talk to other proxies in the community.

Finally, we’re purposefully building this project - and working with the toolchain folks - in a way that everyone who wants to write a proxy can participate.

Does Athens integrate with the go toolchain?

Athens is currently supported by the Go v1.12+ toolchain via the download protocol.

For the TL;DR of the protocol, it’s a REST API that lets the go toolchain (i.e. go get) see lists of versions and fetch source code for a specific version.

Athens is a server that implements the protocol. Both it, the protocol and the toolchain (as you almost certainly know) is open source.

Are the packages served by Athens immutable?

TL;DR Athens does store code in CDNs and has the option to store code in other persistent datastores.

The longer version:

It’s virtually impossible to ensure immutable builds when source code comes from Github. We have been annoyed by that problem for a long time. The Go modules download protocol is a great opportunity to solve this issue. The Athens proxy works pretty simply at a high level:

  1. go get github.com/my/module@v1 happens
  2. Athens looks in its datastore, it’s missing
  3. Athens downloads github.com/my/module@v1 from Github (it uses go get on the backend too)
  4. Athens stores the module in its datastore
  5. Athens serves github.com/my/module@v1 from its datastore forever

To repeat, “datastore” means a CDN (we currently have support for Google Cloud Storage, Azure Blob Storage and AWS S3) or another datastore (we have support for MongoDB, disk and some others).

Can the Athens proxy authenticate to private repositories?

TL;DR: yes, with proper authentication configuration defined on the Athens proxy host.

When the GOPROXY environment variable is set on the client-side, the Go 1.11+ cli does not attempt to request the meta tags, via a request that looks like https://example.org/pkg/foo?go-get=1.

Internally Athens uses go get under the hood (go mod download to be exact) without the GOPROXY environment variable set so that go will in turn request the meta tags using the standard authentication mechanisms supported by go. Therefore, if go before v1.11 worked for you, then go 1.11+ with GOPROXY should work as well, provided that the Athens proxy host is configured with the proper authentication.

Can I exclude a module completely?

Yes, this is possible. The proxy provides a configuration file that will allow users to specify which modules that should not be fetched at all. The filtering modules configuration provides details about the configuration file and how to exclude certain modules.

Can I specify that a module is fetched from an upstream proxy and not stored locally?

Yes, this is possible. Refer to the filtering modules configuration provides details about the configuration file and how to exclude certain modules.

Is there support for monitoring and observability for Proxy?

Right now, we have structured logs for proxy. Along with that, we have added tracing to help developers identify critical code paths and debug latency issues. While there is no setup required for logs, tracing requires some installation. We currently support exporting traces with Jaeger, GCP Stackdriver & Datadog (untested). Further support for other exporters is in progress.

To try out tracing with Jaeger, do the following:

  • Set the environment to development (otherwise traces will be sampled)

  • Run docker-compose up -d that is found in the athens source root directory to initialize the services required

  • Run the walkthrough tutorial

  • Open http://localhost:16686/search

    Observability is not a hard requirement for the Athens proxy. So, if the infrastructure is not properly set up, it will fail with an information log. For example, if Jaeger is not running or if the wrong URL to the exporter is provided, the proxy will continue to run. However, it will not collect any traces or metrics while the exporter backend is unavailable.

What VCS servers does Athens support?

Athens uses go mod download under the hood, so it supports anything go mod suppports.

Which currently includes:

  • git
  • svn
  • hg
  • bzr
  • fossil

When should I use a vendor directory, and when should I use Athens?

The Go community has used vendor directories for a long time before module proxies like Athens came along, so naturally each group collaborating on code should decide for themselves whether they want to use a vendor directory, use Athens, or do both!

Using a vendor directory (without a proxy) is valuable when:

  • CI/CD systems don’t have access to an Athens (even if it’s internal)
  • When the vendor directory is so small that it is still faster to check it out from a repo than it is to pull zip files from the server
  • If you’re coming from glide/dep or another dependency management system that leveraged the vendor directory

Athens (without a vendor directory) is valuable when:

  • You have a new project
  • You are upgrading a Go project to use Go modules
  • Your team requires that you use Athens (i.e. for isolation or dependency auditing)
  • Your vendor directory is large and causing slow checkouts and downloading from Athens speeds the build up
    • For developers slow checkouts will not be as much of a problem as for ci tools which frequently need to checkout fresh copies of the project
  • You want to remove the vendor directory from your project to:
    • Reduce noise in pull requests
    • Reduce difficulty doing fuzzy file searching in your project

Try it out!

Try out Athens

To quickly see Athens in action, follow these steps:

First, make sure you have Go v1.12+ installed, that GOPATH/bin is on your path, and that you have enabled the Go Modules feature.

Bash

export GO111MODULE=on

PowerShell

$env:GO111MODULE = "on"

Next, use git and Go to install and run the Athens proxy in a background process.

$ git clone https://github.com/gomods/athens
$ cd athens/cmd/proxy
$ go install
$ proxy &
[1] 37186
INFO[0000] Exporter not specified. Traces won't be exported
INFO[0000] Starting application at http://127.0.0.1:3000

Next, you will need to configure Go to use the Athens proxy!

Bash

export GOPROXY=http://127.0.0.1:3000

PowerShell

$env:GOPROXY = "http://127.0.0.1:3000"

Now, when you build and run this example application, go will fetch dependencies via Athens!

$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ go run .
go: finding github.com/athens-artifacts/samplelib v1.0.0
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.info [200]
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.mod [200]
go: downloading github.com/athens-artifacts/samplelib v1.0.0
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.zip [200]
The 🦁 says rawr!

The output from go run . includes attempts to find the github.com/athens-artifacts/samplelib dependency. Since the proxy was run in the background, you should also see output from Athens indicating that it is handling requests for the dependency.

This should give you an overview of what using Athens is like!

Walkthrough

First, make sure you have Go v1.12+ installed and that GOPATH/bin is on your path.

Without the Athens proxy

Let’s review what everything looks like in Go without the Athens proxy in the picture:

Bash

$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ GO111MODULE=on go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!

PowerShell

$ git clone https://github.com/athens-artifacts/walkthrough.git
$ cd walkthrough
$ $env:GO111MODULE = "on"
$ go run .
go: downloading github.com/athens-artifacts/samplelib v1.0.0
The 🦁 says rawr!

The end result of running this command is that Go downloaded the package source and packaged it into a module, saving it in the Go Modules local storage.

Now that we have seen Go Modules in action without the Athens proxy, let’s take a look at how the Athens proxy changes the workflow and the output.

With the Athens proxy

Using the most simple installation possible, let’s walk through how to use the Athens proxy, and figure out what is happening at each step.

Before moving on, let’s clear our Go Modules local files so that we can see the Athens proxy in action without any modules locally populated:

Bash

sudo rm -fr $(go env GOPATH)/pkg/mod

PowerShell

rm -recurse -force "$(go env GOPATH)\pkg\mod"

Now run the Athens proxy in a background process:

Bash

$ mkdir -p $(go env GOPATH)/src/github.com/gomods
$ cd $(go env GOPATH)/src/github.com/gomods
$ git clone https://github.com/gomods/athens.git
$ cd athens
$ GO111MODULE=on go run ./cmd/proxy -config_file=./config.dev.toml &
[1] 25243
INFO[0000] Starting application at 127.0.0.1:3000

PowerShell

$ mkdir "$(go env GOPATH)\src\github.com\gomods"
$ cd "$(go env GOPATH)\src\github.com\gomods"
$ git clone https://github.com/gomods/athens.git
$ cd athens
$ $env:GO111MODULE = "on"
$ $env:GOPROXY = "https://proxy.golang.org"
$ Start-Process -NoNewWindow go 'run .\cmd\proxy -config_file=".\config.dev.toml"'
[1] 25243
INFO[0000] Starting application at 127.0.0.1:3000

The Athens proxy is now running in the background and is listening for requests from localhost (127.0.0.1) on port 3000.

Since we didn’t provide any specific configuration the Athens proxy is using in-memory storage, which is only suitable for trying out the Athens proxy for a short period of time, as you will quickly run out of memory and the storage doesn’t persist between restarts.

With Docker

For more details on running Athens in docker, take a look at the install documentation

In order to run the Athens Proxy using docker, we need first to create a directory that will store the persitant modules. In the example below, the new directory is named athens-storage and is located in our userspace (i.e. $HOME). Then we need to set the ATHENS_STORAGE_TYPE and ATHENS_DISK_STORAGE_ROOT environment variables when we run the Docker container.

Bash

export ATHENS_STORAGE=$HOME/athens-storage
mkdir -p $ATHENS_STORAGE
docker run -d -v $ATHENS_STORAGE:/var/lib/athens \
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens \
   -e ATHENS_STORAGE_TYPE=disk \
   --name athens-proxy \
   --restart always \
   -p 3000:3000 \
   gomods/athens:latest

PowerShell

$env:ATHENS_STORAGE = "$(Join-Path $HOME athens-storage)"
md -Path $env:ATHENS_STORAGE
docker run -d -v "$($env:ATHENS_STORAGE):/var/lib/athens" `
   -e ATHENS_DISK_STORAGE_ROOT=/var/lib/athens `
   -e ATHENS_STORAGE_TYPE=disk `
   --name athens-proxy `
   --restart always `
   -p 3000:3000 `
   gomods/athens:latest

Next, you will need to enable the Go Modules feature and configure Go to use the Athens proxy!

Using the Athens proxy

Bash

export GO111MODULE=on
export GOPROXY=http://127.0.0.1:3000

PowerShell

$env:GO111MODULE = "on"
$env:GOPROXY = "http://127.0.0.1:3000"

The GO111MODULE environment variable controls the Go Modules feature in Go 1.11 only. Possible values are:

  • on: Always use Go Modules
  • auto (default): Only use Go Modules when a go.mod file is present, or the go command is run from outside the GOPATH
  • off: Never use Go Modules

The GOPROXY environment variable tells the go binary that instead of talking to the version control system, such as github.com, directly when resolving your package dependencies, instead it should communicate with a proxy. The Athens proxy implements the Go Download Protocol, and is responsible for listing available versions for a package in addition to providing a zip of particular package versions.

Now, you when you build and run this example application, go will fetch dependencies via Athens!

$ cd ../walkthrough
$ go run .
go: finding github.com/athens-artifacts/samplelib v1.0.0
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.info [200]
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.mod [200]
go: downloading github.com/athens-artifacts/samplelib v1.0.0
handler: GET /github.com/athens-artifacts/samplelib/@v/v1.0.0.zip [200]
The 🦁 says rawr!

The output from go run . includes attempts to find the github.com/athens-artifacts/samplelib dependency. Since the proxy was run in the background, you should also see output from Athens indicating that it is handling requests for the dependency.

Let’s break down what is happening here:

  1. Before Go runs our code, it detects that our code depends on the github.com/athens-artifacts/samplelib package which is not present in the Go Modules local storage.

  2. At this point the Go Modules feature comes into play because we have it enabled. Instead of looking in the GOPATH for the package, Go reads our go.mod file and sees that we want a particular version of that package, v1.0.0.

    module github.com/athens-artifacts/walkthrough
    
    require github.com/athens-artifacts/samplelib v1.0.0
  3. Go first checks for github.com/athens-artifacts/samplelib@v1.0.0 in the Go Modules local storage, located in GOPATH/pkg/mod. If that version of the package is already local storage, then Go will use it and stop looking. But since this is our first time running this, our local storage is empty and Go keeps looking.

  4. Go requests github.com/athens-artifacts/samplelib@v1.0.0 from our proxy because it is set in the GOPROXY environment variable.

  5. The Athens proxy checks its own storage (in this case is in-memory) for the package and doesn’t find it. So it retrieves it from github.com and then saves it for subsequent requests.

  6. Go downloads the module zip and puts it in the Go Modules local storage GOPATH/pkg/mod.

  7. Go will use the module and build our application!

Subsequent calls to go run . will be much less verbose:

$ go run .
The 🦁 says rawr!

No additional output is printed because Go found github.com/athens-artifacts/samplelib@v1.0.0 in the Go Module local storage and did not need to request it from the Athens proxy.

Lastly, quitting from the Athens proxy. This cannot be done directly because we are starting the Athens proxy in the background, thus we must kill it by finding it’s process ID and killing it manually.

Bash

lsof -i @localhost:3000
kill -9 <<PID>>

PowerShell

netstat -ano | findstr :3000 (local host Port number)
taskkill /PID typeyourPIDhere /F

Next Steps

Now that you have seen Athens in Action:

Fork me on GitHub