Talking to AWS

Basics of using AWS from Go

AWS offers a large number of services, and each service exposes operations you can perform on its resources. For example, S3 is a service, and a bucket is one of its resources. Listing, creating, or deleting buckets are operations.

You can perform these operations through the AWS web console, the command-line line tools, or from your own programs. In every case, the AWS API is what actually performs the work under the hood. In this post, we’ll look at how to interact with that API directly from your own code.

We’ll build a small CLI tool that lists S3 buckets using the AWS SDK for Go v2. The same pattern applies to other services, resources and operations.

Working with the AWS SDK generally involves four steps:

1. Import the required packages

2. Load configuration

3. Get a service client

4. Call operations on the resources

Import the required packages

As usual when writing Go programs, we start by initializing the module:

$ mkdir talking-to-aws
$ cd talking-to-aws
$ go mod init talking-to-aws

Next, we import packages that we need. Some packages come from the standard library—context, fmt, and log. The others come from the AWS SDK: the config package for loading your AWS settings, and the service package for the specific service you want to use—s3 in our case:

// main.go
package main

import (
  "context"
  "fmt"
  "log"

  "github.com/aws/aws-sdk-go-v2/config"
  "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() { 
  // CODE BELOW GOES HERE 
}

Note: your editor may add (and remove) the imports automatically when you write and save your code, so you might not need to add them manually. However, you might still need to download the AWS SDK packages:

$ go mod tidy

Load configuration

Next, we need to load the AWS configuration. This step provides the SDK with your credentials and determines which region to use:

ctx := context.Background()

cfg, err := config.LoadDefaultConfig(ctx)
if err != nil {
  log.Fatal(err)
}

LoadDefaultConfig reads settings from environment variables as well as the standard AWS config and credentials files (~/.aws/config and ~/.aws/credentials on Unix-like systems). It’s the recommended way to initialize the AWS SDK because it automatically picks up whatever configuration the user has already set up.

Get a service client

Next, we create a client for the service API we want to use:

s3Client := s3.NewFromConfig(cfg)

In this example, we’re creating an S3 client, using the configuration we loaded in the previous step.

Call operations on the resources

Now we’re ready to perform the usual CRUD operations: create, read, update, or delete service resources. Let’s list the existing S3 buckets as an example:

result, err := s3Client.ListBuckets(
  ctx, 
  &s3.ListBucketsInput{},
)
if err != nil {
  log.Fatal(err)
}

for _, bucket := range result.Buckets {
  fmt.Printf("%s\n", *bucket.Name)
}

List resources page by page

However, if you read the ListBuckets documentation (for example by hovering over it in your editor), you’ll notice that the recommendation is to use paginated requests. The main reason is that many AWS APIs only return a limited number of results per call. If you have a lot of resources, an unpaginated request may not return all of them.

Here is the paginated version:

paginator := s3.NewListBucketsPaginator(
  s3Client,
  &s3.ListBucketsInput{},
)

for paginator.HasMorePages() {
  page, err := paginator.NextPage(ctx)
  if err != nil {
    log.Fatal(err)
  }

  for _, bucket := range page.Buckets {
    fmt.Printf("%s\n", *bucket.Name)
  }
}

Now, if you’ve been coding along, you’re ready to run your program:

$ go run main.go