Download Guide

Public HMPDACC files stored in Amazon S3 can be listed and downloaded without an AWS account or configured AWS credentials. Add --no-sign-request to each public-bucket command; you do not need to run aws configure for these downloads.

Some human participant metadata and host sequence data remain controlled-access. Those records require the appropriate dbGaP or repository authorization even when a public index or study description is available.

Start by choosing the exact S3 URI you need from the Data Resources catalog.

Install AWS CLI version 2

Install the current AWS CLI version 2 package for your operating system, then open a new terminal window before checking the installation.

Windows

  1. Download the official AWS CLI version 2 Windows MSI installer. A current-user MSI is also available when an all-users installation is not appropriate.
  2. Run the installer and complete its prompts.
  3. Open a new PowerShell or Command Prompt window from the Start menu.
  4. Run aws --version.

macOS

  1. Download and open the official AWS CLI version 2 macOS package, then follow the installer prompts.
  2. Alternatively, open Terminal and run the package commands below.
  3. Open a new Terminal window and run aws --version.
curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /

Linux

  1. Open a terminal.
  2. Download the x86_64 installer below. For ARM64 Linux, replace the filename and URL with awscli-exe-linux-aarch64.zip.
  3. Unzip the package and run the installer.
  4. Open a new terminal and run aws --version.
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Confirm the installation

aws --version

A successful response begins with an AWS CLI version string such as aws-cli/2.... When the shell cannot find aws, close and reopen the terminal, then see the troubleshooting section.

Preview a confirmed S3 path

List the immediate contents of a prefix without downloading them:

aws s3 ls s3://hmpdcc/hmp1/demo/PRJNA46327/ --no-sign-request

To preview every object below a prefix, add --recursive:

aws s3 ls s3://hmpdcc/ihmp/t2d/microbiome/wms/raw/ --recursive --no-sign-request

Review the listing and available disk space before downloading. Study prefixes can contain many large sequencing files.

Download a directory

S3 prefixes are not ordinary folders. Use --recursive to copy all objects below a selected prefix into a local directory:

aws s3 cp s3://hmpdcc/hmp1/demo/PRJNA46327/ <LOCAL_DIRECTORY> --recursive --no-sign-request
aws s3 cp
Runs the AWS CLI high-level copy command.
The s3:// source
Identifies the exact confirmed HMPDACC prefix selected from the Data Resources catalog.
<LOCAL_DIRECTORY>
Is the existing destination on your computer. Quote a destination path when it contains spaces.
--recursive
Copies every object below the prefix; it is required for an entire S3 prefix.
--no-sign-request
Uses anonymous public access instead of AWS credentials.
Inspect first: use one of the listing commands above before starting a recursive transfer.

Verify the download

Check that the destination contains downloaded files and review their reported sizes.

macOS or Linux

ls -lh <LOCAL_DIRECTORY>

Windows PowerShell

Get-ChildItem <LOCAL_DIRECTORY>

Troubleshooting

aws: command not found or “aws is not recognized”

Open a new terminal after installation. Re-run the AWS CLI installer when necessary and confirm that its installation directory is on your system PATH.

Unable to locate credentials

For this public bucket, the error usually means --no-sign-request was omitted. Add it to the command. Do not run aws configure solely for anonymous HMPDACC downloads.

AccessDenied

Confirm that --no-sign-request is present and that the object is intended for public release. Some human data require dbGaP or repository authorization and cannot be accessed anonymously.

NoSuchKey or an empty listing

Copy the S3 path exactly from the Data Resources catalog. S3 names are case-sensitive, and a missing character, release directory, or trailing prefix can identify a different location.

Insufficient disk space or very large files

Use aws s3 ls to inspect object sizes, choose a destination with enough free space, and avoid recursive downloads until the total scope is understood.

Interrupted download

Run the same aws s3 cp command again. For large collections, download smaller prefixes or individual files so failures are easier to recover from.

Corporate, institutional, proxy, or firewall restrictions

Ask your network administrator whether HTTPS access to Amazon S3 endpoints is allowed. Institutional proxy settings, TLS inspection, and outbound firewall rules can prevent otherwise valid public downloads.