I am a big fan of Cloudflare R2, an object storage that provides egress-free downloads.
R2 is compatible with the AWS S3 API, so you can use the AWS CLI tool – with a few caveats. These include:
- You need to add the
--endpoint-url https://<account_id>.r2.cloudflarestorage.com
argument for every call. - When copying to R2, you need to pass the
--checksum-algorithm CRC32
argument. - I often store multiple AWS configurations, which requires passing an additional argument:
--profile <your_r2_profile_name>
.
To perform these steps automatically, I wrote a small Bash function. Add this to your ~/.bashrc
or ~/.zshrc
:
function r2() {
if [ $# -eq 0 ]; then
echo "No arguments provided"
echo
echo "Usage: r2 <command> <arguments>"
return 1
fi
SUBCOMMAND=${1}
shift 1
ARGS=(${@})
LAST_ARG=${ARGS: -1}
if [[ ${SUBCOMMAND} == "cp" && "${LAST_ARG}" =~ "^s3://" ]]; then
CHECKSUM_ARG=("--checksum-algorithm" "CRC32")
else
CHECKSUM_ARG=""
fi
aws s3 $SUBCOMMAND \
--endpoint-url https://<account_id>.r2.cloudflarestorage.com \
--profile <your_r2_profile_name> \
${CHECKSUM_ARG} \
${ARGS[@]}
}
This snippet defines the r2
command, which can used as follows:
r2 ls s3://your_bucket_name/some/path
r2 cp your_local_file s3://your_bucket_name/some/path
r2 cp s3://your_bucket_name/some/path/some_file .