I have a file of s3 paths, sources.lst. Contents:
["s3://bucket-name/path/to/file0.tif","s3://bucket-name/path/to/file1.tif",]I am attempting to iterate over these s3 paths & download the files. I am trying to using jq and a bash while loop. My download.sh:
#!/bin/bashset -o errexitwhile read -r sourcedo echo "Attempting to execute:" echo "aws s3 cp "${source}" ." aws s3 cp "${source}" .done < <(cat sources.lst | jq '.[]')Executing bash download.sh gives me:
Attempting to execute:aws s3 cp "s3://bucket-name/path/to/file0.tif" .usage: aws s3 cp <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>Error: Invalid argument typeWhich is strange, since if I copy the line causing the error, ie
aws s3 cp "s3://bucket-name/path/to/file0.tif" .and execute it in the terminal I run bash download.sh, the download succeeds.
How do I download the files specified in sources.lst using bash?
I am using Mac, bash --version: GNU bash, version 5.2.26(1)-release (aarch64-apple-darwin22.6.0), aws --version: aws-cli/2.13.1 Python/3.11.4 Darwin/23.4.0 exe/x86_64 prompt/off