So, I'm not sure if the commands are actually important, but for background this is the command I was running:
aws s3 ls s3://REDACTED/ | jq -nR '[inputs | split(" +"; null).[3]] | reverse.[] | "bkt --ttl '1h' -- aws s3 cp s3://REDACTED/\(.) - | tac"' -r | bash | head
Basically, I wanted to get the last 10 lines from all the lines of all the objects (by name in lexicographic order). To do this, I list the bucket, then reverse it, then construct some commands to cat them in reverse order, then run them, then limit the output with head. For those who don't know, bkt
is a wrapper that adds caching to any command...
When I ran that I was a bit surprised that it didn't quit after the first 10 lines, in-fact it seems like it is still running all the aws cp commands...
My question here is why didn't the bash
command in the pipe abort? it should have gotten a SIGPIPE
after head
exits right?
EDIT:This is a less complicated command that does the same thing:
seq 100000000000 | sed -E $'s/(.*)/sh -c \'echo \\1; sleep .1\'/' | sh | head
EDIT:Note: As @Fravadona brings up, please use caution when piping commands into a shell interpreter. This is not an example of a robust solution to any specific problem