I have a problem generating the list of my resources in AWS, and later converting them into CSV using the JQ command, exactly this is the command I am running from the AWS CLI
aws configservice select-resource-config --expression "SELECT accountId, awsRegion, resourceType, configuration.instanceType, configuration.dBInstanceClass, arn, configuration.description, configuration.role, configuration.volumeType, configuration.size, configuration.platform, configuration.engineVersion, configuration.architecture, configuration.runtime, configuration.vpcId, configuration.subnetId, configuration.dNSName, configuration.publicIpAddress, configuration.publicDnsName, configuration.privateIpAddress, configuration.privateDnsName, configuration.securityGroups, configuration.status" --output json | \jq -r '["ID Cuenta", "Region AWS", "Tipo Recurso", "Tipo Instancia EC2", "Tipo DB RDS", "ARN", "Descipcion", "Rol", "Tipo Volumen", "Tamaño", "Plataforma", "Engine Version", "Arquitectura", "Runtime", "VPC", "Subnet ID", "DNS", "IP Pública", "DNS Público", "IP Privada", "DNS Privado", "Security Groups", "Estado"], (.Results[] | fromjson | [.accountId, .awsRegion, .resourceType, .configuration.instanceType, .configuration.dBInstanceClass, .arn, .configuration.description, .configuration.role, .configuration.volumeType, .configuration.size, .configuration.platform, .configuration.engineVersion, .configuration.architecture, .configuration.runtime, .configuration.vpcId, .configuration.subnetId, .configuration.dNSName, .configuration.publicIpAddress, .configuration.publicDnsName, .configuration.privateIpAddress, .configuration.privateDnsName, .configuration.securityGroups, .configuration.status]) | @csv'> data.csv &&iconv -f UTF-8 data.csv -o data.csv
The export command works correctly, but the problem occurs when converting from JSON to CSV, and it occurs only with some AWS accounts or resources, an error is generated in the conversion and the CSV generation process fails and stops , this is the output in the shell
Bus error (core dumped)
or
jq: error (at <stdin>:623): array ([{"value":"...) is not valid in a csv row
In this last case in particular, the conversion process with the JQ command was stopped, checking the first export was generated correctly and the line where the exception was generated contains this (comparing CSV file with JSON file), line not processed (not appears in CSV file):
"{\"accountId\":\"ID_ACCOUNT\",\"awsRegion\":\"us-east-1\",\"configuration\":{\"vpcId\":\"vpc-XXXXXX_ID\",\"securityGroups\":[{\"value\":\"sg-XXXXX_ID\"},{\"value\":\"sg-YYYY_UD\"}],\"dNSName\":\"awseb-AWSEB-XXXXXXX.us-east-1.elb.amazonaws.com\"},\"arn\":\"arn:aws:elasticloadbalancing:us-east-1:XXXX_ID:loadbalancer/app/awseb-AWSEB-XXXXXXX/YYYYY\",\"resourceType\":\"AWS::ElasticLoadBalancingV2::LoadBalancer\"}",
The question is, how could I fix this so that JQ processes the line generated by AWS configservice
Or alternatively, how could I skip this line to avoid and continue processing the file (I have more than 50 AWS accounts and doing it manually continuously is not optional)
Referral documentation: https://repost.aws/es/articles/ARQz1QqJx0R7ioSz3nI3noeg/returning-aws-config-aggregator-results-as-csv