Quantcast
Channel: Active questions tagged jq - Stack Overflow
Viewing all articles
Browse latest Browse all 612

JSON conversion error from aws configservice query

$
0
0

I have a problem generating the list of my resources in AWS, and later converting them into CSV using the JQ command, exactly this is the command I am running from the AWS CLI

aws configservice select-resource-config --expression "SELECT  accountId,  awsRegion,  resourceType,  configuration.instanceType,  configuration.dBInstanceClass,  arn,  configuration.description,  configuration.role,  configuration.volumeType,  configuration.size,  configuration.platform,  configuration.engineVersion,  configuration.architecture,  configuration.runtime,  configuration.vpcId,  configuration.subnetId,  configuration.dNSName,  configuration.publicIpAddress,  configuration.publicDnsName,  configuration.privateIpAddress,  configuration.privateDnsName,  configuration.securityGroups,  configuration.status" --output json |  \jq -r '["ID Cuenta", "Region AWS", "Tipo Recurso", "Tipo Instancia EC2", "Tipo DB RDS", "ARN", "Descipcion", "Rol", "Tipo Volumen", "Tamaño", "Plataforma", "Engine Version", "Arquitectura", "Runtime", "VPC", "Subnet ID", "DNS", "IP Pública", "DNS Público", "IP Privada", "DNS Privado", "Security Groups", "Estado"], (.Results[] | fromjson | [.accountId, .awsRegion, .resourceType, .configuration.instanceType, .configuration.dBInstanceClass, .arn, .configuration.description, .configuration.role, .configuration.volumeType, .configuration.size, .configuration.platform, .configuration.engineVersion, .configuration.architecture, .configuration.runtime, .configuration.vpcId, .configuration.subnetId, .configuration.dNSName, .configuration.publicIpAddress, .configuration.publicDnsName, .configuration.privateIpAddress, .configuration.privateDnsName, .configuration.securityGroups, .configuration.status]) | @csv'> data.csv &&iconv -f UTF-8 data.csv -o data.csv

The export command works correctly, but the problem occurs when converting from JSON to CSV, and it occurs only with some AWS accounts or resources, an error is generated in the conversion and the CSV generation process fails and stops , this is the output in the shell

Bus error (core dumped)

or

jq: error (at <stdin>:623): array ([{"value":"...) is not valid in a csv row

In this last case in particular, the conversion process with the JQ command was stopped, checking the first export was generated correctly and the line where the exception was generated contains this (comparing CSV file with JSON file), line not processed (not appears in CSV file):

"{\"accountId\":\"ID_ACCOUNT\",\"awsRegion\":\"us-east-1\",\"configuration\":{\"vpcId\":\"vpc-XXXXXX_ID\",\"securityGroups\":[{\"value\":\"sg-XXXXX_ID\"},{\"value\":\"sg-YYYY_UD\"}],\"dNSName\":\"awseb-AWSEB-XXXXXXX.us-east-1.elb.amazonaws.com\"},\"arn\":\"arn:aws:elasticloadbalancing:us-east-1:XXXX_ID:loadbalancer/app/awseb-AWSEB-XXXXXXX/YYYYY\",\"resourceType\":\"AWS::ElasticLoadBalancingV2::LoadBalancer\"}",

The question is, how could I fix this so that JQ processes the line generated by AWS configservice

Or alternatively, how could I skip this line to avoid and continue processing the file (I have more than 50 AWS accounts and doing it manually continuously is not optional)

Referral documentation: https://repost.aws/es/articles/ARQz1QqJx0R7ioSz3nI3noeg/returning-aws-config-aggregator-results-as-csv


Viewing all articles
Browse latest Browse all 612

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>