Quantcast
Channel: Active questions tagged jq - Stack Overflow
Viewing all articles
Browse latest Browse all 657

Not getting expected result from Bash script to remove IPs from JSON [closed]

$
0
0

I have a couple of bash scripts that I'm trying to put together:

  1. Appends an IP to a JSON file, and adds the IP and Epoch Time added to a CSV (ip_addresses.csv) with the headers "IP_Address" and "Date"
  2. Removes IPs that are older than 90 days from the JSON file, and also adds the IPs that were removed and the date it was removed to a separate CSV (removed.csv)

I'm not getting the results I want from the removal script, and not sure where I'm going wrong.

JSON File Format:

{"version": "1.0","description": "Test IP JSON","objects": [    {"name": "IP_Addresses","id": "NA","description": "Test list of IPs","ranges": ["1.1.1.1","20.0.0.0/24","10.1.1.2-10.1.1.10","2.2.2.2","3.3.3.3"      ]    }  ]}

Removal script:

#!/bin/bash# This script loops through the csv and removes entries older than ninety days.# Input/Output CSV file (same file for both input and output)json_file="ip_addresses.json"file="ip_addresses.csv"removed_file="removed.csv"# Temporary file for storing filtered resultstemp_file=$(mktemp)# Array to store removed IPsremoved_ips=()# Get the current date in seconds since epochcurrent_date=$(date +%s)current_date_human=$(date '+%Y-%m-%d')# Write the headers to the temporary filehead -n 1 "$file" > "$temp_file"# Loop through each line of the input file (skipping the header)tail -n +2 "$file" | while IFS=',' read -r ip date_added; do    # Calculate the age of the entry in days    age_days=$(( (current_date - date_added) / 86400 ))    if [ "$age_days" -le 90 ]; then        # If the entry is less than or equal to 90 days old, keep it        echo "$ip,$date_added" >> "$temp_file"    else        # If the entry is more than 90 days old, move it to the removed file        echo "$ip,$current_date_human" >> "$removed_file"        # Add the IP to the removed_ips array        removed_ips+=("$ip")    fidone# Overwrite the original CSV with the filtered resultsmv "$temp_file" "$file"# Loop over the array and remove the entry in the Generic Data Center JSON filefor removed_ip in "${removed_ips[@]}"; do    # Escape periods in IP for use in jq    escaped_ip=$(echo "$removed_ip" | sed 's/\./\\./g')    # Check if the IP exists in the JSON "ranges" and remove it if found    echo "Checking for $removed_IP and removing if found"    echo "..."    jq --arg ip "$removed_ip" '(.objects[] | select(.name == "IP_Addresses") | .ranges) |= map(select(. != $ip))'"$json_file" > temp_json && mv temp_json "$json_file"done

I'm testing with 3 entries in the JSON file:

  • 4.1.1.2 1727274047 (1 year ago)
  • 4.1.1.3 1711376447 (6 months ago)
  • 4.1.1.4 1724595647 (1 month ago)

However, when I run the removal script the IPs are not removed from the JSON file, and only 4.1.1.3 is removed from the CSV file (when it should be 4.1.1.2 as well). I can see 4.1.1.3 in the removed.csv file.

I'm not sure if it's an issue with my jq or if I'm going wrong with my logic elsewhere, I'm not too experienced with JQ or bash.


Viewing all articles
Browse latest Browse all 657

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>