Using a combination of jq and (if necessary) bash I would like to merge two jsons that are backwards compatible structure-wise; json1 is the base and contains more elements than json2, but all elements in json2 can be copied to jsn1 without adding newly named elements. All order and structure should remain the same.
Everything in json1 must remain in the output, unless json2 contains something different (ordering in arrays and objects should not matter). The structure and order of json1 must be maintained in the output.
json1:
{"idn" : "00000007000000021000","name" : "Nederlandse Gezelligheidsinspectie","organizationCode" : "0555","escapeReasons" : [ ],"units" : [ {"type" : "SPIERING","distinguishedName" : "ou=spiering,ou=abc,o=wiwo,c=nl","escapeReasons" : [ ],"splunkName" : "spiering","reportSets" : [ ],"functionRoles" : [ {"externalId" : "NGIspiering","functionRoleId" : "AB1234","name" : "Spiering Medewerker","functionType" : "EIGEN","taskIds" : ["ngi-spiering-medewerker","ngi-spiering-op-feestnummer" ] }, {"externalId" : "NGISpieringAdministratie","functionRoleId" : "AB5678","name" : "Beheren gebruikers","functionType" : "BEHEER","taskIds" : ["ngi-spiering-byz-beheren-administratie" ] }, {"externalId" : "NGISPieringOphalenRapportages","functionRoleId" : "AB0987","name" : "Spiering Ophalen Generieke Rapportages","functionType" : "BEHEER","taskIds" : ["ngi-spiering-byz-ophalen-rapportages" ] }, {"externalId" : "NGISpieringZoekenWWWteken","functionRoleId" : "DG4567","name" : "Opsporing Zoeken op WWW teken","functionType" : "ZOEKEN","taskIds" : ["ngi-www-kengeenteken" ] }, {"externalId" : "NGISpieringEoan","functionRoleId" : "BC4567","name" : "Spieringregister","functionType" : "VAST","taskIds" : ["ngi-nen" ] }, {"externalId" : "NGIPSpieringZoekenVasteLocatie","functionRoleId" : "HG8475","name" : "Spiering Zoeken Raster - locatie","functionType" : "ZOEKEN","taskIds" : ["ngi-zoeken-raster-locatie" ] }, {"externalId" : "NGISpieringZoekenLasterPenceel","functionRoleId" : "XZ2648","name" : "Opsporing Zoeken Laster - penceel","functionType" : "ZOEKEN","taskIds" : ["ngi-zoeken-laster-penceel" ] }, {"externalId" : "NGISpieringZoekenWWWkkk","functionRoleId" : "FR9558","name" : "Spiering WWW Zoeken op kkk nummer","functionType" : "ZOEKEN","taskIds" : ["ngi-www-kkk" ] }, {"externalId" : "NGISpieringZoekenWWWPersoon","functionRoleId" : "PC7837","name" : "Spiering WWW Zoeken op Persoon","functionType" : "ZOEKEN","taskIds" : ["ngi-www-persoon" ] } ] }, {"type" : "BOEZOEKT","distinguishedName" : "ou=boezoekt,ou=ngi,o=wiwo,c=nl","escapeReasons" : [ ],"splunkName" : "BOEZOEKT","reportSets" : [ ],"functionRoles" : [ {"externalId" : "NGIBoezoektBBB","functionRoleId" : "PW8237","name" : "Boezoekt BBB Uitgebreid","functionType" : "EIGEN","taskIds" : ["ngi-boezoekt-wwwuitgebreid" ] }, {"externalId" : "NGItoezichtABC","functionRoleId" : "RW6374","name" : "Boezoekt ABC Medewerker","functionType" : "EIGEN","taskIds" : ["ngi-boezoekt-amfmedewerker","ngi-zoeken-op-eennummer" ] }, {"externalId" : "NGIBoezoektARBI","functionRoleId" : "TE7374","name" : "Boezoekt ARBI Medewerker","functionType" : "EIGEN","taskIds" : ["ngi-boezoekt-arbimedewerker","ngi-zoeken-op-eennummer" ] }, {"externalId" : "NGIBoezoektBeherenGebruikersadministratie","functionRoleId" : "RE1234","name" : "Beheren gebruikers","functionType" : "BEHEER","taskIds" : ["ngi-boezoekt-byz-beheren-gebruikersadministratie" ] }, {"externalId" : "NGIBoezoektOphalenRapportages","functionRoleId" : "WE8374","name" : "Boezoekt Ophalen Generieke Rapportages","functionType" : "BEHEER","taskIds" : ["ngi-boezoekt-byz-ophalen-rapportages" ] }, {"externalId" : "NGIBoezoektZoekenWWWKenteken","functionRoleId" : "HW7384","name" : "Boezoekt WWW Zoeken op Kenteken","functionType" : "ZOEKEN","taskIds" : ["ngi-www-eenteken" ] }, {"externalId" : "NGIBoezoektNpr","functionRoleId" : "YE7273","name" : "Wandelregister","functionType" : "VAST","taskIds" : ["ngi-npr" ] }, {"externalId" : "NGIBoezoektZoekenKaderLocatie","functionRoleId" : "GE8264","name" : "Boezoekt Zoeken Kader - locatie","functionType" : "ZOEKEN","taskIds" : ["ngi-zoeken-kader-locatie" ] }, {"externalId" : "NGIBoezoektZoekenKaderPers","functionRoleId" : "GE8273","name" : "Boezoekt Zoeken Kader - pers","functionType" : "ZOEKEN","taskIds" : ["ngi-zoeken-kader-pers" ] } ] } ]}json2:
{"idn": "00000007000000021000","name": "Nederlandse Gezelligheidsinspectie","organizationCode": "0531","units": [ {"type": "SPIERING","distinguishedName": "ou=spiering,ou=abc,o=wiwo,c=nl","splunkName": "spiering","reportSets": ["ngi-spiering" ] }, {"type": "BOEZOEKT","distinguishedName": "ou=boezoekt,ou=nla,o=wiwo,c=nl","splunkName": "boezoekt","reportSets": ["ngi-boezoekt" ] } ]}Expected result:
"oin": "00000004000000021000","name": "Nederlandse Arbeidsinspectie","organizationCode": "0531\r","escapeReasons": [],"units": [ {"type": "OPSPORING","distinguishedName": "ou=opsporing,ou=nla,o=suwi,c=nl","escapeReasons": [],"splunkName": "opsporing","reportSets": ["nla-opsporing"],"functionRoles": [ {"externalId": "NLAopsporing","functionRoleId": "FR9550","name": "Opsporing Medewerker","functionType": "EIGEN","taskIds": ["nla-opsporing-medewerker","nla-zoeken-op-loonheffingennummer" ] }, {"externalId": "NLAOpsporingBeherenGebruikersadministratie","functionRoleId": "FR9552","name": "Beheren gebruikers","functionType": "BEHEER","taskIds": ["nla-opsporing-byz-beheren-gebruikersadministratie" ] }, {"externalId": "NLAOpsporingOphalenRapportages","functionRoleId": "FR9553","name": "Opsporing Ophalen Generieke Rapportages","functionType": "BEHEER","taskIds": ["nla-opsporing-byz-ophalen-rapportages" ] }, {"externalId": "NLAOpsporingZoekenRDWKenteken","functionRoleId": "FR9554","name": "Opsporing RDW Zoeken op Kenteken","functionType": "ZOEKEN","taskIds": ["nla-rdw-kenteken" ] }, {"externalId": "NLAOpsporingNhr","functionRoleId": "FR9555","name": "Handelsregister","functionType": "VAST","taskIds": ["nla-nhr" ] }, {"externalId": "NLAOpsporingZoekenKadasterLocatie","functionRoleId": "FR9556","name": "Opsporing Zoeken Kadaster - locatie","functionType": "ZOEKEN","taskIds": ["nla-zoeken-kadaster-locatie" ] }, {"externalId": "NLAOpsporingZoekenKadasterPerceel","functionRoleId": "FR9557","name": "Opsporing Zoeken Kadaster - perceel","functionType": "ZOEKEN","taskIds": ["nla-zoeken-kadaster-perceel" ] }, {"externalId": "NLAOpsporingZoekenRDWKvK","functionRoleId": "FR9558","name": "Opsporing RDW Zoeken op KVK nummer","functionType": "ZOEKEN","taskIds": ["nla-rdw-kvk" ] }, {"externalId": "NLAOpsporingZoekenRDWPersoon","functionRoleId": "FR9559","name": "Opsporing RDW Zoeken op Persoon","functionType": "ZOEKEN","taskIds": ["nla-rdw-persoon" ] } ] }, {"type": "TOEZICHT","distinguishedName": "ou=toezicht,ou=nla,o=suwi,c=nl","escapeReasons": [],"splunkName": "toezicht","reportSets": ["nla-toezicht"],"functionRoles": [ {"externalId": "NLAtoezichtBRP","functionRoleId": "FR9500","name": "Toezicht BRP Uitgebreid","functionType": "EIGEN","taskIds": ["nla-toezicht-brpuitgebreid" ] }, {"externalId": "NLAtoezichtAMF","functionRoleId": "FR9502","name": "Toezicht AMF Medewerker","functionType": "EIGEN","taskIds": ["nla-toezicht-amfmedewerker","nla-zoeken-op-loonheffingennummer" ] }, {"externalId": "NLAtoezichtARBO","functionRoleId": "FR9503","name": "Toezicht ARBO Medewerker","functionType": "EIGEN","taskIds": ["nla-toezicht-arbomedewerker","nla-zoeken-op-loonheffingennummer" ] }, {"externalId": "NLAToezichtBeherenGebruikersadministratie","functionRoleId": "FR9504","name": "Beheren gebruikers","functionType": "BEHEER","taskIds": ["nla-toezicht-byz-beheren-gebruikersadministratie" ] }, {"externalId": "NLAToezichtOphalenRapportages","functionRoleId": "FR9505","name": "Toezicht Ophalen Generieke Rapportages","functionType": "BEHEER","taskIds": ["nla-toezicht-byz-ophalen-rapportages" ] }, {"externalId": "NLAToezichtZoekenRDWKenteken","functionRoleId": "FR9506","name": "Toezicht RDW Zoeken op Kenteken","functionType": "ZOEKEN","taskIds": ["nla-rdw-kenteken" ] }, {"externalId": "NLAToezichtNhr","functionRoleId": "FR9507","name": "Handelsregister","functionType": "VAST","taskIds": ["nla-nhr" ] }, {"externalId": "NLAToezichtZoekenKadasterLocatie","functionRoleId": "FR9508","name": "Toezicht Zoeken Kadaster - locatie","functionType": "ZOEKEN","taskIds": ["nla-zoeken-kadaster-locatie" ] }, {"externalId": "NLAToezichtZoekenKadasterPerceel","functionRoleId": "FR9509","name": "Toezicht Zoeken Kadaster - perceel","functionType": "ZOEKEN","taskIds": ["nla-zoeken-kadaster-perceel" ] } ] }, {"type": "OPSPORING","distinguishedName": "ou=opsporing,ou=nla,o=suwi,c=nl","splunkName": "opsporing","reportSets": ["nla-opsporing" ] }, {"type": "TOEZICHT","distinguishedName": "ou=toezicht,ou=nla,o=suwi,c=nl","splunkName": "toezicht","reportSets": ["nla-toezicht" ] } ]}Note the filled elements reportSets in both units objects.
Here is the code we are currently using:
jq -s 'def deepmerge(a;b): reduce b[] as $item (a; reduce ($item | keys_unsorted[]) as $key (.; $item[$key] as $val | ($val | type) as $type | .[$key] = if ($type == "object") then deepmerge({}; [if .[$key] == null then {} else .[$key] end, $val]) elif ($key == "units") then (.[$key] + $val | unique) else $val end) ); deepmerge({}; .)'"$OUTPUT_DIR/$json1.json" "$OUTPUT_DIR/$json2.json" > "$OUTPUT_DIR/$oin-in-progress.json"As you can see units is being copied manually, but since adding units.reportSets to the if clause would cause that comparison to fire before the units copy (because of reverse execution due to the recursive algorithm 'unwinding'), the units copy would overwrite the units.reportSets copy. We want the functionRoles element to be untouched, this is a json1 only element and must be copied. All other elements found in json2 should overwrite (not append) the same element in the same location in json1.
We were successful in manually copying the "units" element, but with that the "functionRoles" element was also overwritten. We do not want this in the solution.