Working with ⎕JSON#
We’ve laboriously gone through many different ways to work with parent vectors, but we’re yet to do any ‘real work’ with them. This page will cover an interesting application of the techniques we’ve learned so far. We’re going to look at manipulating JSON-formatted data, making use of Dyalog APL’s built-in ⎕JSON.
There’s a wonderful little API which returns some JSON describing every person currently in space.
]load HttpCommand
json←(HttpCommand.Get 'http://api.open-notify.org/astros.json').Data
⎕JSON⎕OPT'Compact' 0⎕JSON json ⍝ prettify it
#.HttpCommand
{
"message": "success",
"number": 12,
"people": [
{
"craft": "ISS",
"name": "Oleg Kononenko"
},
{
"craft": "ISS",
"name": "Nikolai Chub"
},
{
"craft": "ISS",
"name": "Tracy Caldwell Dyson"
},
{
"craft": "ISS",
"name": "Matthew Dominick"
},
{
"craft": "ISS",
"name": "Michael Barratt"
},
{
"craft": "ISS",
"name": "Jeanette Epps"
},
{
"craft": "ISS",
"name": "Alexander Grebenkin"
},
{
"craft": "ISS",
"name": "Butch Wilmore"
},
{
"craft": "ISS",
"name": "Sunita Williams"
},
{
"craft": "Tiangong",
"name": "Li Guangsu"
},
{
"craft": "Tiangong",
"name": "Li Cong"
},
{
"craft": "Tiangong",
"name": "Ye Guangfu"
}
]
}
Let’s say we want to group the names of these astronauts by the craft they are on, and return this as JSON as well. We want a structure like this:
{
"ISS": ["Oleg Kononenko", ...],
"Tiangong": ["Li Guangsu", ...]
}
with ... replaced by the rest of the names of course.
To accomlish this, we’re going to make use of a variant of ⎕JSON which parses JSON and returns a matrix describing its structure (rather than parsing it into an APL namespace, as is the default).
⎕JSON⎕OPT'Format' 'M'⊢json
┌─┬───────┬────────────────────┬─┐ │0│ │ │1│ ├─┼───────┼────────────────────┼─┤ │1│people │ │2│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Oleg Kononenko │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Nikolai Chub │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Tracy Caldwell Dyson│4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Matthew Dominick │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Michael Barratt │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Jeanette Epps │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Alexander Grebenkin │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Butch Wilmore │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │ISS │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Sunita Williams │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │Tiangong │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Li Guangsu │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │Tiangong │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Li Cong │4│ ├─┼───────┼────────────────────┼─┤ │2│ │ │1│ ├─┼───────┼────────────────────┼─┤ │3│craft │Tiangong │4│ ├─┼───────┼────────────────────┼─┤ │3│name │Ye Guangfu │4│ ├─┼───────┼────────────────────┼─┤ │1│number │12 │3│ ├─┼───────┼────────────────────┼─┤ │1│message│success │4│ └─┴───────┴────────────────────┴─┘
The ⎕JSON documentation describes the format of this table in detail. For our purposes, the most important column is the first, which gives the nesting depth for each row. You may recognise this as a form of depth vector, just like those we have used to describe trees previously. We can use this to create a parent vector representing our JSON.
⍪d keys values types←↓⍉⎕JSON⎕OPT'Format' 'M'⊢json ⍝ extract each column as a vector
⊢p←DepthToParent d ⍝ create the parent vector
PPH p
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │0 1 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 2 3 3 1 1 │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │┌┬──────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬┬─────┬────┬──────┬───────┐ │ │││people││craft│name││craft│name││craft│name││craft│name││craft│name││craft│name││craft│name││craft│name││craft│name││craft│name││craft│name││craft│name│number│message│ │ │└┴──────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴┴─────┴────┴──────┴───────┘ │ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │┌┬┬┬───┬──────────────┬┬───┬────────────┬┬───┬────────────────────┬┬───┬────────────────┬┬───┬───────────────┬┬───┬─────────────┬┬───┬───────────────────┬┬───┬─────────────┬┬───┬───────────────┬┬────────┬──────────┬┬────────┬───────┬┬────────┬──────────┬──┬───────┐│ │││││ISS│Oleg Kononenko││ISS│Nikolai Chub││ISS│Tracy Caldwell Dyson││ISS│Matthew Dominick││ISS│Michael Barratt││ISS│Jeanette Epps││ISS│Alexander Grebenkin││ISS│Butch Wilmore││ISS│Sunita Williams││Tiangong│Li Guangsu││Tiangong│Li Cong││Tiangong│Ye Guangfu│12│success││ │└┴┴┴───┴──────────────┴┴───┴────────────┴┴───┴────────────────────┴┴───┴────────────────┴┴───┴───────────────┴┴───┴─────────────┴┴───┴───────────────────┴┴───┴─────────────┴┴───┴───────────────┴┴────────┴──────────┴┴────────┴───────┴┴────────┴──────────┴──┴───────┘│ ├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │1 2 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 1 4 4 3 4 │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
0 0 1 2 2 1 5 5 1 8 8 1 11 11 1 14 14 1 17 17 1 20 20 1 23 23 1 26 26 1 29 29 1 32 32 1 35 35 0 0
┌───────┐ │ ┌∘┬∘│ │ │ └∘│ │ ├∘┬∘│ │ │ └∘│ │ ├∘┬∘│ │ │ └∘│ │ ├∘┬∘│ │ │ └∘│ │ ├∘┬∘│ │ │ └∘│ │ ├∘┬∘│ │ ┌∘┤ └∘│ │∘┤ ├∘┬∘│ │ │ │ └∘│ │ │ ├∘┬∘│ │ │ │ └∘│ │ │ ├∘┬∘│ │ │ │ └∘│ │ │ ├∘┬∘│ │ │ │ └∘│ │ │ ├∘┬∘│ │ │ │ └∘│ │ │ └∘┬∘│ │ │ └∘│ │ ├∘ │ │ └∘ │ └───────┘
The other columns, which we have extracted as k, v, and t, store the key, value, and type for a node respectively. For example, the row corresponding to "number": 12 has a key of 'number', a value of 12, and a type of 3, indicating the data is numeric. Labelling our tree with these vectors shows us exactly how parts of the tree correspond to the original JSON object.
keys PPH p
values PPH p
typeNames←'N/A' 'Object' 'Array' 'Numeric' 'String'
typeNames[types] PPH p
┌────────────────┐ │ ┌∘┬craft│ │ │ └name │ │ ├∘┬craft│ │ │ └name │ │ ├∘┬craft│ │ │ └name │ │ ├∘┬craft│ │ │ └name │ │ ├∘┬craft│ │ │ └name │ │ ├∘┬craft│ │ ┌people┤ └name │ │∘┤ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ │ └name │ │ │ └∘┬craft│ │ │ └name │ │ ├number │ │ └message │ └────────────────┘
┌──────────────────────────┐ │ ┌∘┬ISS │ │ │ └Oleg Kononenko │ │ ├∘┬ISS │ │ │ └Nikolai Chub │ │ ├∘┬ISS │ │ │ └Tracy Caldwell Dyson│ │ ├∘┬ISS │ │ │ └Matthew Dominick │ │ ├∘┬ISS │ │ │ └Michael Barratt │ │ ├∘┬ISS │ │ ┌∘┤ └Jeanette Epps │ │∘┤ ├∘┬ISS │ │ │ │ └Alexander Grebenkin │ │ │ ├∘┬ISS │ │ │ │ └Butch Wilmore │ │ │ ├∘┬ISS │ │ │ │ └Sunita Williams │ │ │ ├∘┬Tiangong │ │ │ │ └Li Guangsu │ │ │ ├∘┬Tiangong │ │ │ │ └Li Cong │ │ │ └∘┬Tiangong │ │ │ └Ye Guangfu │ │ ├12 │ │ └success │ └──────────────────────────┘
┌──────────────────────────┐ │ ┌Object┬String│ │ │ └String│ │ ├Object┬String│ │ │ └String│ │ ├Object┬String│ │ │ └String│ │ ├Object┬String│ │ │ └String│ │ ├Object┬String│ │ │ └String│ │ ├Object┬String│ │ ┌Array┤ └String│ │Object┤ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ │ └String│ │ │ └Object┬String│ │ │ └String│ │ ├Numeric │ │ └String │ └──────────────────────────┘
Now that we have all of this information, we can begin to manipulate our tree to group the astronaut’s names by their craft. Our first step will be to find those nodes which indicate the craft, and extract the unique craft names.
i←⍸'craft'∘≡¨keys ⍝ which nodes have a key of 'craft'
⊢crafts←∪values[i] ⍝ find the unique craft names
┌───┬────────┐ │ISS│Tiangong│ └───┴────────┘
Look again at our target structure.
{
"ISS": ["Oleg Kononenko", ...],
"Tiangong": ["Li Guangsu", ...]
}
We want a root object with two keys ("ISS" and "Tiangong"), each of which maps to an array. Our parent vector already has nodes in the indices 0 up to and including (≢p)-1, so our new nodes should have IDs ≢p and greater. So, we create our new root node with ID ≢p, and child nodes for each group with IDs (≢p)+1, (≢p)+2, and so on.
⊢root←≢p ⍝ new node for root object
⊢groups←root+1+⍳≢crafts ⍝ child nodes for groups
40
41 42
We then want to add these nodes and their associated data to each of the vectors we are using for this tree.
p ,←(1+≢groups)⍴root ⍝ each new node points to root as its parent
keys ,←(⊂''),crafts ⍝ root node has no key, group nodes have craft names as keys
values,←(1+≢groups)⍴⊂⍬ ⍝ no node has values assigned yet
types ,←1,(≢groups)⍴2 ⍝ root node has type 1 (object) and groups have type 2 (array)
keys PPH p
values PPH p
typeNames[types] PPH p
┌────────────────┬──────────┐ │ ┌∘┬craft│∘┬ISS │ │ │ └name │ └Tiangong│ │ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ │ └name │ │ │ ├∘┬craft│ │ │ ┌people┤ └name │ │ │∘┤ ├∘┬craft│ │ │ │ │ └name │ │ │ │ ├∘┬craft│ │ │ │ │ └name │ │ │ │ ├∘┬craft│ │ │ │ │ └name │ │ │ │ ├∘┬craft│ │ │ │ │ └name │ │ │ │ ├∘┬craft│ │ │ │ │ └name │ │ │ │ └∘┬craft│ │ │ │ └name │ │ │ ├number │ │ │ └message │ │ └────────────────┴──────────┘
┌──────────────────────────┬───┐ │ ┌∘┬ISS │∘┬∘│ │ │ └Oleg Kononenko │ └∘│ │ ├∘┬ISS │ │ │ │ └Nikolai Chub │ │ │ ├∘┬ISS │ │ │ │ └Tracy Caldwell Dyson│ │ │ ├∘┬ISS │ │ │ │ └Matthew Dominick │ │ │ ├∘┬ISS │ │ │ │ └Michael Barratt │ │ │ ├∘┬ISS │ │ │ ┌∘┤ └Jeanette Epps │ │ │∘┤ ├∘┬ISS │ │ │ │ │ └Alexander Grebenkin │ │ │ │ ├∘┬ISS │ │ │ │ │ └Butch Wilmore │ │ │ │ ├∘┬ISS │ │ │ │ │ └Sunita Williams │ │ │ │ ├∘┬Tiangong │ │ │ │ │ └Li Guangsu │ │ │ │ ├∘┬Tiangong │ │ │ │ │ └Li Cong │ │ │ │ └∘┬Tiangong │ │ │ │ └Ye Guangfu │ │ │ ├12 │ │ │ └success │ │ └──────────────────────────┴───┘
┌──────────────────────────┬────────────┐ │ ┌Object┬String│Object┬Array│ │ │ └String│ └Array│ │ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ │ └String│ │ │ ├Object┬String│ │ │ ┌Array┤ └String│ │ │Object┤ ├Object┬String│ │ │ │ │ └String│ │ │ │ ├Object┬String│ │ │ │ │ └String│ │ │ │ ├Object┬String│ │ │ │ │ └String│ │ │ │ ├Object┬String│ │ │ │ │ └String│ │ │ │ ├Object┬String│ │ │ │ │ └String│ │ │ │ └Object┬String│ │ │ │ └String│ │ │ ├Numeric │ │ │ └String │ │ └──────────────────────────┴────────────┘
Now, we want to move the astronaut names from where they are, to the correct group. We’re going to use a little trick to find the name nodes - since the original data was in DFPT order, and we haven’t changed the order so far, each "craft" node (which we found earlier as i) appears immediately before the "name" node it accompanies in the parent vector. You can see this in our original table, where each row containing the key 'craft' appears straight before the fellow row with the key 'name'. We can exploit this to be sure that the nodes i+1 are exactly the "name" nodes.
To move the astronaut names to their correct groups, we simply need to update their parents to the appropriate group node. We find the index in the crafts vector of the craft name we are concerned with, and use that to index the vector groups, whose craft names are ordered identically to crafts. We also need to update the keys of these name nodes to be empty, as when we feed this data back to ⎕JSON later, it will not be expecting array elements to have keys.
p[i+1]←groups[crafts⍳values[i]]
keys[i+1]←⊂''
keys PPH p
values PPH p
typeNames[types] PPH p
┌────────────────┬────────────┐ │ ┌∘─craft│ ┌∘ │ │ ├∘─craft│ ├∘ │ │ ├∘─craft│ ├∘ │ │ ├∘─craft│ ├∘ │ │ ├∘─craft│ ┌ISS┼∘ │ │ ┌people┼∘─craft│∘┤ ├∘ │ │∘┤ ├∘─craft│ │ ├∘ │ │ │ ├∘─craft│ │ ├∘ │ │ │ ├∘─craft│ │ └∘ │ │ │ ├∘─craft│ │ ┌∘│ │ │ ├∘─craft│ └Tiangong┼∘│ │ │ └∘─craft│ └∘│ │ ├number │ │ │ └message │ │ └────────────────┴────────────┘
┌──────────────┬────────────────────────┐ │ ┌∘─ISS │ ┌Oleg Kononenko │ │ ├∘─ISS │ ├Nikolai Chub │ │ ├∘─ISS │ ├Tracy Caldwell Dyson│ │ ├∘─ISS │ ├Matthew Dominick │ │ ├∘─ISS │ ┌∘┼Michael Barratt │ │ ┌∘┼∘─ISS │∘┤ ├Jeanette Epps │ │∘┤ ├∘─ISS │ │ ├Alexander Grebenkin │ │ │ ├∘─ISS │ │ ├Butch Wilmore │ │ │ ├∘─ISS │ │ └Sunita Williams │ │ │ ├∘─Tiangong│ │ ┌Li Guangsu │ │ │ ├∘─Tiangong│ └∘┼Li Cong │ │ │ └∘─Tiangong│ └Ye Guangfu │ │ ├12 │ │ │ └success │ │ └──────────────┴────────────────────────┘
┌──────────────────────────┬───────────────────┐ │ ┌Object─String│ ┌String│ │ ├Object─String│ ├String│ │ ├Object─String│ ├String│ │ ├Object─String│ ├String│ │ ├Object─String│ ┌Array┼String│ │ ┌Array┼Object─String│Object┤ ├String│ │Object┤ ├Object─String│ │ ├String│ │ │ ├Object─String│ │ ├String│ │ │ ├Object─String│ │ └String│ │ │ ├Object─String│ │ ┌String│ │ │ ├Object─String│ └Array┼String│ │ │ └Object─String│ └String│ │ ├Numeric │ │ │ └String │ │ └──────────────────────────┴───────────────────┘
Now we have two distinct trees, and we only need the newly created one, so we can simply delete the other using the idiom we described previously.
mask←root=p I@(root∘≠)⍣≡⍳≢p ⍝ mask of nodes which are in the new tree
p←(⍸~mask)(⊢-1+⍸)mask/p ⍝ delete nodes in the old tree
keys ←mask/keys ⍝─┐
values←mask/values ⍝ │ filter the associated data
types ←mask/types ⍝─┘
keys PPH p
values PPH p
typeNames[types] PPH p
┌────────────┐ │ ┌∘ │ │ ├∘ │ │ ├∘ │ │ ├∘ │ │ ┌ISS┼∘ │ │∘┤ ├∘ │ │ │ ├∘ │ │ │ ├∘ │ │ │ └∘ │ │ │ ┌∘│ │ └Tiangong┼∘│ │ └∘│ └────────────┘
┌────────────────────────┐ │ ┌Oleg Kononenko │ │ ├Nikolai Chub │ │ ├Tracy Caldwell Dyson│ │ ├Matthew Dominick │ │ ┌∘┼Michael Barratt │ │∘┤ ├Jeanette Epps │ │ │ ├Alexander Grebenkin │ │ │ ├Butch Wilmore │ │ │ └Sunita Williams │ │ │ ┌Li Guangsu │ │ └∘┼Li Cong │ │ └Ye Guangfu │ └────────────────────────┘
┌───────────────────┐ │ ┌String│ │ ├String│ │ ├String│ │ ├String│ │ ┌Array┼String│ │Object┤ ├String│ │ │ ├String│ │ │ ├String│ │ │ └String│ │ │ ┌String│ │ └Array┼String│ │ └String│ └───────────────────┘
Now, our tree corresponds exactly to the JSON structure we want to return, all that remains is to feed it back into ⎕JSON. Remember that the matrix format ⎕JSON uses has the depth vector as its first column. In order to pass our vectors back to ⎕JSON, we need to recover the depth vector. We could exploit the structure that we know our parent vector to have, since we controlled exactly how it was constructed, but we will instead take this opportunity to show how ParentToDepth (which we defined previously) works in practice.
⍝ find depths and DFPT ordering of p
d perm←ParentToDepth p
⍝ correct d, keys, values, and types to DFPT order
d ←d [perm]
keys ←keys [perm]
values←values[perm]
types ←types [perm]
We can then pack these vectors back into the matrix format expected by ⎕JSON.
⍉↑d keys values types
┌─┬────────┬────────────────────┬─┐ │0│ │ │1│ ├─┼────────┼────────────────────┼─┤ │1│ISS │ │2│ ├─┼────────┼────────────────────┼─┤ │2│ │Oleg Kononenko │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Nikolai Chub │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Tracy Caldwell Dyson│4│ ├─┼────────┼────────────────────┼─┤ │2│ │Matthew Dominick │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Michael Barratt │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Jeanette Epps │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Alexander Grebenkin │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Butch Wilmore │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Sunita Williams │4│ ├─┼────────┼────────────────────┼─┤ │1│Tiangong│ │2│ ├─┼────────┼────────────────────┼─┤ │2│ │Li Guangsu │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Li Cong │4│ ├─┼────────┼────────────────────┼─┤ │2│ │Ye Guangfu │4│ └─┴────────┴────────────────────┴─┘
And finally, we can pass this matrix back to ⎕JSON, and it generates the output we want.
⎕JSON⎕OPT('Format' 'M')('Compact' 0)⍉↑d keys values types
{
"ISS": [
"Oleg Kononenko",
"Nikolai Chub",
"Tracy Caldwell Dyson",
"Matthew Dominick",
"Michael Barratt",
"Jeanette Epps",
"Alexander Grebenkin",
"Butch Wilmore",
"Sunita Williams"
],
"Tiangong": [
"Li Guangsu",
"Li Cong",
"Ye Guangfu"
]
}