Link Search Menu Expand Document
Table of contents
  1. Building and Editing JSON with jq
    1. Introduction
    2. Getting jq
    3. Defining Syntax
      1. The Dot
      2. Paths
      3. Array Access
      4. Editing Values
      5. The Pipe
      6. Logical Operations
        1. If, Then, Else
        2. And, Or, Not
      7. Variables
    4. Builtin Functions
      1. Syntax
      2. Math
      3. Map and Map Values
      4. To and From Entries
      5. Walk
      6. Further Reading
    5. Defining Custom Functions
    6. Command Line Use
    7. Bringing It All Together

Building and Editing JSON with jq

Introduction

“jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.”

— https://stedolan.github.io/jq/

jq is a JSON editing program written in C. It can be thought of similarly to Perl. Because it was specifically written to parse JSON, however, it has many useful features that are not present in other text processors. jq defines its programs as filters, as they take an input, your JSON data, and produce an output, your modified JSON data. Though seemingly simple at first glance, jq contains many advanced features that can be extremely useful when procedurally generating addon files.

I’ll be explaing jq from the perspective of jq as a command line interface. However, wrappers have been designed for jq in essentially every programming language, so you can easily incorporate jq filters into the programing language of your choice, including Go, JavaScript, Java, Ruby, Python, and R. Given that, I will primarily focus on filter design from the perspective of Minecraft addons, and I’ll leave the implementation to your specific use case. I’ll be including embeds from jqterm.com, which uses a JavaScript implementation of jq to display filters in a web browser.

Getting jq

The official source to download the jq binaries is https://stedolan.github.io/jq/download/. The executables downloaded here should be largely portable and just “run”.

You may also use your system package manager, though if you take this approach, ensure that the version installed is the latest, jq-1.6.

A community maintained list of the wrappers available for jq in various languages is available here.

Defining Syntax

The Dot

The simplest possible filter in jq is the dot (.), which will simply take the input JSON and return it unchanged. jq uses the standardized path notation for JSON files. Consider the following structure of a resource pack manifest:

Paths

Here, we see that our . filter simply returns what we expected: the entire JSON file. Now, let’s say we wanted to return some specific part of the JSON file, the header UUID for example. We would then construct a filter that contains the full path to that value. In this case, that would be .header.uuid. Let’s see that in action:

Array Access

We can access arrays in jq similarly to how we would in other languages. Let’s say we wanted to return the last number in our minimum engine version array. We would construct the following filter: .header.min_engine_version[-1]. Note that, of course, we could also specify the positive position 2 if we always wanted to return the second position. Simply using empty brackets, .[], will return all elements of the array in line separated list. Feel free to play around with this and learn the basics:

Editing Values

Now let’s get into the real meat of jq and start manipulating our JSON data. The way we do this is with the plain assignment operator (=). There are a few variants of this operator, referred to as update-assignment operators, such as |= and +=, which you can read more about in jq’s official manual, and experiment with in the example below:

The Pipe

Similarly to unix command line interfaces, jq makes use of the pipe, |. When you use the pipe character, you are essentially telling jq to take the result of your filter on the left of the pipe, and run it through the filter to the right of the pipe. Let’s go back to our array access example, except this time we will output the entire array apply some filter to the output array. We’ll use jq’s builtin map function to demonstrate this, which is a very useful function as it can be used to apply a specified filter to every value in an input array. We’ll add 1 to each value in the minimum engine version array:

Logical Operations

If, Then, Else

Logical statements in jq essentially written in a human readable format. They take the form if A then B else C end. ‘A’ in this case could test for a specific value, or simply test if some key exists. More conditions can be added with elif in the form if A then B elif C then D else E end. For comparisons, the operators are essentially the same as Molang, utilizing ==, !=, >, >=, <, and <=. In the example below, we’ll conditionally change the description of the pack based on the format version. If the format version is greater than 1, we’ll set the description to “Big Format”. Otherwise, we’ll leave the description alone:

And, Or, Not

The operations and and or are also written in plain english. If we wanted to test for condition ‘A’ and condition ‘B’, we’d simple write if (A and B) then C else D.

The operation not behaves like a single argument function, which we’ll discuss in further depth below. When we pipe some boolean into not, it will return the opposite value. We’ll demonstrate this by setting our header description to “Little Version” only if both our header version and module version are not greater than 1:

Variables

Variables can be defined anywhere in filter. When a variable is defined, it is valid for use anywhere downstream in the pipe from where it was initially defined. They are defined in the form: .input as $var | $var, which would effectively return .input. Any valid JSON type can also serve as an input, such as objects, arrays, strings, and numbers. Take the simple example:

Builtin Functions

Syntax

jq comes with a variety of builtin functions. We’ve already encountered the function map while explaining the pipe. One thing of note regarding jq functions is that their syntax differs depending on the number of arguments they take. Multi argument functions will take their values enclosed in parentheses and separated by semicolons. In contrast, for single argument functions, the input is piped in. Take the example below, in which we use the multi-argument function pow to raise format version to the third power, while we also use the single argument function sqrt to find the square root of format version. You’ll also note in this example that we introduce the concept of building an entirely new JSON structure within our jq filter. This is extremely useful in situations where we’d like to somehow reshape some input file. It’s always important to keep in mind that any valid JSON is valid anywhere in a jq filter:

Math

jq incorporates C-style math functions and generally carries over all builtin C math functions from your system. Refer to the manual for a full list of these.

Map and Map Values

We’ve already seen a bit of what map can do, as seen in the pipe section. I’ll explain a bit about its cousin, map_values. While map can apply some filter to all values in an array, map_values is used to apply some filter to all keys in an input list of objects. In the example below, I’ll introduce map_values, as well as couple extra functions. Our goal will be to substitute the string “Example” with “Production” by iterating through all the values with the type string under header. We must separate out the string values from the array values, as we cannot perform the function we will use to make this substitution, gsub, on arrays. The function gsub is a multi-argument function that will take the substring we would like to change, followed by the result we’d like to change that substring to. We’ll also be utilizing the select function to separate our strings from our arrays. We’ll incorporate a variable so we have easy access to all the information contained in header. We’ll make use of the function type, which lets us check the JSON data type of an input.

To and From Entries

The function to_entries is a useful single argument jq builtin that is helpful when we’d like to rearrange keys and values. Lets say we wanted to invert all the key/value pairs in header. Using to_entries, we’ll construct an array of all key value pairs in header, reformatted as objects and assigned “key” or “value” so we can reference them later. To understand how this function works, it’s important to see what this intermediate looks like. Note that this can also be reversed by appending from_entries:

Now, we’ll use map to iterate through each array entry, and finally we’ll introduce a new function, add. This is a single-argument function that will take the array of objects output by map and essentially move them up one level so that they again exist as key value pairs under header. We’ll also need to convert all our values to strings so they can serve as keys. We’ll use the single-argument function tostring. Note that tonumber is the inverse of this:

Walk

The function walk is extremely powerful, as it will recursively apply to everything in the input, at all levels. Take the example of deleting all objects with the value null. We’ll also be introducing the function with_entries, which is just a shorthand for to_entries | map(some filter) | from_entries. We’ll start our filter with walk, indicating we want to go through everything. We’ll then use an if statement to ensure we are only editing objects, else we’ll leave the input unchanged. For all objects, we’ll use with_entries to convert those to mapped keys and values, and run a select function on those to ensure we only carry on values that are not equal to null:

Further Reading

Obviously, jq has far too many functions to list off here. I’ve tried to explain those that I find most useful in my jq projects. For further reading, refer to the jq manual.

Defining Custom Functions

In jq, one can easily define custom functions with the syntax def function_name($input1; $input2): some function;. The variables can then be used all throughout the function. A function can also have no input. Take the boiler plate examples below:

Command Line Use

When invoked from the command line, jq takes the form jq '[filter]' input.json. This will only display the output in the terminal. For bash-based systems, simply use the arrow operator, jq '[filter]' input.json > output.json to write an output file. If you wish to overwrite the input file on the fly, this method will not work. You will either need to generate a temp file, or use a utility like Sponge from moreutils. Then, you may simply jq '[filter]' input.json | sponge input.json. For large filters, it’s generally easier to employ a shell or bash script. You may use the -n flag if you wish to construct a JSON file from scratch rather than specify an input. Finally, you may pass terminal variables to be used in your jq filters as follows:

jq -n --arg jqvar1 $var1 --arg jqvar2 $var2 '
{
"var1": $jqvar1,
"var2": $jqvar2
}'

Bringing It All Together

Many of the previous examples are abstract and contrived, designed to allow you to become familiar with the basics of jq. Now, I’d like to go over an example of a practical use case of jq. I’ll be explaining the jq filter I wrote for converting a Java model to a Bedrock model, for use as an attachable, while remapping all it’s UV values.

First, we’ll define a simple variable so that we can easily change the name of our model geometry. Next, we’ll define a function to deal with our elements, element_array. We’ll first get some information from the array of textures from our Java model so that we know how many textures we have, what order they’re in, and therefore how to shift our UV values. Now that we’ve defined all our variables from our texture array, which we’ll use later in UV calculations, we’ll start to map our elements array.

Iterating through each element in our array of elements, we’ll calculate our origin and size. We can use simple math from jq to do this. In order to go from Java to Bedrock, we’ll need to change 2 absolute corner positions to one origin value, and one size value. We’ll also shift our X and Z coordinates by 8 to account for the different centering of coordinate systems between Bedrock and Java.

Now, we’ll deal with rotation. The way Java defines its rotations uses a string followed by a value, but we’d like to construct an array based on that. We can do so with a fairly simple if, then, else statement. Since rotation is optional, we’ll just set that to null for now if we don’t have a rotation. We can delete that later. Next, we’ll set the element pivot, again shifting by 8 on the X and Z due to the coordinate difference, and setting to null if we lack a pivot.

We will then calculate our UV values. I wish to shift all our UV’s as if we are generating a single square texture atlas from our input textures. We’ll define a nested function, uv_calc, here. We’ll then use some modular division via the fmod function to figure out the offsets. Finally, we’ll change Java’s four coordinate style UV system to Bedrock’s, which uses a point value and a size value, with simple arithmetic. We’ll then run the function for each of the six faces of our element, and finally, use walk to remove any objects with the value null.

In order to make our geometry compatible with custom blocks, we’ll define the function pivot_groups, which we’ll ultimately use to iterate over our element array, find all unique rotation pivot combinations, and create a bone for all elements that share a rotation-pivot combonation.

Lastly, we’ll put down the structure of our actual geometry file. We’ll concatenate in our model name, use our element_array function to bring in all our cubes with no rotation, and add in our special pivot groups. We’ll use map to give each group a unique name, iterating up one value each time.

This filter is a little heavy, so I’ve linked it on a separate page rather than embedding it here.