Restoration and Graphics Programming of Web Artwork

Written by Karl Rosenberg on November 30, 2018

I have recently finished a collaborative project with the Guggenheim Time-based Media Lab and NYU CS as the lead programmer for the restoration of a software-based web artwork.

Called Unfolding Object (by the artist John F. Simon Jr.), the piece was a Java Applet. Java Applets have been deprecated for some time, so almost no one without an antiquated web browser could see it. In short, we decided to rewrite the piece in JavaScript and WebGL. The restoration is viewable and the full story is readable here.

I’d like to focus on some of the graphics programming aspects, but first: a brief, nonexhaustive description of the work:

Unfolding Object is a 2D artwork that begins as a single square in the center of the screen. The user chooses to click any of the flaps (repeatedly), causing new pages to be unfolded. (Picture origami.) Lines drawn atop the unfolded pages represent the number of people in the world who previously explored a path (the sequence of up, down, left, right unfolds across pages).

How are the shapes represented and drawn with WebGL?

Data Representation:

Each page is really two triangulated quads (4 vertices) represented as a series of values in a large contiguous buffer. Each vertex has x, y, z, r, g, b, a values for position and color. (Although the work is 2D, the z value will have some use.) Lines have the same value format and are stored in a different contiguous buffer, but instead of instructing WebGL to draw triangles with the data, I draw lines.

Control Flow:

Unfolding Object follows a painter’s algorithm-like system where every frame, every entity is redrawn in order, in layers. To simulate this with GL buffers, I set two position indices to the start of the quad buffer and line buffers, and every time we need to paint a polygon or line, I check if the previously-existing buffer data needs to be changed. If not, I just move the index along the buffer. Otherwise I add the data. At the end of the frame, after I have issued the draw calls, I “rewind” the indices back to 0. I think of this like magnetic tape in a music studio.

Avoiding Draw Calls:

It’s generally inefficient to issue too many draw calls, and if I were to draw each quad and line separately, then there might be performance penalties down the line–even if the animations aren’t too complex. In the best case, it would be great to do one draw call per primitive (triangle, line). (Note, I realize that lines can be created with triangles, but GL.LINES had the exact aesthetic we needed for the lines in this piece.) To do this, instead of storing quad and line data as separate objects to be drawn separately, we have the aforementioned contiguous buffers. Excellent! Now we can just use two draw calls to generate the quads and lines all at once. There’s one problem: Unfolding Object interleaves the quads and lines. Using two draw calls would just place all of the lines atop the quads. That doesn’t work.

Interleaving:

A simple solution is to use a z-coordinate for each shape to represent a layer on the 2D canvas. Every time I need to change a color or switch the draw primitive (triangle, line), I increment and set a z coordinate for the quad or line. That way, they will be layered correctly when GL do depth testing. Now I can use only one draw call per large contiguous buffer! (Actually, the depth buffer has finite floating-point precision, and I need to keep the shapes from being too close in z, so if there are too many pieces of data in the buffers, I create another large contiguous buffer. The number of draw calls is 2 * (number of quad batches + number of line batches). In practice, I don’t think that most people will exceed the limit, but I wanted to be careful!)

Alternative Unimplemented Approach:

I ought to mention: a nice property of Unfolding Object that I would have liked to use was that new quads and lines always overlapped old ones. I thought I could simply draw all finalized shapes permanently onto a texture, meaning that I could update only a handfull of most-recent shapes before discarding them. The issue was that Unfolding Object sometimes updates old lines’ positions, and since the lines were layered between quads, my alternative approach would be unworkable.

Roughly speaking, that’s how the system works. It’s a simple 2D renderer.

Thank you for reading!

Banner Image Credit: the Solomon R. Guggenheim Foundation. Left: original work, Middle: HTML Canvas 2D prototype, Right: WebGL final

 

 

 

 

 

 

 

 

 

All of the details of

0 thoughts on “Restoration and Graphics Programming of Web Artwork”

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.