Rendering wide-format aerial film digitally is no simple task – and required inventing a new high-powered scanner to do the job
By Brad Kvederis
High-precision photography represents a unique anomaly in the digital age: one of the rare instances in which the modern, electronic state of the art sometimes struggles to keep up with its vintage analog equivalent. It’s true that digital cameras have surpassed their film-based counterparts in nearly every measure of cost, simplicity, capacity, speed, and convenience; and even in relatively demanding applications like motion picture production, they’re good enough to get the job done. But when the goal is to achieve the absolute highest possible resolution, there’s still no substitute for film.
In the rarefied world of high-end aerial photography, you could say that the past has given digital technology more than it can handle. Not only is the level of detail higher than digital cameras can match; the huge size of the film – which can measure more than 10 inches in width – exceeds the limits of modern equipment to convert and render digitally.
In recent years, that’s become an issue of growing importance for entities like the National Geospatial-Intelligence Agency (NGA), U.S. Geological Survey (USGS), the Department of Agriculture, and other agencies both public and private that have accumulated collections of aerial film spanning decades and numbering in the tens of thousands and in some cases millions of rolls. Due to the laborious and highly manual process of rendering the film with existing technology, the backlog of work was years, sometimes even decades long.
In 2018, our company, nextScan, became involved in a bid to develop a new machine for the U.S. government that would be capable of digitizing a large agency’s entire film archive in bulk. We and our parent company, Digital Check, had been working with microfilm and high-precision cameras since the 1950s, so we had plenty of relevant experience from which to draw. Nonetheless, this project provided a great learning experience about a film format that just hadn’t really been accounted for as modern digital technology evolved.
To get an idea of the challenge presented by these enormous rolls of film, consider them in terms of some of the formats we see in everyday life. We’ll take 9-inch film – the type used by the SR-71 “Blackbird” reconnaissance plane and the U2 spy plane, among others – as an example. Compared to the 35 mm film you’d find in a movie camera, each proportionally-equal image is going to have over 40 times more surface area to scan. At the four-micron minimum resolution specified by our government contract – equivalent to 6,350 pixels per inch – each 9-by-9 inch area weighs in at around 3.2 gigapixels. That’s nearly what you could fit on 400 Ultra High-Definition 4K TV screens.
Kurt Breish, lead engineer on the project for nextScan, recalled the daunting task of figuring out the right way to handle media whose format, by any standard of cameras and optics, was gigantic.
“Several years ago, we’d had requests from Israel for 70-millimeter scanners, which we assumed were for aerial film, and we’d delivered a handful of those devices. Internally, that also overlapped with a project to design a system that could capture entire sheets of microfiche in a single pass,” Breish says. “So we had a pretty good starting point – but then suddenly, our task was to devise a way of imaging an area thirteen times that size, and at a higher resolution as well.”
“Our first move was to take the most powerful camera we had, from our Eclipse microfilm scanner, and see how close it came to what we needed. Well, you can’t even back the camera up far enough to get the whole piece of film without sacrificing clarity, and at that focal length, we were getting between a quarter and a third of the required resolution. It was still a high-quality image – in fact, not much worse than what had been achieved by previous state-of-the-art machines that were a lot more complicated. But the goal was to get about triple that resolution: not just an image that was legible to a human viewer, but one that could serve as a permanent historical record, comparable to the original film.”
Getting down to the four-micron level of detail meant the film would have to be scanned in multiple overlapping segments – the question was whether to do that by making multiple passes with a single camera, or by using multiple cameras to capture the whole width of the film at once. Both ways had their advantages and drawbacks, but ultimately it was decided to use the latter option, with an array of four line-scan cameras in fixed positions. That decision was based partly on observations about the machine that the new device would be replacing, and partly on the nextScan engineers’ previous experience with microfilm.
“It may seem counterintuitive, but in this case, using a single camera would’ve made our new scanner significantly more complicated. If you can’t capture the entire width in one pass, that means you have to move either the camera or the film laterally. That means another set of motors, and additional calibration to ensure the scan areas are aligned correctly,” Breish said. “And of course, all that reversing will slow down the scan speed by four to eight times, depending on whether or not you’re capturing during the ‘backswing’ – another added layer of complexity.”
The previous aerial film scanners used by one government agency, which were approaching 20 years old, employed the single-camera approach by necessity, due partly to the limitations of storage technology of that era. Capturing a full roll of the massive film took anywhere from 24 to 32 hours, in a process that required the operator to manually advance the film after each segment was complete. That made a convincing case for the multi-camera array and automation in scanning, Breish said.
“There are two types of cameras in our line of work: an area array sensor, which takes pictures like a normal camera that you or I would use at home; and a line-scan sensor, which continually captures single-pixel ‘slices’ of whatever is moving past it. From our past work with microfilm, we knew that line-scan was the superior way of digitally capturing roll film, not only because of its speed, but because it was continuous – you weren’t left needing to line up the edge of one picture you took with the start of the next. Line-scan was also a necessity for this project because the size of the media and the required resolution were just too much for any current area-array camera to handle, except maybe an experimental one in a laboratory somewhere.”
“Well, if you’re using line-scan capture, and you have to stop and backtrack for multiple passes, you lose the advantage of a continuous line-scan. It becomes more like a series of area scans limited by the length of film you can capture in each pass. So we were committed to solving that problem with an overlapping camera array.”
Since no single camera is capturing the full width of the film in either approach, we’re still left with the challenge of stitching the parallel strips together into a complete image without leaving visible seams. With previous machines, that had to be done manually by a human operator, which more than doubled the amount of time it took to obtain a perfect image. More recent efforts in the field have attempted to use software to detect the overlapping edges from multiple scans; however, the complex nature of the original images often left the recognition algorithms struggling to make a match.
“The stitching is critical; it has to be exact,” Breish said. “We heard of another company using four area cameras, and then the software tried to align the images. But it couldn’t always do it without any reference points. So if you have any cloud cover in the photo, for instance, it might not be able to figure out where the overlap should be.”
“We focused on the simplest solution to the problem – we used basic mechanical alignment. For a 4-micron resolution, by definition, you need to have your scans aligned within 4 microns, but you really want to be closer to one micron in order to do the stitching reliably. We positioned the cameras using calibration film that could show movement on the level of two-tenths of a micron. We were able to get them aligned within 0.7 microns, and then we locked it all in place permanently. So, the software doesn’t have to interpret anything; it has the same overlap every time, and it can stitch the images continuously.”
A side effect of that simpler-is-better approach was that it also eliminated headaches associated with keeping the machines calibrated. Doreen Breish, Kurt’s wife and the project liaison recalls: “With software-based alignment, it’s very important that the calibration stays intact – but if you have a camera that’s able to move, then by definition it’s possible to upset that calibration. Conceivably, since we’re dealing with precision down to the micron level, even moving the machine to the other side of the room could affect it. And if you have to do a full recalibration, that typically needs a visit from a technician, and it starts to get expensive very quickly. It was a welcome coincidence that we were able to reduce the need for that.”
When nextScan shipped the first of its finished devices, dubbed Apex by nextScan, a team of engineers went along to make sure everything went as planned. As it turned out, the machine’s original factory calibration stayed in place even after a 2,400 mile trip to Virginia by truck. While they were there, the team tested it by rolling the Apex around on casters, and still found no trouble – proving fixed mechanical alignment was the way to go after all.
The final major technical challenge to overcome was handling the firehose of raw data coming from four high-resolution cameras at once. At full uncompressed file size – which government agencies and others want for the purpose of historical preservation – the output from a continuous scan is more than a gigabyte of data per second.
This is the place where modern computing technology really made it possible to do things that simply wouldn’t have worked if the same project had been attempted 10 years ago. The Apex uses a 25-gigabit Ethernet link to transfer the camera data smoothly – this wasn’t developed until 2015, although parallel 10 Gb cables may have provided an alternative, slightly more complicated workaround.
The real breakthroughs, though, came in storage and CPU power, where older standards wouldn’t have been able to keep up. Mechanical hard drives, especially the larger ones used for mass storage, tended to have write speeds topping out at about a tenth of what comes out of the Apex’s cameras. That would have required a truly massive array of drives to keep up with the data flow without slowing the scan down. While solid-state drives are still too costly to be practical for storage on such a scale, newer mass storage drives offer speed that can better keep pace with the extreme data output. Even so, it still takes 24 drives configured in a RAID-5 array to capture it all in real-time. Similarly, if the client wants to use any type of image compression, that will max out all 64 threads even on high-end CPUs, and those processors have only been available for a relatively short time as well.
“What we found over the course of this project was that the previous machines designed for converting aerial film were not ‘old’ or ‘bad’ – they were actually quite amazing accomplishments with the technology of the time. There was just no way that you could do what was being asked with the technology that existed then,” Doreen Breish said. “It really impressed upon us how incredible the cameras on those planes were, when it’s taken until fifty, sixty years later for computers to finally catch up enough to deal with them, and even now it’s still hard.”
And then, there’s the issue of storing and accessing the digital images in the long term. The Apex can accommodate the data from a week’s worth of film (five eight-hour shifts) in its internal temporary storage, but that eventually needs to be offloaded to a more permanent data repository. Even at that point, it’s still a challenge to deal with. Whether before or after being transferred, the full-length “ribbon” files of a full roll of film need to be dissected into individual images, which is easier said than done.
“A big problem is that there’s not a lot of software that can handle this size of image,” Kurt Breish explained. “Some of these individual images are three feet long, and they’ll need 256 gigabytes of RAM and several minutes of loading time. There aren’t many computers built to handle that, and even less software designed for it, so that was another major consideration when developing our device.”
One of the nice things we discovered on our foray into aerial film is that it tends to be well-preserved compared to the media we often encounter in our microfilm projects. A lot of film and fiche at, say, a county clerk’s office or a private business ends up being stored in the basement or in a closet because it’s out of the way – without a lot of thought given to the conditions. But places with aerial film repositories tend to be larger organizations that take pretty good care of it in temperature-controlled environments. That’s fortunate because film deterioration and the dreaded “vinegar syndrome” are especially devastating when it comes to images of the age we find in these collections.
In any case, though, physical film won’t last forever no matter how well you take care of it, which is why it’s a race against time to get as much of it digitized as possible. When it comes to subjects like erosion, deforestation, or glacier recession, researchers want as many historical images as possible for comparison – and the ones we have now are the only ones we’re ever going to get.
For us, the Apex project has been a long and interesting journey – one in which we’ve repeatedly watched the work of old analog devices go up against the best that modern digital technology has to offer, often with humbling results for us modernists. One of the more telling moments came when we were testing out our working prototype on a roll of 9-inch aerial film that had timestamps alongside the images. We found that even using the best CPUs and storage devices that the 2020s have to offer, and pushing them right to their limits, we were only scanning the film about a third as fast as the original airplanes had been capturing it. Even today, when more aerial observation is being done than ever, and when thousands of digital photos from satellites can cover the entire globe on a near-daily basis – it turns out there’s still nothing that can capture as much information as fast as film can!