Voyager 1 was in Crisis in Interstellar Space. NASA Wouldn’t Give Up.

By Joel Achenbach / The Washington Post

11:44 JST, May 11, 2024

For the past six months a team of engineers at NASA’s Jet Propulsion Laboratory has been trying to fix a glitchy computer. Three things make the repair job challenging:

The computer is highly customized and unlike anything on the market today.

It was built in the 1970s.

And it is 15 billion miles away.

The computer is on Voyager 1, the most distant human-made spacecraft ever launched. Far beyond the orbit of Pluto, it is riding point for all humanity as it hurtles through interstellar space.

But on Nov. 14, Voyager 1 suddenly stopped sending any data back to Earth. While it remained in radio contact, the transmission had, as NASA engineers put it, “flatlined.” So began the greatest crisis in the history of the fabled Voyager program.

Voyager 1 and its twin, Voyager 2, launched in 1977 and in the years that followed obtained stunning close-up images of Jupiter and Saturn. Voyager 2 also flew by Uranus and Neptune and is the only spacecraft to have visited those ice giants. The Voyagers blew past the heliopause, where the solar wind abates and interstellar space begins, and continued to send back science data about particles and magnetic fields in a realm never before visited.

The two Voyagers are powered by the radioactive decay of plutonium-238, and in the near future that power source will be too feeble to keep the spacecraft warm and functioning. But for now, they have operational scientific instruments that are sending back otherwise unobtainable data on the composition of space beyond the heliopause.

Fixing Voyager 1 quickly became a priority for NASA, and especially for Jeffrey Mellstrom, who has been at JPL in Pasadena for 35 years and is the chief engineer in the astronomy and physics directorate.

Mellstrom took on the challenge even as he planned for retirement in the spring. In January, Mellstrom told a colleague, “The one thing I’m going to regret is if I retire before we solve Voyager 1’s problem.”

– – –

Like kicking a vending machine

After initial attempts to resolve the issue went nowhere, JPL leadership created a “tiger team” made of a multigenerational crew of engineers, some of them veterans of the lab and some born long after the Voyagers launched.

“We didn’t know how to solve this in the beginning because we didn’t know what’s wrong,” said Mellstrom, the team’s leader.

Voyager 1 has three computers. One is the attitude and articulation control system, which makes sure the spacecraft is pointed in the right direction. Another is the command control system, which handles the commands coming from Earth. The third is the flight data subsystem, which takes science and engineering data and packages it for transmission home.

Something had gone wrong somewhere in that trio of computers. Maybe a “cosmic ray” – a particle from deep space – had smashed into a computer chip. Or maybe a piece of hardware just got so old it ceased to work.

“All we had was incoherent data, garbled data,” said Suzanne Dodd, the Voyager project manager since 2010. Dodd has been at JPL for four decades, and in her early years she wrote computer code for Voyager 2’s encounters with Uranus and Neptune. She vividly remembers that first close-up look of Neptune and an image of the ice giant with its huge moon Triton in the background.

“We didn’t know what part of the spacecraft was involved with this,” Dodd said.

So they poked it. They sent commands to Voyager 1, trying to jolt it back to coherence. The team had a list of potential failures and figured that one of the commands might have the equivalent effect of kicking a vending machine.

Here is where the troubleshooting encountered an inviolable obstacle: the speed of light. Even at 186,000 miles per second, a command sent to Voyager 1 would take 22½ hours to arrive. Then the engineers would have to wait another 22½ hours for the spacecraft to send a response.

The planet Earth is kind of a pain, too, because it spins inconveniently on its axis and moves restlessly around the sun. To communicate with distant spacecraft, NASA relies on the Deep Space Network, three arrays of huge radio telescopes in California, Spain and Australia. The idea is that, regardless of Earth’s movement, at least one array can be pointed toward a spacecraft at almost any time.

The tiger team developed a pattern of sending a command on a Friday and waiting for the return signal on Sunday. Some dark days and weeks followed.

“None of those commands that we sent were able to make any discernible difference whatsoever,” said David Cummings, an advanced flight software designer and developer.

In late February, the team sent a series of commands to prod the flight data subsystem to place software in each of 10 different “data modes.” The team waited, hoping for a breakthrough. After two days, Voyager responded – still without data. Engineer Greg Chin circulated a technical chart and summarized the situation: “So, at this time, no joy.”

“It was unbelievably depressing,” Cummings said. “Luckily the story doesn’t end there.”

– – –

Cracking the code

Just a day after the “no joy” email, the team felt a surge of optimism.

JPL has specialists in radio transmissions, and they noticed that in some “modes” the return signal from Voyager 1 had been modulated in a pattern consistent with the flight subsystem computer producing data, though not in any normal format. The modulation suggested that the processor was functioning and supported the team’s conjecture that some of the memory had been corrupted.

“That was huge,” Cummings said. “The processor was not dead.”

Painstakingly, the team at last tracked down the origin of the problem: a bad memory chip holding one bit – the smallest unit of binary data – for each of 256 contiguous words of memory.

The flight data subsystem was built with 8K memory, or more exactly 8,192 bytes. (A byte is eight bits, and a modern smartphone has something like 6G memory, or 6 billion bytes.)

The engineers came up with a plan: They would move the software to different parts of the flight data subsystem memory. Unfortunately they couldn’t just move the 256 words in a single batch, because there was no place roomy enough for all of it. They had to break it down into pieces. And they’d have to proofread everything. It was tedious, error-prone work.

Cummings called a young JPL flight software engineer named Armen Arslanian: “Do you want to help me relocate Voyager code?”

Arslanian was the right person for the job. Just six years out of college, he knew how to write code for spacecraft, and he knew how to deal with “assembly language,” the coding that underlies the common languages used by programmers today. That’s the language of Voyager’s 1970s-era computers.

“I ended up needing that skill,” Arslanian said.

The JPL teams had documentation from the 1970s describing the function of the software, but often the descriptions were contingent on other information that could not be found. The team also lacked the tools to verify their coding. They had to do everything essentially by hand. It wasn’t like trying to find a needle in a haystack so much as like trying to examine every piece of hay for possible flaws.

The team prioritized the software for the engineering data so that they could fully restore communication with the spacecraft. If that worked, they could fix the science data later.

On April 18, the team sent a package of commands to the spacecraft and then waited. Two days later the spacecraft sent back the first intelligible engineering data in more than five months.

There is more work to be done, but the end is in sight. The engineers are still working on transferring the code that controls the scientific data. But they know how to do this. They found the problem, figured out the workaround and are just grinding through the code transfer.

Mellstrom and Dodd are fully confident that Voyager 1 has been saved. Mellstrom said he can retire without regret.

“The spacecraft is working,” Dodd said. “Go Voyager!”