Launch sequence failure
|
The Apollo 17 countdown had no significant problems until 30 seconds before launch, when the automatic launch sequencer called a halt. The resulting delay was the only one in the Apollo program caused by a hardware failure.
According to the New York Times, "The 2 hours 40 minutes between the automatic halt in the counting and the blast-off produced some of the most feverish trouble-shooting in the history of the Apollo program. It involved scores of engineers at half a dozen or more major facilities, mountains of technical documents, and repeated tests of quickly devised new, procedures to guard against dangerous oversights."[1]
While Apollo 17's problem was less urgent than, say, Apollo 13's, pressing forward to a successful launch after a last-minute cutoff was still an impressive accomplishment.
[edit] Terminal Countdown Sequencer (TCS)
The final operations before launch were handled automatically by the Terminal Countdown Sequencer (TCS), because there were too many critical steps to be taken at critical time intervals for humans to handle the task.[2]
The TCS went into automatic operation when the hours-long countdown had reached T-187, when there were just 3 minutes 7 seconds to go.
From then on, every new step in the countdown was to be taken and verified automatically. The sequencer, manufactured by General Electric, would initiate the proper action, make sure the action had been taken, and go on to the next step in a precise sequence. If something went wrong, the sequencer was designed to stop the countdown automatically. It did just that.[1]
[edit] S-IVB liquid oxygen tank pressurization
At T-167, the “S-IVB LOX Tank Pressurization” command should have been sent to the mobile launcher, to initiate the following actions:[3]
- Close S-IVB LOX tank vent
- Open S-IVB LOX tank pressurization valve
- Enable “S-IVB LOX tank pressurized” signal
The TCS logic in the firing room generated the command to pressurize the S-IVB LOX tank, but it was not actually sent to the launch pad three miles away. (Later investigation identified a failed diode, one of 1,827 in the logic circuitry of the TCS. Excessive reverse current leakage through the partially shorted diode, which had been in service for six years, caused intermittent operation. Testing of 2,196 similar diodes found 8 additional failures.[3])
R. D. Carlson, a McDonnell-Douglas engineer manning one of the firing room consoles monitoring the S-IVB stage his company had built, saw that the LOX tank pressurization was not taking place automatically, as it was supposed to. Immediately he flipped a switch to start it manually instead.[2]
[edit] Automatic cutoff
The manual override activated actions 1 and 2 but not 3. When the tank was fully pressurized, a sensor raised the "LOX Tank Minimum Low Pressure OK" signal, but the “S-IVB LOX tank pressurized” signal was not passed on. As a result, the "S-IVB Ready for Launch" signal was never generated, which prevented swing arm retraction at T-30. This caused the cutoff command which initiated the countdown hold.[3]
So, even though the tank was up to pressure and everything was ready, the sequencer prevented the launch. Launch Director Walter Kapryan explained, "The problem was that since the TCS did output the command, the logic circuitry said that we really didn't complete all of the launch prep for the S-IVB stage. We have an interlock in our countdown circuitry if this has not occurred, and that is the reason for the cutoff."[4]
With the cutoff command the sequencer spewed out printed data to show technicians what had happened. The proper signal from the sequencer to the third-stage tank had simply not been sent.[2]
The printed data showed something else. A second signal on a spare channel, once used for a vital function but no longer, also had not been transmitted. This double failure gave the engineers confidence that they had an isolated, not a far-reaching problem, to deal with. Why so? A single device — a so-called "relay driver module" — was responsible for initiating both steps. It seemed almost certain something was wrong with this device and this device alone.[1]
[edit] Options
Now flight controllers had three options:
- Develop and test a workaround to permit the launch before the close of the current window in 3 1/2 hours.
- Scrub the launch and start 24 hour turnaround
- Scrub the launch until the next launch window a month later, when the sunlight was again at the correct angle.
With the vehicle, crew, and support staff all ready to go, the preferred option was to identify the problem, fix it, and press on with the launch if possible.
According to astronaut Jack Schmitt, "The one thing that you don’t want to do is have a aborted launch on a launch pad and have to recycle and come back a month later and go through another month of simulator training. You’re ready to go when you’re ready to go. And I think everybody felt that way."[5]
According to Kapryan, "When we were about an hour and fifteen minutes into the troubleshooting, we knew that we would not be able to recycle for tomorrow (or for today, rather). We never did get to the point where we gave up and say, it looks like we're going to have to quit. We knew we had quite a bit of time left, so we just worked the problem and didn't think about scrubbing."[4]
"It didn't take us very long to determine that we should bypass this command failure and go through the pressurization, manually, and go through the rest of the countdown, except we weren't completely certain that the final 30 seconds would all work properly. We did have a problem: How did we know that the last 30 seconds would work properly? How did we know that once we started igniting the 5 first-stage engines that perhaps we would get a cutoff on one of them, which we wanted to avoid at all costs."[4]
[edit] Troubleshooting
"We performed the operation of installing the jumper which we were able to do in the firing room. We have a preplanned design where we can go in with banana plugs and put in jumpers to jumper any point in the circuitry that we desire to without having to go out to the mobile launcher, and that's what we did. The same jumper was installed in the breadboard at Huntsville, and the sequence was run through several times on the bread board, and every time we had a successful run."
There was one big worry in the minds of the engineers. The erratic 'relay driver module' was designed to initiate not just two signals but six. Three had been properly initiated. Two had not, but a solution for those was at hand. But what about the sixth — the initiation of a "guidance reference alert" at just 22 seconds before ignition?[1] "Since we shut off at T-minus 30 seconds we didn't get down to this T-minus 22 command. We thought that perhaps we might not have that command and we wanted to make sure that by not having it we were still okay."
Fortunately, the engineers poring over the manuals soon found that failure of automatic signal to the guidance would not, under the normal computer program, cause a halt in the countdown as it had when the signal did not get to the third-stage tank. There was no need to use a jumper for the guidance signal. "Our logic told us that we could lose that and it would not interrupt the sequence whatsoever. We went back to the bread board in Huntsville to demonstrate that. We actually cut that command off and ran through the sequence again with the jumper and everything worked fine."
According to the New York Times, "The testing at the space agency's Marshall Space Flight Center in Huntsville, Alabama was conducted in a so-called 'breadboard' facility. This is a laboratory where exact duplicates of many of the critical electronic and other components of a Saturn rocket can be hooked together and operated without having to be assembled inside a rocket itself. The tests were described in an interview with Fletcher Kurtz, manager of Huntsville's Mission Operations Office. Several breadboard tests were run. A jumper was attached to the duplicate sequencer while the signal for third-tank pressurization was set manually. And when the simulated countdown reached that point, the computer was indeed fooled and made no move to halt the countdown. The engineers, under the direction of James Bramlet, in charge of the laboratory, also went so far as to simulate a failure of the signal to the guidance. As predicted in the analysis, this did not cause a shutdown. It was not a critical failure."[1]
[edit] Launch
Kapryan concluded, "We had every assurance that the failure we had was in no way connected with engine start circuit, which is the one that personally gave me the greatest concern. Once we were satisfied that we had no problem in that area, we picked up the count and went on our merry way."
The automatic hold command, issued at T-30 seconds, lasted 1 hour 5 minutes 11 seconds. The countdown was recycled to T-22 minutes, but was held again at T-8 minutes to resolve the sequencer corrective action. This hold lasted 1 hour 13 minutes 19 seconds. The countdown was then picked up at T-8 minutes and proceeded smoothly to launch. The delays totaled 2 hours 40 minutes.
[edit] Crew
"The capsule communicator and launch operations manager Paul Donally talked to [the astronauts], kept them up to date on the status of our evaluation and troubleshooting. They took it quite well, and just took advantage of the time to rest as much as they could."
Astronaut Jack Schmitt later said, "Right at thirty seconds, [Launch director (Clarence)] Skip Chauvin came over the line and said we have a hold. I think Gene was more concerned than the rest of us, because none of us knew whether everything in the spacecraft and in the rocket knew we were in a hold. But we went through that thirty-second period and it was quiet for a few minutes. Then Chauvin came back on the line and said, "We have a problem with the launch computer. It's not a major problem. We're going to fix it and when we have it fixed, we'll recycle." I think it was eight minutes for a planned hold and then go through it again. That is exactly what happened. At that point I felt very comfortable. I'd worked with Skip in many chamber tests and things like that, so we knew him very well, and from the sound of his voice, it didn't sound like anything that wasn't going to be fixed. So I fell asleep. Anytime you put fans humming or a little bit of vibration, things like that, I can go to sleep. There's no problem. So I got an hour or so dozing sleep while we were waiting for that problem to be fixed."[5]
References
- ↑ 1.0 1.1 1.2 1.3 1.4 Witkin, Richard (December 7, 1972), "Flight is Halted by a 'Sequencer'", New York Times, p. 60. Retrieved September 9, 2007, from ProQuest Historical Newspapers The New York Times (1851 - 2004) database. (Document ID: 90731417)
- ↑ 2.0 2.1 2.2 Witkin, Richard (December 8, 1972), "Experts Tricked the Computer", New York Times, p. 30. Retrieved September 9, 2007, from ProQuest Historical Newspapers The New York Times (1851 - 2004) database. (Document ID: 79483386)
- ↑ 3.0 3.1 3.2 Apollo 17 Saturn V Flight Evaluation, Section 3.0 Launch Operations
- ↑ 4.0 4.1 4.2 Walter Kapryan, Director of Launch Operations, Kennedy Space Center, Apollo 17 post-launch press conference, 12/7/72 00:37 CST PC 12 A/1
- ↑ 5.0 5.1 Harrison Schmitt, Johnson Space Center Oral History Project, Oral History Transcript: Harrison H. Schmitt 16 March 2000, NASA JSC History Collection, 16 March 2000