Engineering‎ > ‎Stories‎ > ‎

Oscillation Overthruster Now

September 21st, 2010

Once during the early days at my first company me and two other engineers spent an entire Sunday getting ready for a Monday morning customer visit. We had been working long hours for weeks and were spending the weekend trying to get a fully functional demo ready.  The two engineers working with me were the DSP software engineer and the systems engineer. My bit was mainly the outer loop microcontroller software but I also was the catch-all resource for the main controller board, outside of the DSP.

The unit we were working on was a 2MW hard switched inverter with a static bypass switch.The unit's function was to spend most of its time idling, bypassing the incoming utility line to the load via the static switch. When the input voltage dropped below a set level the inverter would kick on and support the two million Watt load (enough to power over 1000 residential homes). My company had a full product line of similar units but this particular design was a one of a kind custom system.

To test the operation of the unit we connected it to what we called the "sag generator". The sag generator was built out of some very large contactors and a three phase auto transformer. One set of contactors would connect in nominal 480VAC and a second set of contactors would apply the lower auto transformer voltage as long as you held the sag button. We had a remote pendant operator panel we used to control it. There were three buttons: one to pre-charge the system, one to produced the temporary "sag voltage", and an emergency off button.

I would like to say we were putting the finishing touches on the unit that afternoon but in reality we were still actively working on the design. Even with that, the day had gone rather well and we were on track to get out of there before dinner. It was going so well that one of the other engineers was feeling rather punchy and had gotten into quoting Buckaroo Banzai, complete with impersonations of John Lithgow's Lord John Whorfin character.

During the morning and early afternoon we had been powering the controls of the unit with a line cord instead of deriving the power from the main 480V input. Like I said we were still actively developing. The unit seemed to be hanging together so we decided to cut the line cord and switch the controls power supply to the bootstrap 480V transformer. Doing that can be a tricky step in the development of power electronics. The power switch gate drivers need to be designed to maintain a known safe state while their power supplies are coming up. The different timing between a bench supply, line cord, and line powered controls transformer can produce glitches that inadvertently pulse on the power switches, even when you intentionally design the circuit to handle it.

When we first turned on the unit with the line cord removed something odd happened. All the overhead lights in the building momentarily dimmed for about a half second before recovering. We all sat there in silence for a second or two then I was the first to say something. I offered up "maybe it was just the pre-charge transformer inrush." Since we had been there all day I think we were all suffering from hopeless optimism and we mutually agreed to pretend everything was probably fine.

We turned the unit off then took a quick look at all the controls to see if there was anything obviously broken. We didn't find anything out of the ordinary so we buttoned the cabinet back up and re-energized the unit. The second power up of the unit without the line cord seemed completely normal.

I was in charge of operating the sag generator. Using the pendant controller I could apply power and sag the unit, which would pulse the inverters on. I was standing over by the sag generator which was attached to the main 480V service on a pole about 20 feet away from the unit. The two other engineers were about five feet from the unit monitoring various signals on a couple of oscilloscopes.

Everybody gave me the thumbs up to attempt the first inverter operation since we had switched over the controls from the line cord to the bootstrap transformer. If this test worked we could all go home in time to watch the new episode of The Simpsons. The punchy engineer shouted one last "Oscillation Overthruster, NOW!" then I hit the sag button.

Looking back, me hitting the sag button reminded me of Wile E. Coyote depressing the dynamite plunger on his latest trap for the Road Runner followed by the swift, inevitable backfire. The inverter, with its several Farads of capacitance (not a typo to those EEs reading), erupted into a fireball of plasma and smoke. To this day it was the most powerful electrical explosion I have ever personally witnessed. The arc blast sent out a shock-wave large enough that twenty feet away I could feel the concussion in my chest. It felt like somebody had hurled a basket ball at ribs at point blank range.

The doors to the unit near the other engineers were shut but the explosion was still loud enough that our ears were ringing for about ten minutes. "Buckaroo" was the closest to the unit when it blew up and after the explosion he stumbled about ten feet way and lurched to a stop on a table. The system engineer ran over to him to make sure he wasn't hurt. Other than being disoriented and temporarily deaf he was ok. While all that was going on I was busy shutting down the power. 

Looking back it was a very good thing we were following the basic common sense safety rules when working on that type of equipment. Rule one is don't work alone. Rule two is keep a solid piece of iron (i.e. the closed door) and a safe distance between you and any potential arc blast that might occur. 

During the first power up of the unit when the lights had dimmed the controls had been damaged. Several of the gate driver circuits had been put in a half-powered, brown-out state. We had tested similar operating points during bench testing but not that exact condition. Half the power switches were glitched on and it took just a few milliseconds of inverter switching for a shoot-thru to occur and the devices became a dead short across the DC link caps. In that contest the capacitors always win over the silicon devices. The result is a molten power module followed by an arc explosion which gets feed by the input source until the upstream breaker blows.

To say we were demoralized after the explosion would be quite the understatement. We had already put in about 60 hours that week and the unit was shrouded in smoke with the customer showing up in less than16 hours.  We briefly discussed calling off the demo but decided it was worth an attempt to repair the unit. We took apart the cabinet and discovered that only one of the three phase modules had failed. A partial demo was better than nothing. After ripping out the blown phase we started retesting the remaining two modules. It was all back together and two thirds functional by 11:00PM and we called it a night.

The next day all three us had to sit there and listen to sarcastic comments from the customer all morning long and well into the afternoon. We deserved it as the project was already late and we only had a partially operation unit to show for our efforts. The system engineer got the worst of it but we all had to sit there and take it, laughing as much as we could bear at the customers disparaging jokes and comments.