When I went to work at Number Nine Visual Technology back in 1995, the PC industry was shifting away from the ISA bus to PCI. The PCI interface for the Imagine series of boards was designed in house and seemed to work fine in the imagine 1 & 2 series of boards. When we started development for Imagine 3 (Ticket to Ride, T2R), #9 purchased a quickturn machine.
The thing about Quickturns is that they were large boxes about the size of a washing machine that housed huge boards with many flex 10K FPGAs on them. This meant that you could astoundingly run your ASIC in a real system at about 1/20th the speed of the final design. This meant that all the timing paths were very exaggerated where signals could travel from FPGA to FPGA or even board to board. At first everything seemed to run fine, then we started getting failures which were narrowed down to the PCI bus. What happened is the signals were not properly synchronized to the control state machines and a metastable signal influencing the next state of the state machine would be seen as two different values, thus the state machine would end up skipping states or in invalid states.
This bug had been in production, lurking for years over two designs. We got lucky.