Mac Pro Graphics Corruption
The contents on this page pertain to forum discussions about Mac Pro (early 2008) systems with ATI Radeon HD 2600 graphics cards and random episodes of screen corruption.
As of March 10, 2008 -- firmware update now available from Apple!
This particular system ran well for 3 weeks before first GPU problems appeared.
January 16, 2008
Acquired system from local Apple Store -- stock config
2 x 2.8 GHz Quad-Core Intel Xeon
320 GB Seagate hard drive
2 x 1 GB RAM
OS X 10.5.1 pre-installed
January 29, 2008
Installed 320 GB Seagate hard drive in Bay 2
Installed 4 x 2 GB OWC qualified RAM for 10 GB total RAM
A1 / B1 OWC 2GB / OWC 2GB
A2 / B2 OWC 2GB / OWC 2GB
A3 / B3 Apple 1GB / __________
A4 / B4 Apple 1GB / __________
February 9, 2008
Complete corruption.
Background processes continued, but all keyboard and mouse input unresponsive.
Required hard shutdown.
Didn’t have camera handy, but here is portion of system.log showing GPU hex dump.
Feb 9 20:26:34 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 9 20:26:34 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 9 20:26:34 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 9 20:26:34 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 9 20:26:34 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
--- 500+ lines ---
Feb 9 20:26:43 Mac-Pro kernel[0]: 0x10645f0a
Feb 9 20:26:43 Mac-Pro kernel[0]: ** GPU Debug Info End **
February 12, 2008
Complete corruption.
Background processes continued, but all keyboard and mouse input unresponsive.
Required hard shutdown.
Didn’t have camera handy, but here is portion of system.log showing GPU hex dump.
Feb 12 18:13:59 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 12 18:13:59 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 12 18:13:59 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 12 18:13:59 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 12 18:13:59 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
--- 500+ lines ---
Feb 12 18:14:08 Mac-Pro kernel[0]: 0x916b289a
Feb 12 18:14:08 Mac-Pro kernel[0]: ** GPU Debug Info End **
Corrective Measures Taken
SMC reset
Many times while swapping hardware during exercises below...
On ATI card,
Disconnected monitor cable from secondary port (top connector)
Attached it to primary port (bottom connector)
Zapped PRAM
Reformated disk0
Clean install OS from OEM discs (not retail discs)
Update 10.5.2, Leopard Graphics Update 1.0, all other availble updates
Combos and manual downloads only -- repair permissions before/after every piece
Re-installed user apps from discs/dmgs
Forced pre-binding
February 14, 2008
Complete corruption.
Background processes continued, but all keyboard and mouse input unresponsive.
Required hard shutdown.
No log output.
Corrective Measures Taken
Made sure ATI card is well-seated in slot
Ran Apple Hardware Test (AHT) -- passed
Tested with only Apple stock memory (removed OWC 4 x 2GB) -- passed
Ran memtest (via Rember) on all 10GB RAM overnight
Max, loops, etc. -- passed
Created new user account
Copied (not migrate) user files, generated prefs from scratch, etc.
Re-attached peripherals one at a time
Monitored console for any suspicious activity -- none
February 16, 2008
Partial corruption.
Notice the block of corruption attached to the cursor -- this followed the cursor across the screen.
No lock-up. Moving windows caused some corrupt areas to be redrawn okay but not all.
Logging out presented a clear screen.
No log output.
Note -- moire pattern is due to camera angle when photographing screen.
February 18, 2008
Partial corruption.
No lock-up.
Launched Grab in order to try a screen capture -- but that instantly cleared the screen of corruption.
Next time will press command+shift+3 and try capturing it that way.
No log output.
February 19, 2008
Partial corruption -- partial lock-up -- full GPU hex dump
iTunes in foreground. Scrolling through cover view when a few small groups of red pixels appeared.
Cursor became spinning beachball -- could be moved by mouse input -- but no response to clicks.
Keyboard input was ignored -- could not screenshot, escape, etc.
The current song in iTunes continued playing, but activity ceased after that.
Hard drive could be heard writing or reading something during the event -- not just the initial dump.
Required hard shutdown.
System.log showed
Feb 19 01:10:12 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 19 01:10:12 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 19 01:10:12 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 19 01:10:12 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 19 01:10:12 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
--- 500+ lines ---
Feb 19 01:10:21 Mac-Pro kernel[0]: 0x64f1f509
Feb 19 01:10:21 Mac-Pro kernel[0]: ** GPU Debug Info End **
Corrective Measures Taken
It occurred to me there is a remote possibility that some ATI cards might have a physical defect. I thought it would be worth investigating to see if the physical act of having two connectors attached to the card might produce a beneficial stress that caused the electrical/signal behavior to become normal. If that were happening it wouldn't be obvious to the people who are driving dual monitors or driving a single monitor with dual inputs...
I don't have two monitors to drive, just a 20" ACD with its single DVI cable. So I attached a DVI-to-VGA adapter to the secondary port and let it dangle...
Also set Console preference so that it jumps to front anytime an event is logged. Doing this to see what actions affect the system -- launching apps, attaching peripherals, fast user switching, etc.
February 21, 2008
Partial corruption that progressed to full corruption -- partial lock-up -- full GPU hex dump
Safari in foreground. Opened a new window, typed in URL, pressed return.
Then, as soon as mouse began dragging cursor to scroll, the Safari window became corrupt.
However, at first only Safari was corrupt. I had time to grab the camera and take photos.
Notice the Console window in background on the right -- it is not corrupt.
The mouse retained motion control over the cursor, but clicks were ignored.
Keyboard input was also ignored.
Then suddenly, after ~ 2 minutes, additional corruption began to dance across the screen.
Eventually corruption accumulated over the entire screen.
Observed this during a period of ~ 8 minutes before chosing to force a hard shut down.
Appeared as if this behavior could have continued indefinitely.
System.log captured a GPU hex dump similar to others, but at the tail end was a loop of callback_client errors and user dropped events that corresponded in time to the accumulating corruption after the initial event -- perhaps this is a new clue? Or just an artifact of the GPU causing system chaos?...
Feb 21 15:53:07 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 21 15:53:07 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 21 15:53:07 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 21 15:53:07 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 21 15:53:08 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
--- actually it’s more like 850+ lines in these dumps ---
Feb 21 15:53:17 Mac-Pro kernel[0]: 0x5a4d50d7
Feb 21 15:53:17 Mac-Pro kernel[0]: ** GPU Debug Info End **
Feb 21 15:53:22 Mac-Pro fseventsd[47]: callback_client: ERROR: d2f_callback_rpc() => (ipc/send) timed out (268435460) for pid 243
Feb 21 15:53:22 Mac-Pro fseventsd[47]: client: 0x808a00 : USER DROPPED EVENTS!
Feb 21 15:53:23 Mac-Pro fseventsd[47]: client: 0x815800 : USER DROPPED EVENTS!
Feb 21 15:53:32 Mac-Pro fseventsd[47]: callback_client: ERROR: d2f_callback_rpc() => (ipc/send) timed out (268435460) for pid 243
Feb 21 15:53:42: --- last message repeated 1 time ---
Feb 21 15:53:42 Mac-Pro fseventsd[47]: client: 0x808a00 : USER DROPPED EVENTS!
Feb 21 15:53:43 Mac-Pro fseventsd[47]: client: 0x815800 : USER DROPPED EVENTS!
Feb 21 15:53:52 Mac-Pro fseventsd[47]: callback_client: ERROR: d2f_callback_rpc() => (ipc/send) timed out (268435460) for pid 243
Feb 21 15:54:02: --- last message repeated 1 time ---
Feb 21 15:54:02 Mac-Pro fseventsd[47]: client: 0x808a00 : USER DROPPED EVENTS!
Feb 21 15:54:03 Mac-Pro fseventsd[47]: client: 0x815800 : USER DROPPED EVENTS!
Feb 21 15:54:12 Mac-Pro fseventsd[47]: callback_client: ERROR: d2f_callback_rpc() => (ipc/send) timed out (268435460) for pid 243
Feb 21 15:54:22: --- last message repeated 1 time ---
The last sequence looped every 20 seconds until I shut it down.
Corrective Measures Taken
Disconnected keyboard from ACD’s USB hub and connected it directly to Mac Pro
Entirely disconnected ACD USB hub
Mouse still connected to keyboard
Perhaps this configuration will retain keyboard control if GPU dumps again? One can hope...
Reselected start-up disk in System Preferences -- someone mentioned ‘give it a try’ -- it’s easy...
Deep cleaned cache using Yasu
Next day (Feb-22) -- zeroed out hard drives and restored from clones
February 25, 2008
Partial corruption -- partial lock-up -- full GPU hex dump
iWeb in foreground.
Swift diagonal motion of mouse to move cursor from upper right screen to lower left dock.
Screen became partially corrupt with columns of dancing b/w pixels.
Mouse maintained control over cursor location, but clicks ignored. Keyboard input also ignored.
Waited ~ 10 minutes to see if corruption progressed -- it did not.
Required hard shutdown.
System.log showed
2/25/08 10:01:30 AM kernel Debug ** Device in slot: SLOT-1 **
2/25/08 10:01:30 AM kernel Debug ** GPU Debug Info Start **
2/25/08 10:01:30 AM kernel Debug 0x01019588 000090a2 00009588 00000000
2/25/08 10:01:30 AM kernel Debug 0x00000000 00000000 00000000 00000000
2/25/08 10:01:30 AM kernel Debug 0x0011a010 00000000 00000000 00a6106b
--- 850+ lines ---
2/25/08 10:01:39 AM kernel Debug 0x2f0bc7b4
2/25/08 10:01:40 AM kernel Debug ** GPU Debug Info End **
Corrective Measures Completed -- February 27, 2008 -- 4:56 PM
Read a post that persuaded me I have not put serious focus on isolating RAM to rule out potentially bad pairs.
May end up testing many configurations before this exercise is finished. To keep track of pairs and locations I'll use the following nomenclature
A1, A2, A3, A4 DIMM slots in top riser
B1, B2, B3, B4 DIMM slots in bottom riser
Apple 1GB Apple stock 1GB RAM delivered in A1
Apple 1GB' Apple stock 1GB RAM delivered in B1
OWC 2GBx OWC 2GB RAM, 1 of pair "x"
OWC 2GBx' OWC 2GB RAM, 2 of pair "x"
OWC 2GBy OWC 2GB RAM, 1 of pair "y"
OWC 2GBy' OWC 2GB RAM, 2 of pair "y"
__________ Empty
If that gets clunky I may revamp it later...
The stock configuration came as
A1 / B1 Apple 1GB / Apple 1GB'
A2 / B2 __________ / __________
A3 / B3 __________ / __________
A4 / B4 __________ / __________
I swapped slots within each pair back on Feb-14. Since then I have been running with
A1 / B1 OWC 2GBx' / OWC 2GBy'
A2 / B2 OWC 2GBx / OWC 2GBy
A3 / B3 Apple 1GB' / __________
A4 / B4 Apple 1GB / __________
In hindsight I can see that the "flavor" of corruption changed.
Before the swap, corruption was full screen.
After the swap, corruption has been partial.
Maybe a coincidence.
I started isolation testing by removing the Apple RAM
A1 / B1 OWC 2GBx' / OWC 2GBy'
A2 / B2 OWC 2GBx / OWC 2GBy
A3 / B3 __________ / __________
A4 / B4 __________ / __________
February 27, 2008 -- 10:56 PM
Partial corruption -- partial lock-up -- 6x GPU hex dump
Mail in foreground, reading a window.
Moved the mouse downward to scroll. Concurrent with this motion the cursor turned into a spinning beach ball with corrupt pixels.
Mouse maintained control over cursor location, but clicks ignored. Interestingly, moving the cursor outside of the Mail window caused the cursor to switch appearance to a b/w striped ball that did not spin.
Moving back into the Mail window reverted to the corrupt spinning beachball. Could switch back and forth repeatedly.
All keyboard input was ignored -- except when I tried command+tab in an effort to pop the Dock and switch applications. When I did that, the cursor turned into the pointing finger -- still with corrupt pixels.
After that, the cursor never reverted back to beachball or any other shape, and further keyboard input was ignored.
No other corruption appeared on the screen. Waited ~4 minutes to see if the corruption progressed -- it did not.
Forced shutdown with power button.
System.log recorded 6 back-to-back GPU dumps.
Each dump took 9-10 seconds and then ~11 seconds gap before the next dump commenced.
So for about 1 min 50 secs the GPU was dumping...
Feb 27 22:56:26 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 27 22:56:26 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 27 22:56:27 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 27 22:56:27 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 27 22:56:27 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
Feb 27 22:56:27 Mac-Pro kernel[0]: 0x00f8fc00 00000000 00000001 00008a12
--- 850+ lines ---
Feb 27 22:56:36 Mac-Pro kernel[0]: 511:0x00000000
Feb 27 22:56:36 Mac-Pro kernel[0]: 0x9ee0caf8
Feb 27 22:56:36 Mac-Pro kernel[0]: ** GPU Debug Info End **
Feb 27 22:56:47 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 27 22:56:47 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 27 22:56:47 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 27 22:56:47 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 27 22:56:47 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
Feb 27 22:56:47 Mac-Pro kernel[0]: 0x00f8fc00 00000000 00000001 00008a12
--- 850+ lines ---
Feb 27 22:56:56 Mac-Pro kernel[0]: 511:0x00000000
Feb 27 22:56:56 Mac-Pro kernel[0]: 0x9ee0cb30
Feb 27 22:56:56 Mac-Pro kernel[0]: ** GPU Debug Info End **
Feb 27 22:57:07 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 27 22:57:07 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 27 22:57:07 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 27 22:57:07 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 27 22:57:07 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
Feb 27 22:57:07 Mac-Pro kernel[0]: 0x00f8fc00 00000000 00000001 00008a12
--- 850+ lines ---
Feb 27 22:57:17 Mac-Pro kernel[0]: 511:0x00000000
Feb 27 22:57:17 Mac-Pro kernel[0]: 0x9ee0db70
Feb 27 22:57:17 Mac-Pro kernel[0]: ** GPU Debug Info End **
Feb 27 22:57:27 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 27 22:57:27 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 27 22:57:27 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 27 22:57:27 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 27 22:57:27 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
Feb 27 22:57:27 Mac-Pro kernel[0]: 0x00f8fc00 00000000 00000001 00008a12
--- 850+ lines ---
Feb 27 22:57:37 Mac-Pro kernel[0]: 511:0x00000000
Feb 27 22:57:37 Mac-Pro kernel[0]: 0x9ee0cbb0
Feb 27 22:57:37 Mac-Pro kernel[0]: ** GPU Debug Info End **
Feb 27 22:57:48 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 27 22:57:48 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 27 22:57:48 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 27 22:57:48 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 27 22:57:48 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
Feb 27 22:57:48 Mac-Pro kernel[0]: 0x00f8fc00 00000000 00000001 00008a12
--- 850+ lines ---
Feb 27 22:57:57 Mac-Pro kernel[0]: 511:0x00000000
Feb 27 22:57:57 Mac-Pro kernel[0]: 0x9ee0cbf0
Feb 27 22:57:57 Mac-Pro kernel[0]: ** GPU Debug Info End **
Feb 27 22:58:08 Mac-Pro kernel[0]: ** Device in slot: SLOT-1 **
Feb 27 22:58:08 Mac-Pro kernel[0]: ** GPU Debug Info Start **
Feb 27 22:58:08 Mac-Pro kernel[0]: 0x01019588 000090a2 00009588 00000000
Feb 27 22:58:08 Mac-Pro kernel[0]: 0x00000000 00000000 00000000 00000000
Feb 27 22:58:08 Mac-Pro kernel[0]: 0x0011a010 00000000 00000000 00a6106b
Feb 27 22:58:08 Mac-Pro kernel[0]: 0x00f8fc00 00000000 00000001 00008a12
--- 850+ lines ---
Feb 27 22:58:17 Mac-Pro kernel[0]: 511:0x00000000
Feb 27 22:58:17 Mac-Pro kernel[0]: 0x9ee0cc30
Feb 27 22:58:17 Mac-Pro kernel[0]: ** GPU Debug Info End **
Corrective Measures Completed -- February 28, 2008 -- 11:05 AM
Removed OWC pair "y"
A1 / B1 OWC 2GBx' / __________
A2 / B2 OWC 2GBx / __________
A3 / B3 __________ / __________
A4 / B4 __________ / __________
Ran memtest in single-user mode on OWC pair "x"
memtest all 16 -L
Memtest.log recorded
Memtest version 4.2 (64-bit)
Copyright (C) 2004 Charles Cazabon
Copyright (C) 2004, 2005, 2006 Tony Scaminaci (Macintosh port)
Licensed under the GNU General Public License version 2 only
Mac OS X 10.5.2 (9C31) running in single user mode
Memory Page Size: 4096
System has 8 Intel processor(s) with SSE
Requested memory: 3735MB (3916668928 bytes)
Available memory: 3735MB (3916668928 bytes)
NOTE: Memory request is too large, reducing to acceptable value...
Allocated memory: 3630MB (3807002176 bytes) at local address 0x0000000101000000
Attempting memory lock... locked successfully
Splitting allocated memory into 2 comparison buffers...
Buffer A: 1815MB (1903501088 bytes) starts at local address 0x0000000101000000
Buffer B: 1815MB (1903501088 bytes) starts at local address 0x0000000172751f20
Running 16 test sequences... (CTRL-C to quit)
Test sequence 1 of 16:
Running tests on entire 3630MB region...
Stuck Address : ok
Linear PRN : ok
Running comparison tests using 1815MB buffers...
Random Value : ok
Compare XOR : ok
Compare SUB : ok
Compare MUL : ok
Compare DIV : ok
Compare OR : ok
Compare AND : ok
Sequential Increment: ok
Solid Bits : ok
Block Sequential : ok
Checkerboard : ok
Bit Spread : ok
Bit Flip : ok
Walking Ones : ok
Walking Zeroes : ok
Test sequence 2 of 16: (identical results)
Test sequence 3 of 16: (identical results)
Test sequence 4 of 16: (identical results)
Test sequence 5 of 16: (identical results)
Test sequence 6 of 16: (identical results)
Test sequence 7 of 16: (identical results)
Test sequence 8 of 16: (identical results)
Test sequence 9 of 16: (hit CRTL-C during Walking Ones -- everything still ok to that point)
February 29, 2008 -- 3:23 PM
Full screen whiteout -- full lock-up -- no log output
TextEdit in foreground -- reading in a window.
Moved the mouse. Concurrent with this motion the entire screen went white.
No cursor. No mouse or keyboard response.
Safari had Pandora streaming in the background. A section of the currently playing track looped infinitely -- estimating this to be ~ 250 msec of content.
Forced shutdown with power button. Logs recorded no events in conjunction with this episode.
~15 minutes after the reboot, while editing a file, I moved the cursor to Console window.
Scrolled a little, paused, started scrolling again when suddenly the Console window became sprinkled with jittery corruption.
Grabbed camera, but by the time I squeezed off a shot the corruption had cleared itself except for a small amount beneath the blue scroll bar.
Maintained mouse and keyboard control. Again nothing recorded to logs. Continued on with work.
Corrective Measures Completed -- March 1, 2008 -- 7:49 AM
Swapped DIMM positions of RAM pair OWC 2 x 2GB RAM -- pair "x"
A1 / B1 OWC 2GBx / __________
A2 / B2 OWC 2GBx' / __________
A3 / B3 __________ / __________
A4 / B4 __________ / __________
Also swapped the ATI card from slot 1 to slot 2
Booted to AHT -- looped 3 times, extended -- all ok
Ran memtest from SU, max allocation, 8 complete sequences -- all passed
FYI -- 4 GB took 7 hours 35 mins for 8 sequences
Reasonably certain that this RAM pair (OWC 2GB “x”) is good.
March 10, 2008 -- firmware update
Available here ATI Radeon HD 2600 XT Firmware Update.
March 17, 2008 -- so far so good
More than 7 days uninterrupted uptime, including all 10GB RAM returned to their first configuration
A1 / B1 OWC 2GBx / OWC 2GBy
A2 / B2 OWC 2GBx’ / OWC 2GBy’
A3 / B3 Apple 1GB / __________
A4 / B4 Apple 1GB’ / __________
April 21 -- final entry on this issue
More than 40 days uninterrupted uptime. Haven’t rebooted or restarted. Will do a bunch of software updates this week so the uptime clock will restart. But certainly safe to say the ATI firmware update fixed this problem.
Apple get a pat on the back for finding and fixing.
