locked
Sync Anotations to NEF (Nikon Raw) files RRS feed

  • Question

  • I used the sync annotations function on Expression Media 2 recently and it seems to have corrupted a small number of NEF files. The files in question were all at some time edited with Nikon Capture. Expression Media will no longer read the files and both Nikon Capture and Photoshop report them as corrupted. I have backups for some but not all. Does anyone know of a way to recover these files?

    Thanks.
    Sunday, April 5, 2009 6:40 AM

Answers

  • I used the sync annotations function on Expression Media 2 recently and it seems to have corrupted a small number of NEF files. The files in question were all at some time edited with Nikon Capture. Expression Media will no longer read the files and both Nikon Capture and Photoshop report them as corrupted.
    I know that when you edit with Nikon Capture NX, it puts some processing instructions in the NEF file.  (This is exactly analogous to method by which Photoshop modifies XMP files when editing camera raw files.) 

    And yes, this appears to have some deliterious effect on other apps.  For example, files edited this way break the WIC codec -- a codec written by Nikon!  It's obvious that at Nikon, the left hand doesn't know that the right hand is doing.  There might be a way to recover the original raw image data, by treating the file as a TIFF file. 

    As I pointed out in my earlier post, send me a couple of corrupted files and I'll see what I can do.  You might also want to visit the Nikon forums for other advice.

    I should also mention that it's almost never a good idea to edit your camera raw image masters without having done a backup.  In fact I would make a stronger claim: never modify your raw image masters at all.  You never know if an editor has some bug that will corrupt your file -- a fact you found out the hard way.  I follow a workflow roughly derived from what's described in Peter Krogh's DAM Book.  Here's what I do:

    (1) copy images from the memory card into a capture directory
    (2) rename the raw files and move them into my archive area on my external hard drive
    (3) backup the archive to a separate hard drive, and to DVDs;
         bring the DVDs with me to work, where I store them in a binder
    (4) use EM to select images from the event that are worthy of post-processing
    (5) copy those images into my working directory, and edit them there

    At no point do I ever edit an archive copy of my raw images.  In fact I don't even sync annotations to raw files -- I export the metadata as an XMP side-car file.  Any editing I do on a file is on a copy of the original, in my working directory.  The original files are never touched at all, except for renaming when they are moved into my archive.

    If you want to lighter workflow, then at a minimum you should backup the capture images to a read-only medium such as DVD (and then store the DVDs somewhere off-site).  The rule-of-thumb for backups is: two backups, two different media, two different locations.

    -Matt
    Monday, April 6, 2009 6:27 PM

All replies

  • I used the sync annotations function on Expression Media 2 recently and it seems to have corrupted a small number of NEF files. The files in question were all at some time edited with Nikon Capture. Expression Media will no longer read the files and both Nikon Capture and Photoshop report them as corrupted. I have backups for some but not all. Does anyone know of a way to recover these files?


    Send me a couple of corrupted files and I'll take a look.  (Use You Send It to send me the files -- or just put them up on a website where I can get them.)

    -Matt
    Monday, April 6, 2009 6:14 PM
  • I used the sync annotations function on Expression Media 2 recently and it seems to have corrupted a small number of NEF files. The files in question were all at some time edited with Nikon Capture. Expression Media will no longer read the files and both Nikon Capture and Photoshop report them as corrupted.
    I know that when you edit with Nikon Capture NX, it puts some processing instructions in the NEF file.  (This is exactly analogous to method by which Photoshop modifies XMP files when editing camera raw files.) 

    And yes, this appears to have some deliterious effect on other apps.  For example, files edited this way break the WIC codec -- a codec written by Nikon!  It's obvious that at Nikon, the left hand doesn't know that the right hand is doing.  There might be a way to recover the original raw image data, by treating the file as a TIFF file. 

    As I pointed out in my earlier post, send me a couple of corrupted files and I'll see what I can do.  You might also want to visit the Nikon forums for other advice.

    I should also mention that it's almost never a good idea to edit your camera raw image masters without having done a backup.  In fact I would make a stronger claim: never modify your raw image masters at all.  You never know if an editor has some bug that will corrupt your file -- a fact you found out the hard way.  I follow a workflow roughly derived from what's described in Peter Krogh's DAM Book.  Here's what I do:

    (1) copy images from the memory card into a capture directory
    (2) rename the raw files and move them into my archive area on my external hard drive
    (3) backup the archive to a separate hard drive, and to DVDs;
         bring the DVDs with me to work, where I store them in a binder
    (4) use EM to select images from the event that are worthy of post-processing
    (5) copy those images into my working directory, and edit them there

    At no point do I ever edit an archive copy of my raw images.  In fact I don't even sync annotations to raw files -- I export the metadata as an XMP side-car file.  Any editing I do on a file is on a copy of the original, in my working directory.  The original files are never touched at all, except for renaming when they are moved into my archive.

    If you want to lighter workflow, then at a minimum you should backup the capture images to a read-only medium such as DVD (and then store the DVDs somewhere off-site).  The rule-of-thumb for backups is: two backups, two different media, two different locations.

    -Matt
    Monday, April 6, 2009 6:27 PM
  • Thanks for your thoughts Matt,

    I haven't worked out how to send you files but I have  done a lot more investigation and can note the following:

    1. At some stage all of the damaged files were annotated with Expression Media.

    2. All of the files are stored on a Linux based NAS.

    I copied the damaged files onto a blank CF card and tried a few recovery programs (CardRecovery, PC Inspector Smart Recovery and Disk Internals Flash Recovery) with very limited results. Flash recovery was able to find a handful of previews.

    I have backups of a few of the damaged files so then tried comparing them in a hex editor and found that in the damaged files, the first 294 bytes were mostly zeros. Using the hex editor, I copied the first 294 bytes from the good file to the bad and sucessfully fixed the file.

    Further investigation with Phil Harvey's EXIFtool shows that this part of the file contains the first 23 EXIF entries. The enties are mostly pointers to other data so they seem to contain generic values except for the one that points to the start of the image preview! Using this process I have worked out a way to restore the files but I estimate it will take many weeks to get through them all.

    I'm happy to provide you a few images if you want to investigate this yourself.

    Robert
    Thursday, April 30, 2009 11:44 AM
  • Hello Matt,

    I know this is an old thread and you may never see this, but....

    After accidently changing the setting from jpg to RAW on my D200 during a shoot, I downloaded all the images to my computer and then burned them to a DVD. Thought I had gotten all the images, so I erased the memory card.

    Then I remembered an image that should be there, plus a few others. I used Photo Recovery software to recover the images from the memory card. Unfortunately, many of the RAW files are now corrupt, although the recovered jpgs are fine. Possibly they were corrupted at the time of shooting, hence they didn't appear in the original download.

    I've tried opening them in Photoshop, Lightroom, and Nikon ViewNX. I can see the preview thumbnails in all the programs, including Windows file manager, and most of the images are full size. But when I try to open/edit them, I get the "unexpected end-of-file" error.

    Any help would be greatly appreciated, thank you!

    Wednesday, June 2, 2010 10:46 PM
  • After accidently changing the setting from jpg to RAW on my D200 during a shoot, I downloaded all the images to my computer and then burned them to a DVD. Thought I had gotten all the images, so I erased the memory card.

    I don't ever erase a memory card until the moment I actually put it in the camera to use it.  The card gets erased as a side-effect of formatting the card, which is the first thing I do when I put a memory card in the camera.

    I've tried opening them in Photoshop, Lightroom, and Nikon ViewNX. I can see the preview thumbnails in all the programs, including Windows file manager, and most of the images are full size. But when I try to open/edit them, I get the "unexpected end-of-file" error.


    If you were shooting with RAW+JPG, then you have the JPEG files, so at least you have something.  If you need to adjust the JPGs, then you can always use the Photoshop RAW plugin, or Lightroom, to edit the JPGs.

    If you don't have the JPGs, you can try usng an image preview extractor to extract the JPGs from the NEF raw files.

    I'm not sure why the RAW files became corrupted.  If you still have the card, I would have also done a sector-by-sector (if that term makes sense for non-hard drive media) image backup, such as is available from Acronis or in Win7.  Then you'd have to wait for software to become available that can really recover the RAW camera files.

    Another thing you could try is different image recovery software.  The good thing about an image backup (the term "image" is unfortunately over-loaded here), then you always mount the image file and then re-try the recovery when new or better software becomes available.

    -Matt

     

    Wednesday, July 14, 2010 8:24 PM