locked
FileIO guarantees? RRS feed

  • Question

  • There don't seem to be many guarantees of the FileIO API. For example, "move" on most OSes is a completely atomic operation; on HD-DVD, it seems to be "Delete target, then copy, then delete source".

    For the most part, I can code around this. There's just one problem:

    Is there any kind of reliable way to create/update a playlist in persistent storage? Rolling back isn't enough, as in the case of a corrupt playlist, the A1, at least, just crashes with an ugly error.

    The only way I can think of is to have an on-disc playlist that loads, checks/fixes corruption in the on-disc playlists, and loads them. This is limiting in that it's slow, and that on-disc means static; I can then never update the recovery system.

    I guess what I'm asking is whether there's something I haven't thought of, and whether any players provide reliable ways of doing this -- for instance, a "move" that maps directly to the POSIX "rename" syscall. It would be better to have a system that's reliable on one player than to have a system that can fail anywhere.
    Wednesday, November 21, 2007 6:43 PM

Answers

  •  

    >So, in pseudocode:

    >write/download (foo.xpl)
    >delete (VPLST001.XPL)
    >move (foo.xpl, VPLST001.XPL)

     

    It would be better to:

    download (foo.xpl)

    move(foo.xpl, VPLST###.XPL) where ### is 1 greater than the highest number in existing playlists in persistent storage.

     

    When the player starts up if the DISCID.DAT file can specify that it should check persistent storage first for the playlist. It will then check both the disk and persistent stoage then load playlist with the highest number. If the playlist is invalid (as would be the case for a corrupted playlist) then the player should display an error message, in which case the players troubleshooting instructions should tell the user to clear persistent storage. There is little chance of the playlist being corrupted or truncated during the move to persistent storage - if the power were cut before the move was complete then the file would not appear in the directory list since the directory entry would be the last thing written to the filesystem.

     

    I would recommend following the spec section 4.3.22.3 Update sequence of Advanced Content Playback.

    Thursday, November 22, 2007 1:35 AM

All replies

  • >For example, "move" on most OSes is a completely atomic operation; on HD-DVD, it seems to be "Delete target, then copy, then delete source".

     

    This is not correct. According to the specification FileIO.move works the same as FileIO.copy with overwrite == fase. After a successful copy the player will delete the source file. If the player fails to delete the source file it will delete the copied file.

    Are you handling all of the possible exceptions and checking errorInfo in the callback? If you are seeing a different behaviour from FileIO.move then please post sample code and I will investigate.

     

    >Is there any kind of reliable way to create/update a playlist in persistent storage?

     

    Would it work for you to download the playlist from the network and save it to persistent storage? This would enable you to update it anytime you like.

     

    >Rolling back isn't enough, as in the case of a corrupt playlist, the A1, at least, just crashes with an ugly error.
    >The only way I can think of is to have an on-disc playlist that loads, checks/fixes corruption in the on-disc playlists, and loads them. This is limiting in that it's slow, and that on-disc means static; I can then never update the recovery system.

    How is your playlist getting corrupted on disk?


    Wednesday, November 21, 2007 11:09 PM
  • > This is not correct. According to the specification FileIO.move works the same as FileIO.copy with overwrite == fase.


    Oh, you're right -- somehow, I was thinking it was overwrite == true.

    Which means that if I want to replace one file with another, I do have to delete the target file, then move the source file to the target location. If there is a failure, I will be left with the source file intact, but no target file.

    So, in pseudocode:

    write/download (foo.xpl)
    delete (VPLST001.XPL)
    move (foo.xpl, VPLST001.XPL)

    The spec does say that if the move fails, target file will be deleted. I suppose this can work if that behavior is guaranteed in ALL events -- for instance, if there is a power failure, etc.
    edit: No, actually, it doesn't say that. It says target file will be deleted if source can't. I'm beginning to despair finding a single atomic operation aside from remove or createDirectory.

    >
    Would it work for you to download the playlist from the network and save it to persistent storage?

    In this case, it will most likely be modified from script. New versions may be downloaded, but they would be templates, to be filled in by script.

    >
    How is your playlist getting corrupted on disk?

    My fault... not on-disc, on-storage. So, to rephrase that:

    The only way I can think of is to have an on-disc playlist that loads, checks/fixes corruption in the playlists in persistent storage, then loads them. (Thus, I can never update the recovery system.)
    Wednesday, November 21, 2007 11:26 PM
  •  

    >So, in pseudocode:

    >write/download (foo.xpl)
    >delete (VPLST001.XPL)
    >move (foo.xpl, VPLST001.XPL)

     

    It would be better to:

    download (foo.xpl)

    move(foo.xpl, VPLST###.XPL) where ### is 1 greater than the highest number in existing playlists in persistent storage.

     

    When the player starts up if the DISCID.DAT file can specify that it should check persistent storage first for the playlist. It will then check both the disk and persistent stoage then load playlist with the highest number. If the playlist is invalid (as would be the case for a corrupted playlist) then the player should display an error message, in which case the players troubleshooting instructions should tell the user to clear persistent storage. There is little chance of the playlist being corrupted or truncated during the move to persistent storage - if the power were cut before the move was complete then the file would not appear in the directory list since the directory entry would be the last thing written to the filesystem.

     

    I would recommend following the spec section 4.3.22.3 Update sequence of Advanced Content Playback.

    Thursday, November 22, 2007 1:35 AM
  •  Brandon Davis [MSFT] wrote:

    When the player starts up if the DISCID.DAT file can specify that it should check persistent storage first for the playlist. It will then check both the disk and persistent stoage then load playlist with the highest number. If the playlist is invalid (as would be the case for a corrupted playlist) then the player should display an error message, in which case the players troubleshooting instructions should tell the user to clear persistent storage.

    That is not particularly user-friendly, given how much users may be downloading.


    Right now, required persistent storage is 128 megs, but as I understand it, that's a minimum. The Xbox seems to be using 192 megs, and of course, HDiSim reports the entire hard disk partition as being available for persistent storage. That's a lot to re-download, if the user should ever have to clear it.

     Brandon Davis [MSFT] wrote:

    There is little chance of the playlist being corrupted or truncated during the move to persistent storage - if the power were cut before the move was complete then the file would not appear in the directory list since the directory entry would be the last thing written to the filesystem.

    Thanks, that's what I wanted to know. I'd assume that if all writes are similarly ordered, file removes are safe as well.


    (Overkill would be to save the initial playlist as VPLST999.XPL in pstorage, then download each update as VPLST###.XPL, where ### is one less than the current highest playlist, removing the old one when done.)


    One more question: Are copies likely to be the same? (I know the spec says a move is copy-then-delete. I'm more curious about actual implementations.) I can always copy-then-move, but that seems a waste.


    Although, this is not in the spec anywhere I can find. I know that some filesystems reorder disk writes, to improve performance.

     Brandon Davis [MSFT] wrote:

    I would recommend following the spec section 4.3.22.3 Update sequence of Advanced Content Playback.


    That isn't relevant -- while generated, all of these will be in persistent storage, and are intended to be booted automatically next time. This is for semi-permanent application updates, some of them automatic.
    Thursday, November 22, 2007 2:27 AM
  •  David Masover wrote:

    That is not particularly user-friendly, given how much users may be downloading.


    Right now, required persistent storage is 128 megs, but as I understand it, that's a minimum. The Xbox seems to be using 192 megs, and of course, HDiSim reports the entire hard disk partition as being available for persistent storage. That's a lot to re-download, if the user should ever have to clear it.

     

    The 128 megs is the total available for all discs. On each persistent storage device there will be a common directory and a subdirectory for each Provider ID (studio). Underneath each Provider ID directory there are subdirectories for each Content ID (disc) from that Provider ID. The specification requires that the player implement a persistent storage manager that will enable the user to browse/delete the directories in persistent storage. If the user needs to delete the data for a single disc they may do so without deleting anything that other discs have downloaded.

     

     David Masover wrote:

    One more question: Are copies likely to be the same? (I know the spec says a move is copy-then-delete. I'm more curious about actual implementations.) I can always copy-then-move, but that seems a waste.

     

    Copies will work the same. Doing a copy-then-move would be a waste.

     

     David Masover wrote:

    Although, this is not in the spec anywhere I can find. I know that some filesystems reorder disk writes, to improve performance.

     

    This is beyond the scope of the spec, and is entirely dependent upon the operating system and filesystem used.

    Monday, November 26, 2007 7:27 PM
  •  Brandon Davis [MSFT] wrote:

    The 128 megs is the total available for all discs. On each persistent storage device there will be a common directory and a subdirectory for each Provider ID (studio). Underneath each Provider ID directory there are subdirectories for each Content ID (disc) from that Provider ID. The specification requires that the player implement a persistent storage manager that will enable the user to browse/delete the directories in persistent storage. If the user needs to delete the data for a single disc they may do so without deleting anything that other discs have downloaded.

    But that's the limit of granularity for the user. That means if a particular disk was using a very large amount of data, or storing a particularly large amount of user preferences, being able to delete just that disc (leaving the rest alone) isn't very helpful.

     

     Brandon Davis [MSFT] wrote:

     David Masover wrote:

    Although, this is not in the spec anywhere I can find. I know that some filesystems reorder disk writes, to improve performance.

     

    This is beyond the scope of the spec, and is entirely dependent upon the operating system and filesystem used.


    That's fair.

    It also means there are no guarantees, or even particularly safe assumptions, so it's very difficult to design a robust system by only following the spec.
    Monday, November 26, 2007 9:48 PM