Media syncing issues and possible data loss

Damien Elmes's Avatar

Damien Elmes

27 Sep, 2016 10:25 AM

UPDATE 5:

UPDATE 4:

UPDATE 3:

  • Some accounts could not be fully restored automatically.
  • There is a good chance we can recover your data if you can provide us with the log files Anki creates.
  • Please see this page to check on the status of your account, and to see how to provide the log files.

UPDATE 2:

  • Media syncing has been restored for the majority of affected accounts
  • Restored accounts will not sync deleted files for now - please see this article for more info.
  • We're still working on restoring data for some accounts
  • If you've recovered data from one of your devices, you can manually restore syncing - please see this article for more info.

UPDATE 1:

  • The majority of the lost files have now been restored, with 2/3rds of affected users no longer missing a single file.
  • It will take me up to another day to ensure it's safe to turn back on media syncing and to finish restoring what I can.
  • For those of you who've been able to restore data from another device, I understand you're eager to start syncing again, but I do need to keep syncing of the affected accounts offline for now until I can be sure no further data will be lost.

Thank you all for your patience, and I'll post back as soon as I have new info.

IN SHORT:

  • If syncing gives you an error pointing to the announcements page, you may have lost sound or image files on one or more devices due to an AnkiWeb issue.
  • The issue only affects media files - no card text or review data has been lost.
  • To prevent media files from being deleted on any other devices you own, media syncing has been temporarily disabled for you. We hope to restore functionality in the next 24 hours.
  • You can continue to sync - your card content and study progress will still be transferred, but you will receive an error at the end of the sync because media syncing is disabled, and no media files will be transferred to other devices.
  • It is safe to study, and you will not lose any more media files than have already been lost.
  • It is safe to add new media files to your cards.
  • Automatic backups do not contain media, so restoring from an automatic backup will not help.
  • To determine what has been lost and how to restore it, please see https://anki.tenderapp.com/kb/problems/recovering-from-the-ankiweb-...

Hi all,

I discovered a serious issue with the synchronisation service today that may have resulted in sound and image files in your collection being removed erroneously if you synced Anki in the last 24 hours.

The error occurred when syncing a device that needed to download media from AnkiWeb. Instead of the file data being returned correctly, an empty file was returned in certain cases. When subsequently syncing that device again, the empty file was treated as a removal, and the files were removed from other devices when you synced them afterwards.

To clarify, if the devices you synced in the last 24 hours:
- ...did not have any media changes, then you will not be affected - ...only uploaded media to AnkiWeb, then you will not be affected - ...downloaded media from AnkiWeb, then there is a chance you are affected

I have fixed the original issue that was causing the data loss, and will be making steps to ensure that it can not happen again. To prevent already deleted files from being deleted on more devices, I have temporarily configured AnkiWeb to return an error if your account may be affected by this data loss.

I will be attempting to restore as much data as I can from backups and cached files, and will edit this post when I have more information.

This should not have happened, and I'm very sorry for any inconvenience caused. If you have any questions or need help with making backups, please don't hesitate to contact us.

  1. 1 Posted by Jacob on 27 Sep, 2016 10:58 AM

    Jacob's Avatar

    Thanks for the update. However, I am still unable to download images onto the iPad app from Ankiweb. I see the images on AnkiWeb and on the Mac desktop version of Anki, but an error message continues to come up on the Ipad whenever I try synching that is directing me to the Announcements site on Anki Help. I have version 2.0.25 on my iPad. Was working fine until this morning -- please help!

  2. 2 Posted by huang.sundi on 27 Sep, 2016 11:04 AM

    huang.sundi's Avatar

    If there's a way to get a cached database file (with all the media) right before midnight, I would greatly appreciate it! Otherwise, I understand.

  3. 3 Posted by birdiesama on 27 Sep, 2016 11:08 AM

    birdiesama's Avatar

    Thanks for the notification. I was wondering if I accidentally deleted media files today since some medias were missing. Luckily I didn't sync one of my devices so I have a back up.

    Thanks for your hard work. :)

  4. 4 Posted by huang.sundi on 27 Sep, 2016 11:18 AM

    huang.sundi's Avatar

    "we'd appreciate hearing if restoring from the trash works for you."

    I restored from trash but I think the problem is that the cards containing the media files no longer point to the images. Like "pasterandom.png" is just doesn't show up (blue question mark instead) even if it's in my collection media.

  5. Support Staff 5 Posted by Damien Elmes on 27 Sep, 2016 11:25 AM

    Damien Elmes's Avatar

    Jacob: I have had to disable syncing for users that have deleted files waiting to sync, to protect against further data loss. I hope to have syncing working again within the next 24 hours, once I have been able to ensure it won't cause further issues.

    birdie: Glad you were able to restore your files.

    Sandy: Please edit a card that has a broken image icon, click the rightmost button and choose "edit HTML", and confirm the exact filename. For Anki to be able to show the image, the exact same filename must be in your collection.media folder. Is the file there? Are you able to display it in an image editor? Does restarting Anki help?

  6. 6 Posted by huang.sundi on 27 Sep, 2016 11:40 AM

    huang.sundi's Avatar

    Oh hm, these are things I should have checked during the night but alas I've long since deleted the files and I unwisely limited my local backups to just 5 so I can't retrieve them to see. Thanks, I appreciate the quick feedback.

    Media was retrievable though, and I think if I had been more patient and tinkered around more, the files likely would have shown up after a restart.

  7. 7 Posted by M. on 27 Sep, 2016 11:43 AM

    M.'s Avatar

    Hello!

    I'm sorry if this is a silly question, but I just started using Anki and I am not very used to it yet. So, I synced a lot today, and multiple errors came along with it, I am pretty sure that I am one of those who lost some material. Now, my question(sorry if it's obvious to you): Is it possible to know which cards were deleted in my collection? Because I have a really big one and don't remember all the cards I made... But I worked so hard to make them, last 2 weeks all I did was make these cards... So, is it possible to know which ones were lost so I could make them again?

    Thank you!

  8. 8 Posted by dp on 27 Sep, 2016 11:52 AM

    dp's Avatar

    Hi Damien,
    Similar to M's quesiton, but I clicked the delete unused media cards thinking it was an error on my part and have now lost a bit of data. I have the relevant backup with the saved cards but can you please point me in the right direction for how to restore these deleted cards without interfering with the reviews and other changes I have changed since the backup?

    Cheers

  9. 9 Posted by Cyndi on 27 Sep, 2016 12:06 PM

    Cyndi's Avatar

    Thanks for the notification.
    Seems like I downloaded my AnkiMobile at the worst time!
    I lost 180 pictures for my deck after syncing to my mobile device...
    Just wondering how long it might take to have everything fixed so that I can start using my Mobile app again?

    Also right now it is still not syncing properly onto the Web page from my desktop.
    I could not see the media I just added from desktop on the Web.

  10. 10 Posted by Soren Bjornstad on 27 Sep, 2016 12:17 PM

    Soren Bjornstad's Avatar

    M. & dp (and anyone else): This issue has not affected your actual cards, only your media. So no cards will actually be missing from your collection; if you have lost anything, the media will be missing from your cards. So assuming you know what media is supposed to be on those cards, you should be able to figure out how to restore them yourself.

    Cyndi: If you were having trouble before, syncing has probably been disabled for you in an attempt to limit the damage, so that would explain why it isn't working. I'm afraid we don't have a timeline right now, but it will be as soon as possible.

  11. 11 Posted by Robert Elliot P... on 27 Sep, 2016 12:21 PM

    Robert Elliot Pahel-Short's Avatar

    I first noticed this error yesterday, it seems as if the media is not syncing from the original device where the note was created. For instance on computer A (my laptop) where some of my media notecards were created, the flashcards show up just fine. But on my ankidroid app as well as on computer B (desktop), those same media notecards show up as blue error icons.

    This is the same situation on computer B (desktop), where media file notecards created on that machine show up fine but they show up as blue icons on computer A and ankidroid.

    I hope this helps with resolving the bug!

  12. 12 Posted by dp on 27 Sep, 2016 12:22 PM

    dp's Avatar

    @Soren

    Sorry I did not make myself clear. Upon having the sync issues I went check media -> delete cards on all my devices as that is what AnkiDroid suggested, stupidly, I have now deleted the cards with the missing media. Is there a way to restore those cards from a previous backup without losing the reviews I have completed since?

  13. 13 Posted by Soren Bjornstad on 27 Sep, 2016 12:27 PM

    Soren Bjornstad's Avatar

    @dp:

    check media -> delete cards on all my devices as that is what AnkiDroid suggested

    There is no such option that I'm aware of -- the "delete unused" option in the media check only deletes media that is not used on any cards. Unless you're saying that AnkiDroid has an option to delete cards that are missing media? (We don't develop AnkiDroid, so we're not too familiar with it.) You certainly wouldn't have been able to do that on the desktop or iOS versions, though, since the feature doesn't exist.

    If there really is an option like that on AD and you used it, you could restore the backup into a new profile, find the deleted cards, move them into a new deck, export them as an apkg, and import them into your main profile.

  14. 14 Posted by M. on 27 Sep, 2016 12:28 PM

    M.'s Avatar

    Thank you and sorry about my misunderstanding.

  15. 15 Posted by dp on 27 Sep, 2016 12:32 PM

    dp's Avatar

    @Soren,
    you are correct. Feel free to delete my comments relating to that issue if you think it will cause others confusion

  16. 16 Posted by Shane on 27 Sep, 2016 12:40 PM

    Shane's Avatar

    I was getting a error 502 on AnkiDroid while syncing. The app would sync and then error. I had synced yesterday from both my phone and my computer and added cards today. I regularly delete media that is not in use. So I did that today before syncing.

    After I restored the files from the recycle bin, the syncing errors stopped. I don't think I ever actually lost any data. I am not sure if that helps answer the question about restoring from trash because it is a bit different. However I don't seem to have lost anything.

  17. 17 Posted by huang.sundi on 27 Sep, 2016 12:41 PM

    huang.sundi's Avatar

    Just to get some clarity on how much work I might need to redo:

    How likely do you think it is that you guys will be able to provide particular uploads from "backups and cached files" for download via ankiweb? For example, if I had uploaded cards via sync before midnight, could I potentially download that particular upload? I'm sure this would be tedious to do so I understand if it's not likely (also because most users probably have their own local backups),

  18. 18 Posted by John on 27 Sep, 2016 01:27 PM

    John's Avatar

    So now syncing won't work at all. Is this system wide?

  19. Support Staff 19 Posted by Damien Elmes on 27 Sep, 2016 01:29 PM

    Damien Elmes's Avatar

    I've updated AnkiWeb to only throw an error for people likely affected by this issue, so people who didn't run into this issue should no longer receive errors when syncing.

    As mentioned above this issue does not affect any of your cards, only sounds and images that were attached to them. It will take some time to determine what can be recovered, so users who still have media on another device or a system backup like Time Machine are best off restoring from that. For users who do not have any other backups, I will post back as soon as I have more information.

  20. 20 Posted by Matias on 27 Sep, 2016 01:32 PM

    Matias's Avatar

    I followed the instructions but keep getting this message when I try to sync on my MacBook.
    any ideas?

    Syncing failed:
    Traceback (most recent call last):
      File "aqt/sync.pyc", line 324, in run
      File "aqt/sync.pyc", line 385, in _sync
      File "aqt/sync.pyc", line 416, in _syncMedia
      File "anki/sync.pyc", line 736, in sync
      File "anki/sync.pyc", line 870, in begin
      File "anki/sync.pyc", line 900, in _dataOnly
    Exception: SyncError:Please see http://help.ankisrs.net/discussions/announcements

  21. Support Staff 21 Posted by Damien Elmes on 27 Sep, 2016 01:34 PM

    Damien Elmes's Avatar

    Matias,

    That message indicates that are you likely affected by this issue. You will not be able to sync again until I can ensure that doing so will not cause further data loss - hopefully in the next 24 hours.

  22. 22 Posted by Andrei on 27 Sep, 2016 01:41 PM

    Andrei's Avatar

    Hey guys just wanted to say thank you so much for your efforts in restoring the Sync functionality and media files! We as community really appreciate that!

  23. 23 Posted by Jack on 27 Sep, 2016 01:45 PM

    Jack's Avatar

    Anki is throwing up a sync error for me. I have tried deleting all my decks (after backing them up) from Ankiweb, uninstalling the Anki desktop application and reinstalling it, but this does not help.

    I've attached a picture of the specific error message I'm getting. Any help would be great - I have my medical school finals coming up and have been quite reliant on Anki!

  24. 24 Posted by John on 27 Sep, 2016 01:46 PM

    John's Avatar

    Thanks, Damien. I'm unable to sync right now, probably due to the large number of media files I have. Will I run into scheduling or other issues if I continue to review cards now?

  25. 25 Posted by Christopher Mul... on 27 Sep, 2016 01:58 PM

    Christopher Mullins's Avatar

    Hi Damien, thanks so much for working on this.

    Yesterday I made a bunch of cards and sync'd a few times, and now I get the error so it seems I am affected. (Also AnkiDroid is reporting that there is now media not pointing to cards, and cards pointing to nonexistent media, so this confirms it.)

    I have two questions now:

    (1) Is there a way to tell if the backup I make *now* will have those media? As in, is there a way to check the most recent syncs and see if any media were deleted in them? (I haven't deleted any cards or media in weeks so any deletion would be a sign that I lost data.) I'll make a backup of my deck anyway, but if I'm backing up a deck with erroneous deletions then I'd like to know.

    (2) In the interim, I'd like to just make a temporary deck and work with that. Would you recommend just not syncing that at all?

    Thanks,
    Chris

  26. 26 Posted by Jordan Swenson on 27 Sep, 2016 02:08 PM

    Jordan Swenson's Avatar

    I have lost all media files from 9/15 and after. That is about 2 weeks of medical school. I do not have another device to sync from. I don't know what to do.

    I am not mad but a little frustrated. Anki isn't going to delete earlier media files right?

  27. 27 Posted by John on 27 Sep, 2016 02:28 PM

    John's Avatar

    Jordan,

    I feel ya. My first med school final is this Friday. Great timing! I got lucky and it doesn't look like many of my files were affected (knock on wood). Two things to try:

    1) Open your trash can and see if the media files are in there. If so, there should be some sort of option to "put them back." I don't know what this looks like on your system or if it will work. Try it and see (and let us know here what happens).

    1) Open your preferences in anki and click on 'Backups'. By default, Anki backs up your last 50(?) syncs so you should be able to find the last sync you had. You might lose any new cards and your scheduling info and if you've created or reviewed cards since then but you'll at least have the old files back.

    Neither option is ideal but unfortunately I think it's the best we've got for now. Good luck.

  28. 28 Posted by Thales Alexandr... on 27 Sep, 2016 02:34 PM

    Thales Alexandre Ferreira Albuquerque's Avatar

    Just to clarify: I can still study my cards on a device and have my progress saved even though I am recieving the sync failed message?

  29. 29 Posted by Christopher Mul... on 27 Sep, 2016 02:37 PM

    Christopher Mullins's Avatar

    @John

    I believe media is not backed up. Since only media is affected here, I don't think that will help. Option (1) seems to be the best idea.

  30. 30 Posted by Fernando on 27 Sep, 2016 02:41 PM

    Fernando's Avatar

    Muito obrigado pelo aviso e incansável empenho em consertar o problema.

    Abraço!

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac