Media syncing issues and possible data loss

Damien Elmes's Avatar

Damien Elmes

27 Sep, 2016 10:25 AM

UPDATE 5:

UPDATE 4:

UPDATE 3:

  • Some accounts could not be fully restored automatically.
  • There is a good chance we can recover your data if you can provide us with the log files Anki creates.
  • Please see this page to check on the status of your account, and to see how to provide the log files.

UPDATE 2:

  • Media syncing has been restored for the majority of affected accounts
  • Restored accounts will not sync deleted files for now - please see this article for more info.
  • We're still working on restoring data for some accounts
  • If you've recovered data from one of your devices, you can manually restore syncing - please see this article for more info.

UPDATE 1:

  • The majority of the lost files have now been restored, with 2/3rds of affected users no longer missing a single file.
  • It will take me up to another day to ensure it's safe to turn back on media syncing and to finish restoring what I can.
  • For those of you who've been able to restore data from another device, I understand you're eager to start syncing again, but I do need to keep syncing of the affected accounts offline for now until I can be sure no further data will be lost.

Thank you all for your patience, and I'll post back as soon as I have new info.

IN SHORT:

  • If syncing gives you an error pointing to the announcements page, you may have lost sound or image files on one or more devices due to an AnkiWeb issue.
  • The issue only affects media files - no card text or review data has been lost.
  • To prevent media files from being deleted on any other devices you own, media syncing has been temporarily disabled for you. We hope to restore functionality in the next 24 hours.
  • You can continue to sync - your card content and study progress will still be transferred, but you will receive an error at the end of the sync because media syncing is disabled, and no media files will be transferred to other devices.
  • It is safe to study, and you will not lose any more media files than have already been lost.
  • It is safe to add new media files to your cards.
  • Automatic backups do not contain media, so restoring from an automatic backup will not help.
  • To determine what has been lost and how to restore it, please see https://anki.tenderapp.com/kb/problems/recovering-from-the-ankiweb-...

Hi all,

I discovered a serious issue with the synchronisation service today that may have resulted in sound and image files in your collection being removed erroneously if you synced Anki in the last 24 hours.

The error occurred when syncing a device that needed to download media from AnkiWeb. Instead of the file data being returned correctly, an empty file was returned in certain cases. When subsequently syncing that device again, the empty file was treated as a removal, and the files were removed from other devices when you synced them afterwards.

To clarify, if the devices you synced in the last 24 hours:
- ...did not have any media changes, then you will not be affected - ...only uploaded media to AnkiWeb, then you will not be affected - ...downloaded media from AnkiWeb, then there is a chance you are affected

I have fixed the original issue that was causing the data loss, and will be making steps to ensure that it can not happen again. To prevent already deleted files from being deleted on more devices, I have temporarily configured AnkiWeb to return an error if your account may be affected by this data loss.

I will be attempting to restore as much data as I can from backups and cached files, and will edit this post when I have more information.

This should not have happened, and I'm very sorry for any inconvenience caused. If you have any questions or need help with making backups, please don't hesitate to contact us.

Showing page 2 out of 5. View the first page

  1. 31 Posted by remi on 27 Sep, 2016 02:43 PM

    remi's Avatar

    If our most recent media is intact, does that mean earlier media is intact as well? i.e. Does the bug affect media in any kind of order. I got the error message on my phone and mac, but it looks like my latest media is on my phone. Should I go looking for losses? Thanks.

  2. 32 Posted by John on 27 Sep, 2016 03:02 PM

    John's Avatar

    @Chris

    Ah yes, there is a note on the 'Backup' tab that media is not backed up. Thanks for clarifying.

  3. 33 Posted by John on 27 Sep, 2016 03:07 PM

    John's Avatar

    Maybe somone has already done this, but maybe it wouldn't be a bad idea to either:

    1) provide an option for users to backup media files. Could be done less periodically to save space and time as the user wishes.

    2) make a plugin that automatically exports decks as a workaround to (1).

  4. 34 Posted by Jack H on 27 Sep, 2016 03:41 PM

    Jack H's Avatar

    Would it be fine to go ahead and do my cards as usual? Would any progress be lost whenever I can sync back up again? I synced both my devices before I went to sleep last night as I had been adding cards up until then. The last card I added still has the audio on both devices so I'll join Remi in asking if that means anything or if the bug affects random media.

  5. 35 Posted by N on 27 Sep, 2016 04:00 PM

    N's Avatar

    I have the same question as Jack H

  6. 36 Posted by Saba sheikh bah... on 27 Sep, 2016 04:15 PM

    Saba sheikh bahaee's Avatar

    I have missed more than 200 media from all my files. it is too much for a fault of anki. too much time and energy we spent for making these flashes for USMLE exam, and our exam is too close :(((( really hopeless!!! plzzz repair it and bring them back

  7. 37 Posted by Soren Bjornstad on 27 Sep, 2016 04:18 PM

    Soren Bjornstad's Avatar

    Hi everyone,

    Yes, reviewing on any device is safe. The issue affects media syncing only. Just keep in mind that you may not be able to sync the reviews to another device for a little while longer if you're affected.

    I am not sure whether there is a time aspect to which media files are affected -- if I had to guess based on what I know of the bug, I would expect that only items that had changed since the last time you synced to a device would be affected, but only Damien knows for sure, and hopefully we'll hear from him again as soon as he can take a break from fixing the problem.

    @Jordan:

    Anki isn't going to delete earlier media files right?

    No further media files will be deleted by this bug unless something further goes wrong (and Damien is being very careful to make sure syncing isn't enabled until it's safe), so the state you're currently in is the worst it will get. If the device you're on has a trash folder, please make sure you check in there to see if the recent media made it there.

  8. 38 Posted by Soren Bjornstad on 27 Sep, 2016 04:23 PM

    Soren Bjornstad's Avatar

    @Chris:

    Is there a way to tell if the backup I make now will have those media?

    Check in Tools -> Check Media -- if the media you've recently added are listed as missing, they will not be included in a backup taken now. As noted in Damien's original post, having things appear does not necessarily mean your collection was affected, though, since media could have been missing before. You should search your collection for the specific filenames the media check shows you to make sure those weren't ones that were missing before.

    In the interim, I'd like to just make a temporary deck and work with that. Would you recommend just not syncing that at all?

    To be totally safe, you can create a new profile from File -> Switch Profile (which will not sync unless you connect a different AnkiWeb account to it), and then export your deck as an apkg and import it back into your main profile later if you decide you want to do that.

    @Jack: The error indicates that you're affected by the issue -- please read Damien's original post.

  9. 39 Posted by Ari on 27 Sep, 2016 04:29 PM

    Ari's Avatar

    I am missing media and looked in my trash folder but none of it was there. If I go to edit card, edit HTML, a filename for the media is still there, but searching for the filename on my PC doesn't bring up any results.

  10. 40 Posted by Christopher Mul... on 27 Sep, 2016 04:35 PM

    Christopher Mullins's Avatar

    Thanks for the reply @Soren.

    I think I may have bungled all my decks systematically by being diligent about syncing on all three of my devices. Doing a grep for paste-67426691579908.jpg (one of the missing files added recently) in my Anki/ directory shows several references but no file, and that appears to be the case for about 20 media files. Glad that it's only that many.

    Thanks for the help! Please update when we can sync again.

  11. 41 Posted by Enrico on 27 Sep, 2016 05:08 PM

    Enrico's Avatar

    Falha na sincronização:
    Traceback (most recent call last):
      File "aqt\sync.py", line 324, in run
      File "aqt\sync.py", line 385, in _sync
      File "aqt\sync.py", line 416, in _syncMedia
      File "anki\sync.py", line 736, in sync
      File "anki\sync.py", line 870, in begin
      File "anki\sync.py", line 900, in _dataOnly
    Exception: SyncError:Please see http://help.ankisrs.net/discussions/announcements

  12. 42 Posted by saman.nazemian on 27 Sep, 2016 05:13 PM

    saman.nazemian's Avatar

    I have lost lots of images!
    please when you fixed the bug and hopefully recovered the media
    put an option in sync temporarily to add images bilaterally( from web to device and vice versa) and do not assume any images deleted. but let the flash timings and notes sync as usual.

    I noted every time I have synced I have lost about 200 images :((

  13. 43 Posted by sdimons on 27 Sep, 2016 06:09 PM

    sdimons's Avatar

    Hi, I have same error. I can't sync media files since last night.

    Синхронизация не удалась:
    Traceback (most recent call last):
      File "aqt\sync.py", line 324, in run
      File "aqt\sync.py", line 385, in _sync
      File "aqt\sync.py", line 416, in _syncMedia
      File "anki\sync.py", line 736, in sync
      File "anki\sync.py", line 870, in begin
      File "anki\sync.py", line 900, in _dataOnly
    Exception: SyncError:Please see http://help.ankisrs.net/discussions/announcements

  14. 44 Posted by Joseph on 27 Sep, 2016 07:28 PM

    Joseph's Avatar

    It says I have 109 files present but not used by any notes.
    Does this mean media on 109 of my cards essentially got deleted?

  15. 45 Posted by Soren Bjornstad on 27 Sep, 2016 07:34 PM

    Soren Bjornstad's Avatar

    @Enrico, sdimons: Please read Damien's initial post in this thread.

    @Joseph: No, files present but not used on any notes means you have extra media, so at least the device you ran the check on has not been affected.

  16. 46 Posted by gardnerlex on 27 Sep, 2016 07:42 PM

    gardnerlex's Avatar

    I was able to recover media from a backup disk and put it into my Anki folder on my mac. I'm now getting the same message as Joseph about having extra cards. As it appears that I have fixed my issue, is there a way to resume syncing since it appears my account is no longer impacted?

  17. 47 Posted by Vicky Johnson on 27 Sep, 2016 07:46 PM

    Vicky Johnson's Avatar

    Hello, I've been directed here after getting the prompt from Anki on my phone. I use Anki on an iPhone 4 and a Macbook. I'm pretty sure I synced both within the last 24 hours, but not sure whether I'm affected.

    I know this is likely to be a stupid question, but re. "You can check if you are missing any media by using the "Check Media" function", where should I be doing this? I can't see this option in the iOS Anki.

    Should I also (or instead) do this in the Macbook Anki?

  18. 48 Posted by Serg on 27 Sep, 2016 08:00 PM

    Serg's Avatar

    Hello! I have lost cards too. Please, what i have to do to solve the problem likes sdimons : "Traceback (most recent call last):
      File "aqt\sync.py", line 324, in run
      File "aqt\sync.py", line 385, in _sync
      File "aqt\sync.py", line 416, in _syncMedia
      File "anki\sync.py", line 736, in sync
      File "anki\sync.py", line 870, in begin
      File "anki\sync.py", line 900, in _dataOnly
    Exception: SyncError:Please see http://help.ankisrs.net/discussions/announcements"

    I can't sync on computer.

  19. 49 Posted by Christopher Mul... on 27 Sep, 2016 08:01 PM

    Christopher Mullins's Avatar

    @gardnerlex

    In my experience today, it's still syncing despite the error, it's just not
    syncing any media.

    Chris

  20. 50 Posted by Bannet Muhoozi on 27 Sep, 2016 08:04 PM

    Bannet Muhoozi's Avatar

    I managed to back up my files so no lossess thankfully. I would like to continue as normal but now i cant upload to anki web from my computer. How do i fix this?

    Syncing failed:
    Traceback (most recent call last):
      File "aqt\sync.py", line 324, in run
      File "aqt\sync.py", line 385, in _sync
      File "aqt\sync.py", line 416, in _syncMedia
      File "anki\sync.py", line 736, in sync
      File "anki\sync.py", line 870, in begin
      File "anki\sync.py", line 900, in _dataOnly
    Exception: SyncError:Please see http://help.ankisrs.net/discussions/announcements

  21. 51 Posted by gardnerlex on 27 Sep, 2016 08:06 PM

    gardnerlex's Avatar

    @Chris that appears to be the case here too

  22. 52 Posted by FJ on 27 Sep, 2016 08:26 PM

    FJ's Avatar

    So, I have 'check media' telling me that some images aren't used by any cards.

    However, I search for those cards in 'Browse' and they are all there (only found one card that had broken images that were on the 'check media' list) -- when I restored the images to the card (by selecting them from the attach button) I noticed in the media folder they appeared with a (1) appended to the filename.

    I guess I'm confused as to why 'check media' thinks I'm not using any cards with a dozen or so images, when browsing the cards reveals that I am. I don't want to 'delete unused' because that seems like it is definitely going to cause a problem, given that I am using those images in cards.

    Confused as hell at the moment on what exactly the course forward is.

  23. 53 Posted by curbuntu on 27 Sep, 2016 08:34 PM

    curbuntu's Avatar

    @Damien
    This glitch has made me realize more clearly just how vital Anki is to me -- and that I haven't "paid my dues." But I can't seem to find a URL that will allow me to make a donation. Thanks for all you've done on this project over the years (since 2006?!) and for tackling the issue du jour.

  24. 54 Posted by Soren Bjornstad on 27 Sep, 2016 08:49 PM

    Soren Bjornstad's Avatar

    Everyone new to the discussion (and Serg): If you are about to post an error message, please first go read Damien's original post. It outlines why you are receiving this error message, and posting another copy of the same message just clutters up the thread.

    @gardnerlex: Not yet, but hopefully we'll have it resolved soon.

    @FJ:

    I guess I'm confused as to why 'check media' thinks I'm not using any cards with a dozen or so images, when browsing the cards reveals that I am.

    Is there any chance you're using media references on the template?

    @veejay: The instructions I gave are for the desktop version, but there's also a "check media" button at the very bottom of the settings screen on iOS.

    @curbuntu: Due to Anki's current structure as a business, we're not able to accept donations. You can help by buying the iOS version, or by pointing friends who have iOS devices and would find Anki useful to us. We are hoping to add other options in the future.

  25. 55 Posted by Marco Aurélio on 27 Sep, 2016 09:01 PM

    Marco Aurélio's Avatar

    Guys, congratulations for the good work.

    Suggestion: it'd be a good idea backups including media also.

    I agree with @curbuntu. I only use anki for two months, building card by card on my own and they're now 1.300 cards with 150 media (pics). I've lost 7 pics only, but it turned on the yellow flag. I can't afford loose them, so, I'm the whole day making a script to backup the whole directory to google drive, daily. It would be nice if anki delivered that function automatically.

    Once more, I'm very thankfull anyway.

  26. 56 Posted by Vicky Johnson on 27 Sep, 2016 09:10 PM

    Vicky Johnson's Avatar

    @Soren Thanks for replying. I have "Check database" at the bottom of the settings screen is iOS rather than 'Check media'. I just clicked on it and got the message "Database rebuilt and optimized." Not sure what that means in relation to this issue.

    I did "Check media" for the database version, and got hundreds of results. There are too many to check them all, but it looks as though many are images from decks which I've deleted in the past, and some are from my current decks.

    What do you advise? Should I just sit tight and not use Anki for the moment?

  27. 57 Posted by Ketetha Olengue on 27 Sep, 2016 09:27 PM

    Ketetha Olengue's Avatar

    This comment was split into a new discussion: Restoring media from a shared deck

    I had an issue a few months ago where I lost all my media files as well. If I were to re-downlaoded the deck, could I somehow restore those media files without losing my scheduling information? I rather have my stats saved then the photos but it is super hard as the deck was not made by me and the photos are from a board review book that really make it easier to reference the answer rather than going to find it in the book.

    Thanks.

  28. 58 Posted by ifwhatwhen on 27 Sep, 2016 10:08 PM

    ifwhatwhen's Avatar

    Has anyone with this problem tried opening from an automatic backup to see if that helps? I don't want to be set back a day but I guess I'd take that over losing so much media...

  29. 59 Posted by Thales Alexandr... on 27 Sep, 2016 10:10 PM

    Thales Alexandre Ferreira Albuquerque's Avatar

    I was answering questions on anki after losing my media, and realized even though some of my more recent midia was still there, others were missing. So, in answer to a question that has been asked: yes, it seems like we can lose any media, not just the latest.

    It would be interesting if, as @Ketheta suggested, we could download only the midia of a deck from the shared page. I believe many of us here are medical students, and if others, like me, are using Brosencephalon's step 1 deck, we might be in real trouble if we have to find and replace among its 2572 images. I also think of those who use the program to study japanese. The core 10k for japanese has an even larger number of mp3 and jpeg files.

  30. 60 Posted by Soren Bjornstad on 27 Sep, 2016 11:11 PM

    Soren Bjornstad's Avatar

    @Marco: Including media in all backups is not a viable option if people have a large amount of media, especially on mobile (the backups might end up taking up many gigabytes), but it's definitely a good idea to rethink the possibilities.

    @veejay: That suggests you aren't using the latest version of the app. But no matter, the desktop version's check is identical.

    it looks as though many are images from decks which I've deleted in the past

    Are you sure those aren't the ones that say "in media folder but not used by any cards"? If all of the results are under that heading and not "Used on cards but missing from media folder", then you can just click "delete unused," or even do nothing, and you'll be in good shape.

    If you do have some media under each of the two categories, then the ones in the "used on cards but missing from media folder" are the only ones you need to worry about here.

    @ifwhatwhen: Automatic backups unfortunately do not include media due to space issues (see earlier in this post), so that would not accomplish anything besides setting you back a day.

    @Ketetha, Thales: You could import the deck into a new profile, then move the media in Documents/Anki/[the new profile name]/collection.media into the collection.media folder for your other profile. Your cards themselves (and therefore the information about which media file to display on the card) will not have been touched by this issue, so if you can just get the media back in your media folder everything will be fine.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac