Frequent Database Corruptions / Database Code

Toni's Avatar

Toni

14 Jan, 2020 09:21 PM

Hi,

I frequently experience database corruptions for no real reason (eg. after pressing the Escape key while in the card browser), which has become a major pain in the behind for me. These now happen usually several times a day, even though I only edit my Anki database on one computer, and currently - due to this problem - refrain from syncing. But looking into the code, which seems to use plain DBAPI, and in the vein of having generally safer and better tested code, I consider trying to hack on Anki with the goal of porting the database logic to SQLAlchemy's declarative, which is battle-tested and gives high-level abstractions and integrity checks, amongst other niceties. I hope that this would make Anki quite a bit more robust, instead of the file on disk, which is a bit unwieldy.

What's your opinion?

TIA!

  1. Support Staff 1 Posted by Damien Elmes on 14 Jan, 2020 10:36 PM

    Damien Elmes's Avatar

    Anki moved away from SQLAlchemy years ago as it had a measurable impact on performance. What exactly are the errors you're getting?

  2. 2 Posted by Toni on 14 Jan, 2020 10:55 PM

    Toni's Avatar

    I get lots of database corruption errors, which I have not yet debugged, and I have also not kept the myriad of tracebacks, but they frequently involve the equivalent of null pointer dereferences, and occur on many, many occasions. Also, when they occur, the only way to actually quit Anki to run the database check upon restart, is to kill the Python process, because inside Anki, it is no longer possible to quit. But if you can leave the thread open, I'll post some tracebacks tne next time around. Shouldn't take too long.

    Apart from that, computers have become much more beefy in the last few years. Anki sometimes runs with quite acceptable speed on a 10 year old computer of mine, but not having to repair the database so often would be a massive speedup for me in any case.

  3. Support Staff 3 Posted by Damien Elmes on 14 Jan, 2020 11:29 PM

    Damien Elmes's Avatar

    null pointer deferences sound like you're using broken libraries or modifying the database file externally (by putting it in a synced folder for example). They are not something you should see under normal use, and not something moving to SQLAlchemy would change, as it is simply a layer on top of the database library Anki is already using.

    You could try the packaged versions on the Anki website to see if the issue is with your system's sqlite libraries.

  4. Support Staff 4 Posted by Damien Elmes on 15 Jan, 2020 12:09 AM

    Damien Elmes's Avatar

    Actually, ignore that - sqlite is dynamically linked in the Linux builds, so your system version will still be picked up.

  5. 5 Posted by Toni on 28 Mar, 2020 01:48 AM

    Toni's Avatar

    So I tried to use a pre-packaged version of Anki, after building it from source seems to be a pretty brittle operation. I used the 2.1.22 package, and I get this on startup:

    When loading '⁨Chinese Support Redux v0130⁩':
    Traceback (most recent call last):
      File "aqt/addons.py", line 208, in loadAddons
      File "/home/toni/.local/share/Anki2/addons21/1128979221/__init__.py", line 23, in <module>
        from . import main
      File "/home/toni/.local/share/Anki2/addons21/1128979221/main.py", line 34, in <module>
        from .templates import chinese, ruby
      File "/home/toni/.local/share/Anki2/addons21/1128979221/templates/chinese.py", line 14, in <module>
        from anki.template.hint import hint
    ModuleNotFoundError: No module named 'anki.template.hint'; 'anki.template' is not a package

    Btw, it would be great if there were a file that would shed some light onto the design philosophy and explain why now Rust and NodeJS are required to build the system, too, or where the project is headed.

  6. Support Staff 6 Posted by Damien Elmes on 28 Mar, 2020 04:26 AM

    Damien Elmes's Avatar

    The add-on author has not updated the add-on to support recent Anki versions yet: https://github.com/luoliyan/chinese-support-redux/issues/118

    Rust performs much better than Python and tends to result in more reliable code, and Node is required for the scss and ts files that Anki makes use of.

  7. 7 Posted by Toni on 28 Mar, 2020 01:32 PM

    Toni's Avatar

    Thank you for the answer! So you are completely rewriting Anki in Rust?

  8. Support Staff 8 Posted by Damien Elmes on 29 Mar, 2020 09:13 AM

    Damien Elmes's Avatar

    I wouldn't go that far :-) I expect there will be more Rust introduced in the future, but I'm taking it one step at a time, and the Qt side of things will remain fully Python.

  9. 9 Posted by Toni on 29 Mar, 2020 06:58 PM

    Toni's Avatar

    FWIW: https://github.com/luoliyan/chinese-support-redux/pull/120

    I'm not sure how to polish it, though.

  10. 10 Posted by Toni on 29 Mar, 2020 07:39 PM

    Toni's Avatar

    I found that my database corruption problems are usually this, across several versions of anki, both source and binary, and also several versions of plugins:

    Caught exception:
    Traceback (most recent call last):
      File "/home/...anki/aqt/progress.py", line 72, in handler
      File "/home/...anki/aqt/main.py", line 535, in reset
      File "/home/...anki/aqt/main.py", line 485, in moveToState
      File "/home/...anki/aqt/main.py", line 515, in _reviewState
      File "/home/...anki/aqt/reviewer.py", line 46, in show
      File "/home/...anki/aqt/reviewer.py", line 64, in nextCard
      File "/home/....anki/anki/collection.py", ine 661, in timeboxReached
    AttributeError: '_Collection' object has no attribute '_startTime'
    

    Btw, not being able to run anki under PDB is not an improvement from my current position. Can we have a way to run several instances of Anki in parallel, with eg. different databases or plugins, please? So far, when I try to invoke another instance of Anki, it always tells me that I have one instance already running, and that it is switching to that.

  11. Support Staff 11 Posted by Damien Elmes on 30 Mar, 2020 11:17 AM

    Damien Elmes's Avatar

    Can you reproduce that error with all add-ons disabled? If not, please disable them in groups until you discover which one is causing it.

    Debugging is covered in the docs: https://addon-docs.ankiweb.net/#/more?id=debugging

    You may be able to get multiple instances going with an add-on, but I'm afraid this functionality is not officially supported at the moment.

  12. 12 Posted by Toni on 31 Mar, 2020 04:37 PM

    Toni's Avatar

    Since the error occurs quite randomly, I have just added the DEBUG var to my environment when starting Anki, and now I wait for the bug to happen, at which point I hope I can gain more insight with the debugger. Btw, the last time this problem happened, Anki proceeded to consume all available memory. I had to kill it.

Comments are closed, but you can start a new discussion.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac