Is there a hard limit when importing?

Hi, I'm struggling on importing a csv file I generated from a deck I already have just to add a new field and update it to the deck. I have done this several times without an issue. But this time the file to import it’s relatively large. How I try to import Anki gets stuck in “importing...” window. I have checked and the cpu load stays constant at 0% and having plenty of free RAM memory. So it’s not hardware bottleneck.

Let me explain the reason I’m trying to import this:

I have about 6000+ Japanese kanji notes with a filed linking to a svg file, which many of them weights 3-7kB. But since the sd card where I storage the media has a bytes per sector too low it ends up taking so much space and fills the whole sd card, it take so much space inefficiently.

So I thought that since many small files is not good for fat32 I tried to embed the svg into a field. I converted the svg into base64 string into the field and then with the help of css can show it. SO, no dependencies whatsoever and faster to transfer on the cell phone because one large file is preferible that many small files.

This increases the size of the deck, ergo anki database, but theorically just in 18MB at maximum. Not a big deal. It is already at 110MB (uncompressed).

My front template contains this:
<object type="image/svg+xml" data = "data:image/svg+xml;base64,{{uri}}">

And my uri field is like this:
PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCEtLQpUaGlzIGZpbGUgaGFzIGJlZW4gbW9kaWZpZWQgZnJvbSB0aGUgb3JpZ2luYWwgdmVyc2lvbiBieSB0aGUga2FuamlfY29sb3JpemUucHkKc2NyaXB0IChhdmFpbGFibGUgYXQgaHR0cDovL2dpdGh1Yi5jb20vY2F5ZW5uZXMva2FuamktY29sb3JpemUpIHdpdGggdGhlc2UKc2V0dGluZ3M6CiAgICBtb2RlOiBjb250cmFzdAogICAgc2F0dXJhdGlvbjogMC43CiAgICB2YWx1ZTogMC44NQogICAgaW1hZ2Vfc2l6ZTogMTAwCkl0IHJlbWFpbnMgdW5kZXIgYSBDcmVhdGl2ZSBDb21tb25zLUF0dHJpYnV0aW9uLVNoYXJlIEFsaWtlIDMuMCBMaWNlbnNlLgoKVGhlIG9yaWdpbmFsIFNWRyBoYXMgdGhlIGZvbGxvd2luZyBjb3B5cmlnaHQ6CgpDb3B5cmlnaHQgKEMpIDIwMDkvMjAxMC8yMDExIFVscmljaCBBcGVsLgpUaGlzIHdvcmsgaXMgZGlzdHJpYnV0ZWQgdW5kZXIgdGhlIGNvbmRpdGlvbnMgb2YgdGhlIENyZWF0aXZlIENvbW1vbnMKQXR0cmlidXRpb24tU2hhcmUgQWxpa2UgMy4wIExpY2VuY2UuIFRoaXMgbWVhbnMgeW91IGFyZSBmcmVlOgoqIHRvIFNoYXJlIC0gdG8gY29weSwgZGlzdHJpYnV0ZSBhbmQgdHJhbnNtaXQgdGhlIHdvcmsKKiB0byBSZW1peCAtIHRvIGFkYXB0IHRoZSB3b3JrCgpVbmRlciB0aGUgZm9sbG93aW5nIGNvbmRpdGlvbnM6CiogQXR0cmlidXRpb24uIFlvdSBtdXN0IGF0dHJpYnV0ZSB0aGUgd29yayBieSBzdGF0aW5nIHlvdXIgdXNlIG9mIEthbmppVkcgaW4KICB5b3VyIG93biBjb3B5cmlnaHQgaGVhZGVyIGFuZCBsaW5raW5nIHRvIEthbmppVkcncyB3ZWJzaXRlCiAgKGh0dHA6Ly9rYW5qaXZnLnRhZ2FpbmkubmV0KQoqIFNoYXJlIEFsaWtlLiBJZiB5b3UgYWx0ZXIsIHRyYW5zZm9ybSwgb3IgYnVpbGQgdXBvbiB0aGlzIHdvcmssIHlvdSBtYXkKICBkaXN0cmlidXRlIHRoZSByZXN1bHRpbmcgd29yayBvbmx5IHVuZGVyIHRoZSBzYW1lIG9yIHNpbWlsYXIgbGljZW5zZSB0byB0aGlzCiAgb25lLgoKU2VlIGh0dHA6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LXNhLzMuMC8gZm9yIG1vcmUgZGV0YWlscy4KLS0+CjwhRE9DVFlQRSBzdmcgUFVCTElDICItLy9XM0MvL0RURCBTVkcgMS4wLy9FTiIgImh0dHA6Ly93d3cudzMub3JnL1RSLzIwMDEvUkVDLVNWRy0yMDAxMDkwNC9EVEQvc3ZnMTAuZHRkIiBbCjwhQVRUTElTVCBnCnhtbG5zOmt2ZyBDREFUQSAjRklYRUQgImh0dHA6Ly9rYW5qaXZnLnRhZ2FpbmkubmV0Igprdmc6ZWxlbWVudCBDREFUQSAjSU1QTElFRAprdmc6dmFyaWFudCBDREFUQSAjSU1QTElFRAprdmc6cGFydGlhbCBDREFUQSAjSU1QTElFRAprdmc6b3JpZ2luYWwgQ0RBVEEgI0lNUExJRUQKa3ZnOnBhcnQgQ0RBVEEgI0lNUExJRUQKa3ZnOm51bWJlciBDREFUQSAjSU1QTElFRAprdmc6dHJhZEZvcm0gQ0RBVEEgI0lNUExJRUQKa3ZnOnJhZGljYWxGb3JtIENEQVRBICNJTVBMSUVECmt2Zzpwb3NpdGlvbiBDREFUQSAjSU1QTElFRAprdmc6cmFkaWNhbCBDREFUQSAjSU1QTElFRAprdmc6cGhvbiBDREFUQSAjSU1QTElFRCA+CjwhQVRUTElTVCBwYXRoCnhtbG5zOmt2ZyBDREFUQSAjRklYRUQgImh0dHA6Ly9rYW5qaXZnLnRhZ2FpbmkubmV0Igprdmc6dHlwZSBDREFUQSAjSU1QTElFRCA+Cl0+CjxzdmcgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB3aWR0aD0iMTAwIiBoZWlnaHQgPSAiMTAwIiB2aWV3Qm94PSIwIDAgMTAwIDEwMCI+CjxnIGlkPSJrdmc6U3Ryb2tlUGF0aHNfMDRlMDAiIHN0eWxlPSJmaWxsOm5vbmU7c3Ryb2tlOiMwMDAwMDA7c3Ryb2tlLXdpZHRoOjM7c3Ryb2tlLWxpbmVjYXA6cm91bmQ7c3Ryb2tlLWxpbmVqb2luOnJvdW5kOyIgdHJhbnNmb3JtPSJzY2FsZSgwLjkxNzQzMTE5MjY2MDU1MDUsMC45MTc0MzExOTI2NjA1NTA1KSI+CjxnIGlkPSJrdmc6MDRlMDAiIGt2ZzplbGVtZW50PSLkuIAiIGt2ZzpyYWRpY2FsPSJnZW5lcmFsIj4KCTxwYXRoIGlkPSJrdmc6MDRlMDAtczEiIGt2Zzp0eXBlPSLjh5AiIGQ9Ik0xMSw1NC4yNWMzLjE5LDAuNjIsNi4yNSwwLjc1LDkuNzMsMC41YzIwLjY0LTEuNSw1MC4zOS01LjEyLDY4LjU4LTUuMjRjMy42LTAuMDIsNS43NywwLjI0LDcuNTcsMC40OSIvPgo8L2c+CjwvZz4KPGcgaWQ9Imt2ZzpTdHJva2VOdW1iZXJzXzA0ZTAwIiBzdHlsZT0iZm9udC1zaXplOjg7ZmlsbDojODA4MDgwIiB0cmFuc2Zvcm09InNjYWxlKDAuOTE3NDMxMTkyNjYwNTUwNSwwLjkxNzQzMTE5MjY2MDU1MDUpIj4KCQo8L2c+Cjwvc3ZnPgoK

This works. I have done it manually in one card to make sure it is feasible. The problem is the bulk import with the URI fields that Anki seems unable to do. It is just the bulk import of 6000+ lines with a string of data URI of 3k-7k and is when it never ends importing and doesn’t seems to be using computer resources at all.

Thank you for your amazing support.

  1. 1 Posted by Guillem Palau-S... on 15 Aug, 2014 09:33 PM

    Guillem Palau-Salvà (contributor user)'s Avatar

    Update:

    I changed the way I was importing the data.
    I found out that several fields of base64 are long because they are from png that represents a kanji (when the picture is just a part of a kanji, where is not coded by itself in unicode) and is larger than the default limit of python and I get this message:

    Importing complete.
    Aborted: field larger than field limit (131072)

    I've searched on the net and I found that this limit can be incremented with the following line of code:

    csv.field_size_limit([new_limit])

    My question is, where and which file do I have to modify to see the changes.

  2. 2 Posted by Soren Bjornstad on 15 Aug, 2014 10:15 PM

    Soren Bjornstad's Avatar

    Hi guillem,

    The issue is caused by the limit on line length, as you appear to have figured out now. The reason we have the limit is because extremely large fields like this can cause slow performance along with the increased collection size. We would encourage you not to try to get around it, but since it appears that your other option isn't great either, it might be worth a try. The right place to insert it would be at the top of anki/importing/csvfile.py (I haven't tested an abnormally large import with the change, but I have confirmed it didn't cause an error when importing a file). If it's more convenient to do so, writing an add-on that imports anki.importing.csvfile and then sets it from there should be equivalent.

  3. 3 Posted by Guillem Palau-S... on 16 Aug, 2014 09:03 AM

    Guillem Palau-Salvà (contributor user)'s Avatar

    Awesome !

    I just did my first add-on :-) Its just 2 lines, but does what I need.
    When I build up enough programming skills I would be delighted to contribute to the project.

  4. 4 Posted by Soren Bjornstad on 16 Aug, 2014 01:48 PM

    Soren Bjornstad's Avatar

    Glad you found a solution.

  5. Soren Bjornstad closed this discussion on 16 Aug, 2014 01:48 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac