Is an implementation of SM 2 with only two answer choices (pass/fail) viable?

jc.imbeault's Avatar

jc.imbeault

07 Oct, 2017 09:30 AM

Quick summary of my question for the community:

1- What the impact on learning would be with only two buttons.
2- If using only two buttons is it correct to set their “quality” to 0 and 3.5?
3- What are the downsides (if any) of using just pass/fail?

The reason I ask is ...

I'm thinking of using SRS for a group of people where it's not possible for them to rate the quality of their answer. The best that can be done is a simple yes/no (pass/fail).

I've looked at the original SM 2 algorithm, and it had 6 'quality of response' levels 0-5. If only pass/fail were used, this would mean q (quality of response) would always be either 3 or 0.

So using:

constant q = 3.5
Initial EF = 2.5
New EF = EF - 0.8 + (0.28 * 3.5) - (0.02 * 3.5 * 3.5) (equivalent to EF' = EF - 0.065)
I(1) = 1
I(2) = 6
I(n) = I(n-1)*EF

I calculated the EF's and interval periods assuming always passing, and this is what the values look like:

Repetition Interval in days EF
1 1 2.5
2 6 2.435
3 14.61 2.37
4 34.6257 1 Months, 5 Days 2.305
5 79.8122385 2 Months, 20 Days 2.24
6 178.7794142 5 Months, 29 Days 2.175
7 388.845226 1 Years, 0 Months, 24 Days 2.11
8 820.4634268 2 Years, 3 Months, 0 Days 2.045
9 1677.847708 4 Years, 7 Months, 8 Days 1.98
10 3322.138461 9 Years, 1 Months, 7 Days 1.915

So it looks like using just pass/fail will at least generate sane spacing intervals. Though I am surprised that the EF is continuously decreasing.

Thank you!

  1. 1 Posted by Vit on 07 Oct, 2017 11:18 PM

    Vit's Avatar

    Look at this add-on
    Button Colours (Good, Again)

  2. 2 Posted by jc.imbeault on 08 Oct, 2017 12:36 AM

    jc.imbeault's Avatar

    Thanks for the reply.

    I had a look at the add-on but all it does is change the button colors.
    From the add-on’s screenshot you can see that there are three buttons.

    I’m asking what the impact on learning would be with only two buttons.

    And also, I’m asking if I use only two buttons is it correct to set their
    “quality” to 0 and 3.5?

  3. 3 Posted by ZapBeeb on 08 Oct, 2017 08:29 AM

    ZapBeeb's Avatar

    I fear nobody knows the answer to this.
    I am not rightly sure that the SM2 algorithm itself has been thorougly and scientifically scrutinized, so it's anybody's guess.

    It's almost a given that the review time will go up, but how much? No idea.

  4. 4 Posted by jc.imbeault on 08 Oct, 2017 08:38 AM

    jc.imbeault's Avatar

    @ZapBeeb, thanks for the reply.

    I agree that the algorithm itself hasn't been thoroughly vetted.

    That being said I'm hoping someone has some experience with using just two choices when reviewing and they can share it with me.

    Or that someone with a better understand of the the algorithm than me can point out what the impact (good and bad) are (or might be).

    SRS is an amazing technique and I would like to be able to share it with those that don't have the ability to rate their own answers besides pass or fail.

  5. 5 Posted by Eric on 17 Nov, 2017 10:24 AM

    Eric's Avatar

    Disclaimer: everything I write is pure speculation and isn't based on any experience.

    1- What the impact on learning would be with only two buttons?

    It would be much easier for the learner to grade his answers. It's a lot easier to determine if you're satisfied with your memory, than how much you're satisfied with it. Also, remember that the grading is subjective, and the more possible grades, the more the process is prone to inconsistencies.

    2- If using only two buttons is it correct to set their “quality” to 0 and 3.5?

    Like you and ZapBeeb mentioned, SM-2 is arbitrary and cannot even claim to be proved scientifically, as Wozniak puts it gently in https://www.supermemo.com/english/ol/sm4.htm (I'd advise you to read these two threads: https://groups.google.com/forum/#!topic/mnemosyne-proj-users/ppiFre... and https://anki.tenderapp.com/discussions/effective-learning/1068-does... to see how unscientifically it is), so there aren’t any "correct" values.

    But because you're asking from the algorithm's perspective, the answer has to be no, these values are incorrect. Since you're pleased with the way Anki works, which is to look for "the true" EF, any q<4 won't fit the "pass" grade, as it will reduce the EF. You were misled by SM-2's wording, for inasmuch as the q values are arbitrary, the q labels are all the more so. The "pass" label that he gave to q=3 actually means the EF was too high and will be decreased, so in Boolean terms it ought to be labeled "fail". You'd be better off with 4 for the "pass". Maybe even a little bigger, so you could increase the EF gently, but that might be risky. Needless to say, you don’t really have to use the SM-2 formulas; just define how each grade change the EF.

    Having said that, I personally find that reducing the EF slightly when passing is a reasonable approach, as I explained in the fixed ease thread (linked above), so these values do fit my taste.

    3- What are the downsides (if any) of using just pass/fail?

    As I mentioned in the fixed ease thread (linked above), a Boolean feedback might be too crude to adjust the EF, if you want to increase it occasionally. But since you seem okay with even constantly reducing it (I hope your users agree with that), I'd say there are no downsides for you. You don't even have to implement it anew; the users can simply use Anki with only the "good" and "again" buttons. You can even write a small add-on to remove the other buttons, if you'd like. Assuming always passing with the default settings means you can only squeeze 12 repetitions in a lifetime. That's tolerable, I guess.

    But if you do consider it a downside, there are other ways to implement it, which will even let you increase the EF sensibly.

    First of all I'd recommend considering the REWISE algorithm (http://www.langmaster.com/lmcom/com/web/en-gb/pages/help/downloads/...) which is the most elegant implementation of SM-2 which I have ever seen (actually, I'm not even sure they ever saw SM-2, as their charts resemble Pilmsuer's original paper). This algorithm also deals with out-of-schedule repetitions, former familiarity and much more, and is easy to tweak to Boolean feedback. Though I suspect there's a problem with the initial acquisition, because the algorithm presumes that there's an interval right after learning, and that after forgetting the interval should decrease, while we all know that at first it might take some rehearsals until the item is remembered, even without reducing the interval.

    Here are some other algorithms which have been tested scientifically, that I am unfortunately unable to understand, being illiterate as I am. But perhaps you can implement them.
    http://aclweb.org/anthology/P/P16/P16-1174.pdf
    https://arxiv.org/pdf/1602.07032.pdf
    http://pdfs.semanticscholar.org/81df/0033b74fe8b7a942411d89fc7e19a6...
    http://www.cs.colorado.edu/~mozer/Research/Selected%20Publications/...
    http://www.cs.colorado.edu/~mozer/Research/Selected%20Publications/...
    http://www.cs.colorado.edu/~mozer/Research/Selected%20Publications/...

    these are but a few, google the articles and authors referenced there for more articles of the same sort.

    good luck, and please share your results with us.

  6. 6 Posted by Eric on 17 Nov, 2017 12:03 PM

    Eric's Avatar

    (Shortened version, the longer wasn't posted for some reason.) Disclaimer: everything I write is pure speculation and isn't based on any experience.

    1- What the impact on learning would be with only two buttons?
    It would be much easier for the learner to grade his answers. It's a lot easier to determine if you're satisfied with your memory, than how much you're satisfied with it. Also, remember that the grading is subjective, and the more possible grades, the more the process is prone to inconsistencies.

    2- If using only two buttons is it correct to set their “quality” to 0 and 3.5?
    The algorithm looks for "the true" EF, so any q<4 won't fit the "pass" grade, as it will reduce the EF. You were misled by SM-2's arbitrary wording: it labels q=3 as "pass", while it actually means the EF was too high and will be decreased, so in Boolean terms it ought to be labeled "fail". You'd be better off with 4 for the "pass". Maybe even a little bigger, so you could increase the EF gently, but that might be risky. Needless to say, you don’t really have to use the SM-2 formulas; just define how each grade change the EF.

    Having said that, I personally find that reducing the EF slightly when passing is a reasonable approach, as I explained in the fixed ease thread (https://anki.tenderapp.com/discussions/effective-learning/1068), so these values do fit my taste.

    3- What are the downsides (if any) of using just pass/fail?
    As I mentioned in the fixed ease thread (linked above), a Boolean feedback might be too crude to adjust the EF, if you want to increase it occasionally. But since you seem okay with even constantly reducing it (I hope your users agree with that), I'd say there are no downsides for you. You don't even have to implement it anew; the users can simply use Anki with only the "good" and "again" buttons. You can even write a small add-on to remove the other buttons, if you'd like. Assuming always passing with the default settings means you can only squeeze 12 repetitions in a lifetime. That's tolerable, I guess.

    But if you do consider it a downside, there are other ways to implement it, which will even let you increase the EF sensibly.

    First of all I'd recommend considering the REWISE algorithm (http://www.langmaster.com/lmcom/com/web/en-gb/pages/help/downloads/...) which is the most elegant implementation of SM-2 which I have ever seen (actually, I'm not even sure they ever saw SM-2, as their charts resemble Pilmsuer's original paper). This algorithm also deals with out-of-schedule repetitions, former familiarity and much more, and is easy to tweak to Boolean feedback. Though I suspect there's a problem with the initial acquisition, because the algorithm presumes that there's an interval right after learning, and that after forgetting the interval should decrease, while we all know that at first it might take some rehearsals until the item is remembered, even without reducing the interval.

    Here are some other algorithms which have been tested scientifically, that I am unfortunately unable to understand, being illiterate as I am. But perhaps you can implement them.
    http://aclweb.org/anthology/P/P16/P16-1174.pdf
    https://arxiv.org/pdf/1602.07032.pdf
    http://www.cs.colorado.edu/~mozer/Research/Selected%20Publications/...
    These are but a few, google the articles and authors referenced there for more articles of the same sort.

  7. 7 Posted by jabamboleo on 06 Mar, 2020 08:55 AM

    jabamboleo's Avatar

    I'm late to the party but I think Low-Key Anki is exactly what your looking for:
    https://massimmersionapproach.com/table-of-contents/anki/low-key-an...

Comments are closed, but you can start a new discussion.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac