Duplicating cells results in autograding failure because of duplicate IDs, and confusing error message

[rkdarst]

This is similar to #981, as in that may solve the problem, but I'll document core problem here and a workaround.

Students have submitted notebooks in which they have copied cells. This duplicates the metadata: of importance here is "nbgrader": {"grade_id": ...}. When this happens, the following error message is given:

[AutogradeApp | WARNING] Cell with id 'Task_3_1_test' exists multiple times!
...
[AutogradeApp | WARNING] Removing failed assignment: /m/jhnas/jupyter/course/mlpython2019/files/autograded/staafv1/R1_Introduction
[AutogradeApp | ERROR] One or more notebooks in the assignment use an old version of the nbgrader metadata format. Please **back up your class files directory** and then update the metadata using: nbgrader update .

The first warning is the key to the problem. However, the error message shows that nbgrader metadata should be updated, which not the real problem here (though I haven't tested it if actually works). At least, the error message could be improved or the failure could be noticed earlier.

It isn't a perfect solution, but it would be nice if nbgrader could also continue on in this case. Clearly the student has done something weird (and notebook/jupyterlab could solve it by not duplicating metadata). But could nbgrader also survive this and make some choice one way or the other itself, considering that duplicating metadata is probably a type of problem that will appear again? That way the common case won't cause a failure stopping all autograding... and we leave it to instructors to tell of the problem later.

Who knows what the right nbgrader-only solution would be - simple would be to pick the last cell with the metadata. But many corner cases could break this.

Workaround for anyone else who gets this problem: My tests showed that you have to delete the whole nbgrader metadata dict from the notebook cell - changing the grade_id is not enough because it can't find that grade_id in the database, nor is removing just the grade_id cell.

Operating system

Linux

nbgrader --version

Modified version, based on upstream d56f75b (close to current master).

[jhamrick]

Hmm, I am pretty sure it was the case that in the notebook when you copy a cell it does not copy the metadata. Are your students by chance either using JupyterLab (which I think may not implement this feature), or a very old version of the notebook? It's also possible there's been a regression.

In any case, I agree it would be nice if nbgrader handled this more gracefully. Certainly the error message could be better at least. I'll look into options for how to handle this better and report back.

[rkdarst]

In our provided hub, courses use Notebook by default (5.7.2 it seems we are using now). But Lab is available, and the way to switch between them is documented but not. One previous course also instructed their students to download notebooks and move them to another service for actually doing the assignments - who knows what errors could be inserted in this process!

I'm not an instructor and don't interact with the students, so don't know that much about what students are actually doing and can't ask them questions.

Random thought: are the old values of the cells re-inserted during the fetch stage or the autograde stage? Because in both of the cases where I saw this, a cell was duplicated but both of the sources were the same. I wonder if this is a coincidence (they dupilacated cells and didn't edit them) or by design (same cell IDs are updated with the same contents, meaning it really doesn't matter if a duplicate is dropped).

[jhamrick]

Hmm, I would expect your version of the notebook is new enough (it's a really old version that doesn't support it). Maybe they're using JupyterLab or maybe there's been a regression 😕

Regarding your last question, the answer depends on the cell type I think... for read-only cells (including cells with tests in them) this is the intended behavior in case students manage to edit them somehow, and so I wouldn't be surprised if it just worked with the duplicated cells too. But for other types of cells that have nbgrader ids (e.g. duplicate answer cells) it shouldn't happen that they have the same source, unless the student copied the source.

[elesiuta]

I experienced this issue as well in the class I recently TA’d (nbgrader 0.5.4 on jupyterhub 0.9.4). My solution was a script that merges cells when it detects duplicate grade_ids by concatenating the cell’s source to the first then removing the duplicate.

I uploaded the script as part of a small collection I wrote for that course in case it’s of use to anyone https://github.com/elesiuta/jupyter-nbgrader-helper (run with --fix AssignName NbName.ipynb). As an aside, if there’s any feature in there you think might be useful to implement properly into nbgrader I’d be happy to open a pull request and work on it.

[akskuchi]

This also happened when I used the Jupyter notebook split cell editing (CTRL + SHIFT + -). The new cell created as a result of the split has the same cell-ID (as the original cell). Having the same cell-ID for two cells prevents me from saving the notebook. Annoying!

Likely something to do with this - #758?

[fredcallaway]

I'm having the same problem. You can accidentally copy a read-only cell and then you can't delete it. This causes nbgrader validate to crash down the road.

jupyPython version 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0]
nbgrader version 0.6.0

[jhamrick]

I can confirm this is an issue, and it sounds like it is a regression in the notebook as this was not a problem previously.

[Sefriol]

This also causes problems with the feedback system, since Instructor needs to manually fix the notebook, i.e. it's hash changes which is used to distribute the feedback form.