Adds a resolve undo entry for a file based on the given parameters.
The resolve undo entry contains the OIDs of files that were involved in a merge conflict after the conflict has been resolved. This allows conflicts to be re-resolved later.
If there exists a resolve undo entry for the given path in the index, it will be removed.
This method will fail in bare index instances.
filename to add
mode of the ancestor file
mode of our file
mode of their file
0 or an error code