Problem
The situation in which a system answer file and a dataset file are uploaded occurs often in practice but there is no unit test covering it. Hence, bugs like the one fixed with commit 794c41a are hard to detect.
Solution
Implement a unit test that simulates an experiment in which 2 files have been uploaded and are used as prediction and gold standard.
The implementation should be done on the following branch, which still has the bug mentioned above: QA-name-bug
The unit test should simulate an experiment task that is created by submitting the following configuration from the UI:
{"type":"QA","matching":"STRONG_ENTITY_MATCH","annotator":[],"dataset":["NIFDS_gold1(gold_1.json)"],"answerFiles":["AF_System(pred_1.json)(undefined)(AFDS_gold_1.json)"],"questionLanguage":"en"}pred_1.json and gold_1.json should have the same content and may have to be moved into the upload directory as preparation of the unit test. It is expected that the resulting F1-measures are 1.0.
GoldStdEvalTest can be used as inspiration.
Problem
The situation in which a system answer file and a dataset file are uploaded occurs often in practice but there is no unit test covering it. Hence, bugs like the one fixed with commit 794c41a are hard to detect.
Solution
Implement a unit test that simulates an experiment in which 2 files have been uploaded and are used as prediction and gold standard.
The implementation should be done on the following branch, which still has the bug mentioned above: QA-name-bug
The unit test should simulate an experiment task that is created by submitting the following configuration from the UI:
{"type":"QA","matching":"STRONG_ENTITY_MATCH","annotator":[],"dataset":["NIFDS_gold1(gold_1.json)"],"answerFiles":["AF_System(pred_1.json)(undefined)(AFDS_gold_1.json)"],"questionLanguage":"en"}pred_1.jsonandgold_1.jsonshould have the same content and may have to be moved into the upload directory as preparation of the unit test. It is expected that the resulting F1-measures are 1.0.GoldStdEvalTest can be used as inspiration.