alt Hacker NewsFurther question—the model produces an answer to the question, it sends the answer, and then gets graded. Does it get to know immediately how it did, or does it get the grade back at the end after answering all the questions?
If it is the former case, it would be possible to add the generated question/answer pair into the training set as well. Would that be considered fair? (Of course this is a moot point if the answers all get graded simultaneously at the end). Then the model could explore interesting strategies around what order to answer questions in.
In my uninformed opinion, the various permutations of question ordering/answer revealing all map to different real-world scenarios… and any of them could be interesting!
Replies
It does not, if it gets the answer (or any information about them, even % of qns solved) and is able to adjust itself in response, then that is considered training on the test set and is wrong.
Nope, it never learns how it did on the questions.
During test time, you have to submit all the answers at once and you get the total score (so you dont even know which puzzles were solved)