To extract a relation, the user picks a document with sentences from the system. Therefore, the user is focused on one sentence at a time. By splitting documents up into sentences, I attempt to create a more organized and clear interface. For the task of manual triplet extraction, we do not put strict lexical constraints in place, in contrast to multiple automatic triplet extraction approaches. Since the imported text does not necessarily adhere to any specific structure, I am cautious to implement such strict constraints. However, a sentence may have many words what leads to a lot of options for a possible relation. To reduce the number of options and to attempt to steer the focus of the user slightly, we implement the following lexical guidelines:
Since one user has now extracted a triplet, one vote is counted for this triplet (one for correct, zero votes for incorrect). Therefore, the triplet is saved to the IE triplet store, until majority voting indicates that the triplet is incorrect.
After the triplet is saved, the user receives points for his effort. These points are added to the total amount of points the user has collected.
Triplet verification is conducted by either agreeing or disagreeing with a given triplet. The triplet verification task contributes to the following goals:
After a user has verified a triplet, the vote is submitted to the HC datastore after which we calculate with majority voting whether the triplet is correct or incorrect.
When majority voting calculates the triplet is found incorrect, the triplet is removed from our IE datastore. When the triplet is found correct, the triplet is added to our IE datastore. In case the outcome of majority voting has not changed compared to the previous calculation, only the vote is saved.
When the status of the triplet has either changed from correct to incorrect or from incorrect to correct, points are withdrawn from or added to the user who initially extracted the triplet.