Design and Evaluation of an IR-Benchmark for SPARQL Fulltext Queries

In this thesis, we further describe indexing, ranking, and query processing techniques that we implement in order to process the new kind of SPARQL-FT queries, provided in the context of Jeopardy task of the INEX 2012 Linked Data track, by introducing the SPAR-Key engine. For the rapid development of the new query engine that could handle this particular combination of XML mark-up and RDF-style resource/property-pairs, we decide to opt for a relational-DBMS as storage back-end, which allows us to index the collection and to retrieve both the SPARQL- and keyword-related conditions of the Jeopardy queries under one common application layer. Additionally, our engine comes with a rewriting layer that translates the SPARQL-based query patterns into unions of conjunctive SQL queries, thus formulating joins over both the DBpedia triples and the keywords extracted from the XML articles.
Finally, we perform a detailed evaluation of the effectiveness of our query engine by processing the benchmark queries. We present the results from the official INEX'12 evaluations for the Jeopardy task that was performed with Ad-hoc search style relevance assessments, obtained with the help of crowd sourcing. However, we show that such an evaluation does not truly comply with the task definition, and hence a re-evaluation with a QA-style assessment is required. For the re-evaluation, we create gold result set, or ground truth, by mapping already known correct answers of the NL questions to the Wikipedia entities. By outperforming our competitors in terms of MRR and NDCG, we show definite advantages of exploiting both structured information and unstructured information to improve Question-Answering and Entity-retrieval tasks.