Comprehensive and Practical Policy Compliance in Data Retrieval Systems

In this thesis, we present comprehensive and practical compliance systems to address this challenge. To be comprehensive, compliance systems must be able to enforce policies specific to individual data items or to a particular client's data items, the service provider's own policies, and policies that capture legal requirements. To be practical, compliance systems need to meet stringent requirements: runtime overhead must be low; existing applications can run with little modifications; and bugs, misconfigurations, compromises in application code, or actions by unprivileged operators cannot violate policies.

We present the design and implementation of two comprehensive and practical compliance systems: Thoth and Shai. At a high-level, data use policies are stated in a declarative language separate from application code, and a small reference monitor ensures compliance with these policies. Thoth and Shai differ in enforcement techniques. Thoth tracks data flows through the system at runtime by intercepting I/O at processes' boundaries, and enforces the associated policies. Shai, on the other hand, combines static flow analysis and light-weight runtime monitoring (sandboxes and capabilities) to ensure compliance of data flows. We demonstrate the practicality of these systems using a prototype search engine based on the popular Apache Lucene.