These notes give a broad overview of the background and some aspects of it that we touch upon in the initial readings. In general, you should be conversant with all of the concepts mentioned in these notes as part of the background for the course. Background material is available in the S&K textbook, as noted below. Classic references include:
Threads of control execute within code modules. As code from a module executes, it perform operations on data. Modules are supplied by a provider and execute on behalf of some user.
Protection systems are based on a reference monitor that is empowered to approve or deny each operation request by executing code. Each request is made by some subject to operate upon some object. The reference monitor makes each choice according to an access control policy. Protection system policy can be thought of in terms of an access control matrix that enumerates all operations allowable for all pairs of subjects and objects. Two fundamental ways to represent the set of allowable operations in the matrix are access control lists (in which each object has a list of the subjects permitted to access it) and capabilities (in which each subject has a protected and unforgeable list of the objects it is permitted to access).
Example: If you want to go to backstage at a concert, the reference monitor is the bouncer at the stage door. A backstage pass is a capability. A guest list is an access control list.
How does a protection system define the subject? Under various circumstances, it may be important to consider the module, the identity of the provider of the module, other modules or providers in the call chain, and/or the user on whose behalf the code is running. The term principal is often used to denote identities or compositions of identities associated with a subject. The readings explore a range of conceptions of "subject" in various contexts ranging from classical operating systems (a process running on behalf of a user identity, with user and kernel mode), to virtual machine systems, to dynamic Web applications (Wallach97, MashupOS) and extensible operating systems (SPIN).
Protection systems use many different mechanisms to intercept requested operations and submit them to the reference monitor for approval. An important example is memory system protection, which defines the memory regions accessible to executing code. Memory system protection creates a "sandbox" for programs to run in, preventing them from interfering with other code and data. A variety of mechanisms exist to enforce this isolation. The mechanism to enforce this isolation is crucial for performance, since access checks must be made on every memory reference. Examples of these mechanisms include: hardware-based memory protection (virtual address spaces and virtual page protection), language-based protection (type-safe compiled languages), interpreted languages in which the interpreter serves as the reference monitor (e.g., Javascript), Software Fault Isolation in which instructions are inserted statically into the code stream to check bounds on each reference, and Proof Carrying Codes, in which a code segment includes a static and verifiable proof that it will not reference memory outside of of some defined set. (SFI and PCC are advanced topics rather than background.)
Generally speaking and regardless of the mechanism, an executing thread or module is associated with a protection domain. The protection domain defines the set of objects and operations that are allowable to the thread. Protection domains give us a vocabulary to talk about communication and interaction between protected entities.
For example, it is often useful for an executing thread or stream of control to cross from one protection domain to another deliberately and in a controlled way. This can be thought of as "changing the subject" in order to either constrain the privileges of some invoked module (to contain it if it is untrusted), or to request some privileged operation from a trusted module.
For example, some mechanism is needed to invoke a trusted subsystem, i.e., a code module with higher privilege than the caller.
Another example is a transfer of control between two peer protection domains. It can be thought of as a special kind of procedure call that crosses protection domain boundaries: a cross-domain procedure call. It is often called LPC or LRPC when between processes on the same host or operating system image. The "L" stands for "local". The performance of cross-domain control transfer has been a dominating concern among systems researchers dealing with memory system protection. Some take the "L" to stand for "lightweight", because it must be fast. Efficient LPC is a foundational mechanism in MS-Windows.
Cross-domain control transfer abstractions can be built using other abstractions for message-passing across protection domains. For example, two guest virtual machines (often called domains because they are instances of a kind of protection domain) on the same host can communicate through sockets. Of course, the messages may be sent over a network (RPC), and the network boundary itself provides strong isolation between protection domains or modules on either side of the network. This is the focus of Chapter 4 in S&K.
If two protection domains are co-located on the same home, then some other mechanism must be employed to protect objects (e.g., memory) from access by unauthorized protection domains. This is the problem alluded to above: by what mechanism does the trusted reference monitor interpose on any requested object operation? In the memory protection case, we use hardware support to create multiple naming contexts (virtual address spaces) for each process to use in accessing its memory. The virtual address space is an implementation of the protection domain concept.
Virtual memory hardware is one element of the support in modern processors for protected operating system kernels and/or virtual machine systems. For example, virtual memory hardware also allows a trusted kernel to control which addresses are accessible for each protection domain, and their mappings to the underlying physical pages. This enables the operating system to manage the physical memory resource as well as to protect data in memory from unauthorized access. Modern processors also include protected mode bits: protected mode (e.g., kernel mode) activates an additional set of accessible translations for the virtual address space, e.g., allowing the module to access memory regions reserved to the operating system kernel. Entry into kernel mode is protected by hardware-supported exception mechanisms (traps, faults, and interrupts). The kernel serves as the reference monitor for memory pages and other objects including files and I/O channels. These topics are discussed in Chapter 5 of S&K.
The relationship between naming and protection is discussed more generally in Chapters 2 and 3 of S&K.
It is important to understand that protection systems are only as strong as the weakest link. In particular, the protection systems we discuss presume that the reference monitor and isolation mechanisms (e.g., Java type system) cannot be circumvented, and that subjects control their own code. Many significant security failures occur because an attacker takes control of a subject and injects its own code that allows it to issue its own actions with the subject's identity. For example, in 1988 the Internet was crippled by a worm that used a variety of means to gain control of processes running with root (superuser/administrator) identity on Unix systems. The attacker exploited programming errors and vulnerabilities in the message handling code (e.g., buffer overflow) of certain Unix server programs. To a large extent these problems can be addressed in programming systems, but it is important in particular for C programmers to program carefully to avoid such vulnerabilities. For a detailed treatment of the topic, see Secure Programming for Linux and Unix
Butler Lampson of Microsoft has argued rather provocatively in Accountability and Freedom that software security techniques and quality processes can never be powerful enough to develop operating systems that are both as secure and as richly featured as users demand. The argument is that rich features always create opportunities for attackers to compromise the protection system and gain control of a subject. The inconvenience of preventive security (e.g., principle of least privilege) places too much burden on the user to judge the trustworthiness of each piece of imported code in order to determine what privileges to assign to it. Lampson has suggested that virtual machines would allow consumers to run multiple separate operating system images with different levels of security. For example, a user might manage their finances from a different system image than the one they use to try out Internet freeware and otherwise "live wild and free on the Internet".
There are several good papers that are relevant, and a couple will show up on the reading list for any graduate OS offering. Here are a few that we have used: