Format Factory 4.6.2.0 + Portable [Latest]

Incident Response

Additional Context

Indicators

File Details

All Details:

Screenshots

Hybrid Analysis

Network Analysis

conferenceproceedingsProceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation Broomfield, CO October 6–8, 201411th USENIX Symposium onOperating Systems Designand ImplementationBroomfield, COOctober 6–8, 2014Sponsored byIn cooperation with ACM SIGOPS

Thanks to Our OSDI ’14 SponsorsDiamond SponsorsGold SponsorsThanks to Our USENIX and LISA SIG SupportersUSENIX PatronsGoogle Microsoft Research NetApp VMwareSilver SponsorsUSENIX BenefactorsAkamai Facebook Hewlett-Packard IBM ResearchLinux Pro Magazine Puppet LabsUSENIX and LISA SIG PartnersCambridge Computer GoogleBronze SponsorsUSENIX PartnersCan Stock Photo EMCGeneral SponsorACM QueueADMIN magazineCRC PressDistributed ManagementTask Force (DMTF)Media Sponsors and Industry PartnersThe FreeBSD FoundationHPCwireInfoSec NewsLinux Pro MagazineLXerNo Starch PressO’Reilly MediaRaspberry Pi GeekUserFriendly.org© 2014 by The USENIX AssociationAll Rights ReservedThis volume is published as a collective work. Rights to individual papers remain with theauthor or the author’s employer. Permission is granted for the noncommercial reproduction ofthe complete work for educational or research purposes. Permission is granted to print, primarilyfor one person’s exclusive use, a single copy of these Proceedings. USENIX acknowledges alltrademarks herein.ISBN 978-1-931971-16-4

USENIX AssociationProceedings of the 11th USENIX Symposiumon Operating Systems Design andImplementation (OSDI ’14)October 6–8, 2014Broomfield, CO

Conference OrganizersProgram Co-ChairsJason Flinn, University of MichiganHank Levy, University of WashingtonProgram CommitteeLorenzo Alvisi, The University of Texas at AustinDave Andersen, Carnegie Mellon UniversityRemzi Arpaci-Dusseau, University of Wisconsin—MadisonMihai Budiu, Microsoft ResearchGeorge Candea, École Polytechnique Fédérale deLausanne (EPFL)Peter Chen, University of MichiganAllen Clement, Google and Max Planck Institute forSoftware Systems (MPI-SWS)Landon Cox, Duke UniversityNick Feamster, Georgia Institute of TechnologyBryan Ford, Yale UniversityRoxana Geambasu, Columbia UniversityGernot Heiser, University of New South WalesAustralia/NICTAM.

Radu BanabicJulian BangertPramod BhatotiaStefan BucurHaogang ChenVitaly ChipounovAustin ClementsDavid CockPatrick ColpCody CutlerRicardo DiasPedro FonsecaJoão GarciaQian GeZhenyu GuoJi HongAnuj KaliaManos KapritsosBaris KasikciVolodymyr KuznetsovDavid LazarGeoffrey LefebvreJoão LeitãoHyeontaek LimYunxin LiuYandong MaoSyed Akbar MehdiDutch MeyerExternal ReviewersHenrique MonizIulian MoraruToby MurrayMihir NanavatiNeha NarulaDaniel PeekRaluca Ada PopaDaniel PortoDan PortsZhengping QianShriram RajagopalanFranzi RoesnerChunzhi SuPhilippe SuterStephen TuJelle van den HooffJonas WagnerXi WangYang WangMing WuChao XieFan YangCristian ZamfirHuanchen ZhangTimmy Zhu

11th USENIX Symposium onOperating Systems Design and ImplementationOctober 6–8, 2014Broomfield, COMessage from the Program Chair. ...............................................................ixMonday, October 6, 2014Who Put the Kernel in My OS Conference?Arrakis: The Operating System is the Control Plane. ............................................... 1Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug Woos, Arvind Krishnamurthy, and Thomas Anderson,University of Washington; Timothy Roscoe, ETH ZürichDecoupling Cores, Kernels, and Operating Systems. ............................................... 17Gerd Zellweger, Simon Gerber, Kornilios Kourtis, and Timothy Roscoe, ETH ZürichJitk: A Trustworthy In-Kernel Interpreter Infrastructure. ......................................... 33Xi Wang, David Lazar, Nickolai Zeldovich, and Adam Chlipala, MIT CSAIL; Zachary Tatlock, Universityof WashingtonIX: A Protected Dataplane Operating System for High Throughput and Low Latency. .................. 49Adam Belay, Stanford University; George Prekas, École Polytechnique Fédérale de Lausanne (EPFL);Ana Klimovic, Samuel Grossman, and Christos Kozyrakis, Stanford University; Edouard Bugnion, ÉcolePolytechnique Fédérale de Lausanne (EPFL)Data in the AbstractWillow: A User-Programmable SSD............................................................. 67Sudharsan Seshadri, Mark Gahagan, Sundaram Bhaskaran, Trevor Bunker, Arup De, Yanqin Jin, Yang Liu,and Steven Swanson, University of California, San DiegoPhysical Disentanglement in a Container-Based File System ........................................ 81Lanyue Lu, Yupu Zhang, Thanh Do, Samer Al-Kiswany, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, University of Wisconsin—MadisonCustomizable and Extensible Deployment for Mobile/Cloud Applications ............................. 97Irene Zhang, Adriana Szekeres, Dana Van Aken, and Isaac Ackerman, University of Washington; Steven D.Gribble, Google and University of Washington; Arvind Krishnamurthy and Henry M. Levy, University ofWashingtonPebbles: Fine-Grained Data Management Abstractions for Modern Operating Systems ................ 113Riley Spahn and Jonathan Bell, Columbia University; Michael Lee, The University of Texas at Austin; SravanBhamidipati, Roxana Geambasu, and Gail Kaiser, Columbia UniversityMy InsecuritiesProtecting Users by Confining JavaScript with COWL............................................ 131Deian Stefan and Edward Z. Yang, Stanford University; Petr Marchenko, Google; Alejandro Russo, ChalmersUniversity of Technology; Dave Herman, Mozilla; Brad Karp, University College London; David Mazières,Stanford UniversityCode-Pointer Integrity. ...................................................................... 147Volodymyr Kuznetsov, École Polytechnique Fédérale de Lausanne (EPFL); László Szekeres, Stony BrookUniversity; Mathias Payer, Purdue University; George Candea, École Polytechnique Fédérale de Lausanne(EPFL); R. Sekar, Stony Brook University; Dawn Song, University of California, Berkeley(Monday, October 6, continues on next page)

Ironclad Apps: End-to-End Security via Automated Full-System Verification. ........................ 165Chris Hawblitzel, Jon Howell, and Jacob R. Lorch, Microsoft Research; Arjun Narayan, Universityof Pennsylvania; Bryan Parno, Microsoft Research; Danfeng Zhang, Cornell University; Brian Zill,Microsoft ResearchShill: A Secure Shell Scripting Language. ...................................................... 183Scott Moore, Christos Dimoulas, Dan King, and Stephen Chong, Harvard UniversityVariety PackGPUnet: Networking Abstractions for GPU Programs ............................................ 201Sangman Kim, Seonggu Huh, Yige Hu, Xinya Zhang, and Emmett Witchel, The University of Texas at Austin;Amir Wated and Mark Silberstein, Technion—Israel Institute of TechnologyThe Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services .............. 217Michael Chow, University of Michigan; David Meisner, Facebook, Inc.; Jason Flinn, University of Michigan;Daniel Peek, Facebook, Inc.; Thomas F. Wenisch, University of MichiganEnd-to-end Performance Isolation Through Virtual Datacenters. ................................... 233Sebastian Angel, The University of Texas at Austin; Hitesh Ballani, Thomas Karagiannis, Greg O’Shea,and Eno Thereska, Microsoft ResearchSimple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed ..... 249Data-Intensive SystemsDing Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain,and Michael Stumm, University of TorontoTuesday, October 7, 2014Head in the CloudShielding Applications from an Untrusted Cloud with Haven....................................... 267Andrew Baumann, Marcus Peinado, and Galen Hunt, Microsoft ResearchApollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing ........................... 285Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, and Jingren Zhou, Microsoft; Zhengping Qian, Ming Wu,and Lidong Zhou, Microsoft ResearchThe Power of Choice in Data-Aware Cluster Scheduling. .......................................... 301Shivaram Venkataraman and Aurojit Panda, University of California, Berkeley; Ganesh Ananthanarayanan,Microsoft Research; Michael J. Franklin and Ion Stoica, University of California, BerkeleyHeading Off Correlated Failures through Independence-as-a-Service ............................... 317Ennan Zhai, Yale University; Ruichuan Chen, Bell Labs and Alcatel-Lucent; David Isaac Wolinsky andBryan Ford, Yale UniversityStorage Runs Hot and ColdCharacterizing Storage Workloads with Counter Stacks. .......................................... 335Jake Wires, Stephen Ingram, Zachary Drudi, Nicholas J. A. Harvey, and Andrew Warfield, Coho DataPelican: A Building Block for Exascale Cold Data Storage. ........................................ 351Shobana Balakrishnan, Richard Black, Austin Donnelly, Paul England, Adam Glass, Dave Harper, andSergey Legtchenko, Microsoft Research; Aaron Ogus, Microsoft; Eric Peterson and Antony Rowstron,Microsoft ResearchA Self-Configurable Geo-Replicated Cloud Storage System. ....................................... 367Masoud Saeida Ardekani, INRIA and Sorbonne Universités; Douglas B. Terry, Microsoft Research

f4: Facebook’s Warm BLOB Storage System..................................................... 383Subramanian Muralidhar, Facebook, Inc.; Wyatt Lloyd, University of Southern California and Facebook, Inc.;Sabyasachi Roy, Cory Hill, Ernest Lin, Weiwen Liu, Satadru Pan, Shiva Shankar, and Viswanath Sivakumar,Facebook, Inc.; Linpeng Tang, Princeton University and Facebook, Inc.; Sanjeev Kumar, Facebook, Inc.Pest ControlSAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems. ........... 399Tanakorn Leesatapornwongsa and Mingzhe Hao, University of Chicago; Pallavi Joshi, NEC Labs America;Jeffrey F. Lukman, Surya University; Haryadi S. Gunawi, University of ChicagoSki: Exposing Kernel Concurrency Bugs through Systematic Schedule Exploration. ................... 415Pedro Fonseca, Max Planck Institute for Software Systems (MPI-SWS); Rodrigo Rodrigues, CITI/NOVAUniversity of Lisbon; Björn B. Brandenburg, Max Planck Institute for Software Systems (MPI-SWS)All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications. ... 433Thanumalayan Sankaranarayana Pillai, Vijay Chidambaram, Ramnatthan Alagappan, Samer Al-Kiswany,Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, University of Wisconsin–MadisonTorturing Databases for Fun and Profit ........................................................ 449Mai Zheng, The Ohio State University; Joseph Tucek, HP Labs; Dachuan Huang and Feng Qin, The Ohio StateUniversity; Mark Lillibridge, Elizabeth S. Yang, and Bill W. Zhao, HP Labs; Shashank Singh, The Ohio StateUniversityTransaction ActionFast Databases with Fast Durability and Recovery Through Multicore Parallelism .................... 465Wenting Zheng and Stephen Tu, Massachusetts Institute of Technology; Eddie Kohler, Harvard University;Barbara Liskov, Massachusetts Institute of TechnologyExtracting More Concurrency from Distributed Transactions. ..................................... 479Shuai Mu, Tsinghua University and New York University; Yang Cui and Yang Zhang, New York University;Wyatt Lloyd, University of Southern California and Facebook, Inc.; Jinyang Li, New York UniversitySalt: Combining ACID and BASE in a Distributed Database. ...................................... 495Chao Xie, Chunzhi Su, Manos Kapritsos, Yang Wang, Navid Yaghmazadeh, Lorenzo Alvisi, and Prince Mahajan,The University of Texas at AustinPhase Reconciliation for Contended In-Memory Transactions. ..................................... 511Neha Narula and Cody Cutler, MIT CSAIL; Eddie Kohler, Harvard University; Robert Morris, MIT CSAILWednesday, October 8, 2014Play It Again, SamEidetic Systems. ............................................................................ 525David Devecsery, Michael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen, University of MichiganDetecting Covert Timing Channels with Time-Deterministic Replay. ................................ 541Ang Chen, University of Pennsylvania; W. Brad Moore, Georgetown University; Hanjun Xiao, AndreasHaeberlen, and Linh Thi Xuan Phan, University of Pennsylvania; Micah Sherr and Wenchao Zhou, GeorgetownUniversityIdentifying Information Disclosure in Web Applications with Retroactive Auditing. ................... 555Haogang Chen, Taesoo Kim, Xi Wang, Nickolai Zeldovich, and M. Frans Kaashoek, MIT CSAIL(Wednesday, October 8, continues on next page)

Help Me LearnProject Adam: Building an Efficient and Scalable Deep Learning Training System.................... 571Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman, Microsoft ResearchScaling Distributed Machine Learning with the Parameter Server. ................................. 583Mu Li, Carnegie Mellon University and Baidu; David G. Andersen and Jun Woo Park, Carnegie MellonUniversity; Alexander J. Smola, Carnegie Mellon University and Google, Inc.; Amr Ahmed, Vanja Josifovski,James Long, Eugene J. Shekita, and Bor-Yiing Su, Google, Inc.GraphX: Graph Processing in a Distributed Dataflow Framework. ................................. 599Joseph E. Gonzalez, University of California, Berkeley; Reynold S. Xin, University of California, Berkeley, andDatabricks; Ankur Dave, Daniel Crankshaw, and Michael J. Franklin, University of California, Berkeley; IonStoica, University of California, Berkeley, and DatabricksHammers and SawsNail: A Practical Tool for Parsing and Generating Data Formats ................................... 615Julian Bangert and Nickolai Zeldovich, MIT CSAILlprof: A Non-intrusive Request Flow Profiler for Distributed Systems. ............................... 629Xu Zhao, Yongle Zhang, David Lion, Muhammad Faizan Ullah, Yu Luo, Ding Yuan, and Michael Stumm,University of TorontoPydron: Semi-Automatic Parallelization for Multi-Core and the Cloud .............................. 645Stefan C. Müller, ETH Zürich and University of Applied Sciences Northwestern Switzerland; Gustavo Alonsoand Adam Amara, ETH Zürich; André Csillaghy, University of Applied Sciences Northwestern SwitzerlandUser-Guided Device Driver Synthesis. .......................................................... 661Leonid Ryzhyk, University of Toronto, NICTA, and University of New South Wales; Adam Walker, NICTAand University of New South Wales; John Keys, Intel Corporation; Alexander Legg, NICTA and University ofNew South Wales; Arun Raghunath, Intel Corporation; Michael Stumm, University of Toronto; Mona Vij, IntelCorporation

Message from theOSDI ’14 Program Co-ChairsWe are delighted to welcome you to the 11th USENIX Symposium on Operating Systems Design and Implementation.This year’s program contains 42 papers representing some of the best research from academia and industry inthe area of experimental systems.This year, we received 228 submissions that met the formatting guidelines for the conference. We used three roundsof review, followed by electronic discussion after each round, to evaluate the submissions. Eleven papers weredesignated as “quick-accepts” based on reviews and electronic discussions. An additional 66 papers were discussedduring a day-and-a-half-long PC meeting of which 31 were selected for the program.For the first time at OSDI, the submission process included a response period in which authors could answerreviewer questions and address factual errors in the reviews. Responses had a measurable impact on PC meetingdiscussions, helping some papers and hurting others. Overall, we believe responses were quite useful in improvingthe fairness of the review process and the quality of the selected program.From the beginning of the process, our goal was to increase the number of papers appearing at OSDI to approximately40 accepted papers. As the chart below shows, the number of submissions to the conference has steadilyincreased over the years, while the number of accepted papers has not kept pace. At the same time, flagship conferencesin related fields (architecture, networking, programming languages, etc.) have all substantially increasedthe number of accepted papers. This harms our field in several ways: some of our top work does not appear at ourflagship conferences, our researchers are at a competitive disadvantage compared to researchers in other fields, andthe review process becomes excessively negative. We believe that the 42 papers in the final program reflect boththe overall strength and the breadth of research in our community.250200150100SubmittedAccepted5001996 1999 2000 2002 2004 2006 2008 2010 2012 2014We wish to thank the members of the program committee and the external review committee who produced 950reviews in approximately 2 months time. We especially thank the many members of the external review committeewho volunteered to do extra reviews when the number of submissions came in higher than we expected. We alsothank the external reviewers who helped with reviews and the USENIX staff who organized the conference.Jason Flinn, University of MichiganHank Levy, University of WashingtonOSDI ’14 Program Co-ChairsUSENIX Association11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) ix

Arrakis: The Operating System is the Control PlaneSimon Peter ∗ Jialin Li ∗ Irene Zhang ∗ Dan R. K. Ports ∗ Doug Woos ∗Arvind Krishnamurthy ∗ Thomas Anderson ∗ Timothy Roscoe †University of Washington ∗ ETH Zurich †AbstractRecent device hardware trends enable a new approach tothe design of network server operating systems. In a traditionaloperating system, the kernel mediates access todevice hardware by server applications, to enforce processisolation as well as network and disk security. We have designedand implemented a new operating system, Arrakis,that splits the traditional role of the kernel in two. Applicationshave direct access to virtualized I/O devices, allowingmost I/O operations to skip the kernel entirely, while thekernel is re-engineered to provide network and disk protectionwithout kernel mediation of every operation. Wedescribe the hardware and software changes needed totake advantage of this new abstraction, and we illustrate itspower by showing improvements of 2-5× in latency and9× in throughput for a popular persistent NoSQL storerelative to a well-tuned Linux implementation.1 IntroductionReducing the overhead of the operating system processabstraction has been a longstanding goal of systems design.This issue has become particularly salient with modernclient-server computing. The combination of high speedEthernet and low latency persistent memories is considerablyraising the efficiency bar for I/O intensive software.Many servers spend much of their time executing operatingsystem code: delivering interrupts, demultiplexing andcopying network packets, and maintaining file systemmeta-data. Server applications often perform very simplefunctions, such as key-value table lookup and storage, yettraverse the OS kernel multiple times per client request.These trends have led to a long line of research aimedat optimizing kernel code paths for various use cases:eliminating redundant copies in the kernel [45], reducingthe overhead for large numbers of connections [27],protocol specialization [44], resource containers [8, 39],direct transfers between disk and network buffers [45],interrupt steering [46], system call batching [49], hardwareTCP acceleration, etc. Much of this has been adopted inmainline commercial OSes, and yet it has been a losingbattle: we show that the Linux network and file systemstacks have latency and throughput many times worse thanthat achieved by the raw hardware.Twenty years ago, researchers proposed streamliningpacket handling for parallel computing over a network ofworkstations by mapping the network hardware directlyinto user space [19, 22, 54]. Although commerciallyunsuccessful at the time, the virtualization market has nowled hardware vendors to revive the idea [6, 38, 48], andalso extend it to disks [52, 53].This paper explores the OS implications of removingthe kernel from the data path for nearly all I/O operations.We argue that doing this must provide applications withthe same security model as traditional designs; it is easy toget good performance by extending the trusted computingbase to include application code, e.g., by allowingapplications unfiltered direct access to the network/disk.We demonstrate that operating system protection is notcontradictory with high performance. For our prototypeimplementation, a client request to the Redis persistentNoSQL store has 2× better read latency, 5× better write latency,and 9× better write throughput compared to Linux.We make three specific contributions:• We give an architecture for the division of labor betweenthe device hardware, kernel, and runtime for directnetwork and disk I/O by unprivileged processes, andwe show how to efficiently emulate our model for I/Odevices that do not fully support virtualization (§3).• We implement a prototype of our model as a set ofmodifications to the open source Barrelfish operatingsystem, running on commercially available multi-corecomputers and I/O device hardware (§3.8).• We use our prototype to quantify the potential benefitsof user-level I/O for several widely used networkservices, including a distributed object cache, Redis, anIP-layer middlebox, and an HTTP load balancer (§4).We show that significant gains are possible in terms ofboth latency and scalability, relative to Linux, in manycases without modifying the application programminginterface; additional gains are possible by changing thePOSIX API (§4.3).2 BackgroundWe first give a detailed breakdown of the OS and applicationoverheads in network and storage operations today,followed by a discussion of current hardware technologiesthat support user-level networking and I/O virtualization.To analyze the sources of overhead, we recordtimestamps at various stages of kernel and user-space processing.Our experiments are conducted on a six machinecluster consisting of 6-core Intel Xeon E5-2430 (SandyBridge) systems at 2.2 GHz running Ubuntu Linux 13.04.1USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 1

LinuxArrakisReceiver running CPU idle Arrakis/P Arrakis/NNetwork stackin 1.26 (37.6%) 1.24 (20.0%) 0.32 (22.3%) 0.21 (55.3%)out 1.05 (31.3%) 1.42 (22.9%) 0.27 (18.7%) 0.17 (44.7%)Scheduler 0.17 (5.0%) 2.40 (38.8%) - -Copyin 0.24 (7.1%) 0.25 (4.0%) 0.27 (18.7%) -out 0.44 (13.2%) 0.55 (8.9%) 0.58 (40.3%) -Kernel crossingreturn 0.10 (2.9%) 0.20 (3.3%) - -syscall 0.10 (2.9%) 0.13 (2.1%) - -Total 3.36 (σ =0.66) 6.19 (σ =0.82) 1.44 (σ

AppApp UserspaceCore Core Core KernelIncoming Q's Outgoing Q's NICSystem call duration [us]9080706050403020100writefsyncbtrfs 1KBbtrfs 64Bext4 1KBext4 64Bext3 1KBext3 64Bext2 1KBext2 64BFigure 1: Linux networking architecture and workflow.removed, there is an opportunity to rethink the POSIX APIfor more streamlined networking. In addition to a POSIXcompatible interface, Arrakis provides a native interface(Arrakis/N) which supports true zero-copy I/O.2.2 Storage Stack OverheadsTo illustrate the overhead of today’s OS storage stacks,we conduct an experiment, where we execute small writeoperations immediately followed by an fsync 1 system callin a tight loop of 10,000 iterations, measuring each operation’slatency. We store the file system on a RAM disk,so the measured latencies represent purely CPU overhead.The overheads shown in Figure 2 stem from data copyingbetween user and kernel space, parameter and accesscontrol checks, block and inode allocation, virtualization(the VFS layer), snapshot maintenance (btrfs), as well asmetadata updates, in many cases via a journal [53].While historically these CPU overheads have beeninsignificant compared to disk access time, recent hardwaretrends have drastically reduced common-case writestorage latency by introducing flash-backed DRAM ontothe device. In these systems, OS storage stack overheadbecomes a major factor. We measured average writelatency to our RAID cache to be 25 µs. PCIe-attachedflash storage adapters, like Fusion-IO’s ioDrive2, reporthardware access latencies as low as 15 µs [24]. Incomparison, OS storage stack overheads are high, addingbetween 40% and 200% for the extended file systems,depending on journal use, and up to 5× for btrfs. The largestandard deviation for btrfs stems from its highly threadeddesign, used to flush non-critical file system metadata andupdate reference counts in the background.2.3 Application OverheadsWhat do these I/O stack overheads mean to operationlatencies within a typical datacenter application? Considerthe Redis [18] NoSQL store. Redis persists each write viaan operational log (called append-only file) 2 and servesreads from an in-memory data structure.To serve a read, Redis performs a series of operations:First, epoll is called to await data for reading, followed1 We also tried fdatasync, with negligible difference in latency.2 Redis also supports snapshot persistence because of the highper-operation overhead imposed by Linux.Figure 2: Average overhead in µs of various Linux file systemimplementations, when conducting small, persistent writes.Error bars show standard deviation.by recv to receive a request. After receiving, the (textual)request is parsed and the key looked-up in memory. Oncefound, a response is prepared and then, after epoll iscalled again to check whether the socket is ready, sentto the client via send. For writes, Redis additionallymarshals the operation into log format, writes the logand waits for persistence (via the fsync call) beforeresponding. Redis also spends time in accounting, accesschecks, and connection handling (Other row in Table 2).Table 2 shows that a total of 76% of the latency in anaverage read hit on Linux is due to socket operations.In Arrakis, we reduce socket operation latency by 68%.Similarly, 90% of the latency of a write on Linux is due toI/O operations. In Arrakis we reduce I/O latency by 82%.We can also see that Arrakis reduces some applicationleveloverheads. This is due to better cache behavior of theuser-level I/O stacks and the control/data plane separationevading all kernel crossings. Arrakis’ write latency is stilldominated by storage access latency (25µs in our system).We expect the gap between Linux and Arrakis performanceto widen as faster storage devices appear on the market.2.4 Hardware I/O VirtualizationSingle-Root I/O Virtualization (SR-IOV) [38] is ahardware technology intended to support high-speed I/Ofor multiple virtual machines sharing a single physicalmachine. An SR-IOV-capable I/O adapter appears on thePCIe interconnect as a single “physical function” (PCIparlance for a device) which can in turn dynamically createadditional “virtual functions”. Each of these resembles aPCI device, which can be directly mapped into a differentvirtual machine and access can be protected via IOMMU(e.g. Intel’s VT-d [34]). To the guest operating system,each virtual function can be programmed as if it wasa regular physical device, with a normal device driverand an unchanged I/O stack. Hypervisor software withaccess to the physical hardware (such as Domain 0 ina Xen [9] installation) creates and deletes these virtualfunctions, and configures filters in the SR-IOV adapterto demultiplex hardware operations to different virtualfunctions and therefore different guest operating systems.3USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 3

Read hitDurable writeLinux Arrakis/P Linux Arrakis/Pepoll 2.42 (27.91%) 1.12 (27.52%) 2.64 (1.62%) 1.49 (4.73%)recv 0.98 (11.30%) 0.29 (7.13%) 1.55 (0.95%) 0.66 (2.09%)Parse input 0.85 (9.80%) 0.66 (16.22%) 2.34 (1.43%) 1.19 (3.78%)Lookup/set key 0.10 (1.15%) 0.10 (2.46%) 1.03 (0.63%) 0.43 (1.36%)Log marshaling - - 3.64 (2.23%) 2.43 (7.71%)write - - 6.33 (3.88%) 0.10 (0.32%)fsync - - 137.84 (84.49%) 24.26 (76.99%)Prepare response 0.60 (6.92%) 0.64 (15.72%) 0.59 (0.36%) 0.10 (0.32%)send 3.17 (36.56%) 0.71 (17.44%) 5.06 (3.10%) 0.33 (1.05%)Other 0.55 (6.34%) 0.46 (11.30%) 2.12 (1.30%) 0.52 (1.65%)Total 8.67 (σ =2.55) 4.07 (σ =0.44) 163.14 (σ =13.68) 31.51 (σ =1.91)99th percentile 15.21 4.25 188.67 35.76Table 2: Overheads in the Redis NoSQL store for memory reads (hits) and durable writes (legend in Table 1).In Arrakis, we use SR-IOV, the IOMMU, and supportingadapters to provide direct application-level access to I/Odevices. This is a modern implementation of an idea whichwas implemented twenty years ago with U-Net [54], butgeneralized to flash storage and Ethernet network adapters.To make user-level I/O stacks tractable, we need ahardware-independent device model and API that capturesthe important features of SR-IOV adapters [31, 40, 41, 51];a hardware-specific device driver matches our API to thespecifics of the particular device. We discuss this modelin the next section, along with potential improvements tothe existing hardware to better support user-level I/O.Remote Direct Memory Access (RDMA) is anotherpopular model for user-level networking [48]. RDMAgives applications the ability to read from or write to aregion of virtual memory on a remote machine directlyfrom user-space, bypassing the operating system kernel onboth sides. The intended use case is for a parallel programto be able to directly read and modify its data structureseven when they are stored on remote machines.While RDMA provides the performance benefitsof user-level networking to parallel applications, it ischallenging to apply the model to a broader class of clientserverapplications [21]. Most importantly, RDMA ispoint-to-point. Each participant receives an authenticatorproviding it permission to remotely read/write a particularregion of memory. Since clients in client-server computingare not mutually trusted, the hardware would need to keepa separate region of memory for each active connection.Therefore we do not consider RDMA operations here.3 Design and ImplementationArrakis has the following design goals:• Minimize kernel involvement for data-plane operations:Arrakis is designed to limit or remove kernel mediationfor most I/O operations. I/O requests are routedto and from the application’s address space withoutrequiring kernel involvement and without sacrificingsecurity and isolation properties.• Transparency to the application programmer: Arrakisis designed to significantly improve performancewithout requiring modifications to applications writtento the POSIX API. Additional performance gains arepossible if the developer can modify the application.• Appropriate OS/hardware abstractions: Arrakis’ abstractionsshould be sufficiently flexible to efficientlysupport a broad range of I/O patterns, scale well on multicoresystems, and support application requirements forlocality and load balance.In this section, we show how we achieve these goals inArrakis. We describe an ideal set of hardware facilities thatshould be present to take full advantage of this architecture,and we detail the design of the control plane and dataplane interfaces that we provide to the application. Finally,we describe our implementation of Arrakis based on theBarrelfish operating system.3.1 Architecture OverviewArrakis targets I/O hardware with support for virtualization,and Figure 3 shows the overall architecture. In thispaper, we focus on hardware that can present multipleinstances of itself to the operating system and the applicationsrunning on the node. For each of these virtualizeddevice instances, the underlying physical device providesunique memory mapped register files, descriptor queues,and interrupts, hence allowing the control plane to mapeach device instance to a separate protection domain. Thedevice exports a management interface that is accessiblefrom the control plane in order to create or destroy virtualdevice instances, associate individual instances withnetwork flows or storage areas, and allocate shared resourcesto the different instances. Applications conduct I/O44 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

NICApplibosVNICSwitchApplibosVNICUserspaceVSICVSAVSAControlPlaneVSICVSAStorage ControllerFigure 3: Arrakis architecture. The storage controller mapsVSAs to physical storage.through their protected virtual device instance without requiringkernel intervention. In order to perform these operations,applications rely on a user-level I/O stack that is providedas a library. The user-level I/O stack can be tailored tothe application as it can assume exclusive access to a virtualizeddevice instance, allowing us to remove any featuresnot necessary for the application’s functionality. Finally,(de-)multiplexing operations and security checks are notneeded in this dedicated environment and can be removed.The user naming and protection model is unchanged.A global naming system is provided by the control plane.This is especially important for sharing stored data.Applications implement their own storage, while thecontrol plane manages naming and coarse-grain allocation,by associating each application with the directories andfiles it manages. Other applications can still read thosefiles by indirecting through the kernel, which hands thedirectory or read request to the appropriate application.3.2 Hardware ModelA key element of our work is to develop a hardwareindependentlayer for virtualized I/O—that is, a devicemodel providing an “ideal” set of hardware features.This device model captures the functionality requiredto implement in hardware the data plane operations of atraditional kernel. Our model resembles what is alreadyprovided by some hardware I/O adapters; we hope it willprovide guidance as to what is needed to support secureuser-level networking and storage.In particular, we assume our network devices providesupport for virtualization by presenting themselves asmultiple virtual network interface cards (VNICs) andthat they can also multiplex/demultiplex packets based oncomplex filter expressions, directly to queues that can bemanaged entirely in user space without the need for kernelintervention. Similarly, each storage controller exposesmultiple virtual storage interface controllers (VSICs)in our model. Each VSIC provides independent storagecommand queues (e.g., of SCSI or ATA format) that aremultiplexed by the hardware. Associated with each suchvirtual interface card (VIC) are queues and rate limiters.KernelVNICs also provide filters and VSICs provide virtualstorage areas. We discuss these components below.Queues: Each VIC contains multiple pairs of DMAqueues for user-space send and receive. The exact formof these VIC queues could depend on the specifics ofthe I/O interface card. For example, it could support ascatter/gather interface to aggregate multiple physicallydisjointmemory regions into a single data transfer. ForNICs, it could also optionally support hardware checksumoffload and TCP segmentation facilities. These featuresenable I/O to be handled more efficiently by performingadditional work in hardware. In such cases, the Arrakissystem offloads operations and further reduces overheads.Transmit and receive filters: A transmit filter is a predicateon network packet header fields that the hardwarewill use to determine whether to send the packet or discardit (possibly signaling an error either to the application orthe OS). The transmit filter prevents applications fromspoofing information such as IP addresses and VLANtags and thus eliminates kernel mediation to enforce thesesecurity checks. It can also be used to limit an applicationto communicate with only a pre-selected set of nodes.A receive filter is a similar predicate that determineswhich packets received from the network will be deliveredto a VNIC and to a specific queue associated with the targetVNIC. For example, a VNIC can be set up to receive allpackets sent to a particular port, so both connection setupand data transfers can happen at user-level. Installationof transmit and receive filters are privileged operationsperformed via the kernel control plane.Virtual storage areas: Storage controllers need to providean interface via their physical function to map virtualstorage areas (VSAs) to extents of physical drives, andassociate them with VSICs. A typical VSA will be largeenough to allow the application to ignore the underlyingmultiplexing—e.g., multiple erasure blocks on flash, orcylinder groups on disk. An application can store multiplesub-directories and files in a single VSA, providing precisecontrol over multi-object serialization constraints.A VSA is thus a persistent segment [13]. Applicationsreference blocks in the VSA using virtual offsets,converted by hardware into physical storage locations. AVSIC may have multiple VSAs, and each VSA may bemapped into multiple VSICs for interprocess sharing.Bandwidth allocators: This includes support for resourceallocation mechanisms such as rate limiters andpacing/traffic shaping of I/O. Once a frame has beenremoved from a transmit rate-limited or paced queue, thenext time another frame could be fetched from that queueis regulated by the rate limits and the inter-packet pacingcontrols associated with the queue. Installation of thesecontrols are also privileged operations.5USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 5

In addition, we assume that the I/O device driversupports an introspection interface allowing the controlplane to query for resource limits (e.g., the number ofqueues) and check for the availability of hardware supportfor I/O processing (e.g., checksumming or segmentation).Network cards that support SR-IOV have the keyelements of this model: they allow the creation of multipleVNICs that each may have multiple send and receivequeues, and support at least rudimentary transmit andreceive filters. Not all NICs provide the rich filtering semanticswe desire; for example, the Intel 82599 can filter onlybased on source or destination MAC addresses and VLANtags, not arbitrary predicates on header fields. However,this capability is within reach: some network cards (e.g.,Solarflare 10Gb adapters) can already filter packets on allheader fields, and the hardware support required for moregeneral VNIC transmit and receive filtering is closely relatedto that used for techniques like Receive-Side Scaling,which is ubiquitous in high-performance network cards.Storage controllers have some parts of the technologyneeded to provide the interface we describe. For example,RAID adapters have a translation layer that is ableto provide virtual disks above physical extents, andSSDs use a flash translation layer for wear-leveling.SCSI host-bus adapters support SR-IOV technology forvirtualization [40, 41] and can expose multiple VSICs,and the NVMe standard proposes multiple commandqueues for scalability [35]. Only the required protectionmechanism is missing. We anticipate VSAs to be allocatedin large chunks and thus hardware protection mechanismscan be coarse-grained and lightweight.Finally, the number of hardware-supported VICsmight be limited. The 82599 [31] and SAS3008 [41]support 64. This number is adequate with respect to thecapabilities of the rest of the hardware (e.g., the numberof CPU cores), but we expect it to rise. The PCI workinggroup has already ratified an addendum to SR-IOV thatincreases the supported number of virtual functions to2048. Bandwidth allocation within the 82599 is limitedto weighted round-robin scheduling and rate limiting ofeach of the 128 transmit/receive queues. Recent researchhas demonstrated that precise rate limiting in hardwarecan scale to tens of thousands of traffic classes, enablingsophisticated bandwidth allocation policies [47].Arrakis currently assumes hardware that can filterand demultiplex flows at a level (packet headers, etc.)corresponding roughly to a traditional OS API, but nohigher. An open question is the extent to which hardwarethat can filter on application-level properties (includingcontent) would provide additional performance benefits.3.3 VSIC EmulationTo validate our model given limited support from storagedevices, we developed prototype VSIC support bydedicating a processor core to emulate the functionalitywe expect from hardware. The same technique can be usedto run Arrakis on systems without VNIC support.To handle I/O requests from the OS, our RAID controllerprovides one request and one response descriptor queueof fixed size, implemented as circular buffers along witha software-controlled register (PR) pointing to the head ofthe request descriptor queue. Request descriptors (RQDs)have a size of 256 bytes and contain a SCSI command, ascatter-gather array of system memory ranges, and a targetlogical disk number. The SCSI command specifies the typeof operation (read or write), total transfer size and on-diskbase logical block address (LBA). The scatter-gather arrayspecifies the request’s corresponding regions in systemmemory. Response descriptors refer to completed RQDsby their queue entry and contain a completion code. AnRQD can be reused only after its response is received.We replicate this setup for each VSIC by allocatingqueue pairs and register files of the same format in systemmemory mapped into applications and to a dedicatedVSIC core. Like the 82599, we limit the maximum numberof VSICs to 64. In addition, the VSIC core keeps anarray of up to 4 VSA mappings for each VSIC that isprogrammable only from the control plane. The mappingscontain the size of the VSA and an LBA offset within alogical disk, effectively specifying an extent.In the steady state, the VSIC core polls each VSIC’sPR and the latest entry of the response queue of thephysical controller in a round-robin fashion. Whena new RQD is posted via PR i on VSIC i, the VSICcore interprets the RQD’s logical disk number n asa VSA mapping entry and checks whether the correspondingtransfer fits within that VSA’s boundaries (i.e.,RQD.LBA + RQD.size ≤ VSA n .size). If so, the corecopies the RQD to the physical controller’s queue, addingVSA n .offset to RQD.LBA, and sets an unused RQD fieldto identify the corresponding RQD in the source VSICbefore updating the controller’s PR register. Upon aresponse from the controller, the VSIC core copies theresponse to the corresponding VSIC response queue.We did not consider VSIC interrupts in our prototype.They can be supported via inter-processor interrupts.To support untrusted applications, our prototype has totranslate virtual addresses. This requires it to traverse applicationpage tables for each entry in an RQD’s scatter-gatherarray. In a real system, the IOMMU carries out this task.On a microbenchmark of 10,000 fixed size write operationsof 1KB via a single VSIC to a single VSA, the averageoverhead of the emulation is 3µs. Executing virtualizationcode takes 1µs on the VSIC core; the other 2µs are due tocache overheads that we did not quantify further. To measurethe expected VSIC performance with direct hardwaresupport, we map the single RAID hardware VSIC directlyinto the application memory; we report those results in §4.66 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

3.4 Control Plane InterfaceThe interface between an application and the Arrakiscontrol plane is used to request resources from the systemand direct I/O flows to and from user programs. Thekey abstractions presented by this interface are VICs,doorbells, filters, VSAs, and rate specifiers.An application can create and delete VICs, and associatedoorbells with particular events on particular VICs. Adoorbell is an IPC end-point used to notify the applicationthat an event (e.g. packet arrival or I/O completion) hasoccurred, and is discussed below. VICs are hardwareresources and so Arrakis must allocate them amongapplications according to an OS policy. Currently thisis done on a first-come-first-served basis, followed byspilling to software emulation (§3.3).Filters have a type (transmit or receive) and a predicatewhich corresponds to a convex sub-volume of thepacket header space (for example, obtained with a setof mask-and-compare operations). Filters can be usedto specify ranges of IP addresses and port numbersassociated with valid packets transmitted/received at eachVNIC. Filters are a better abstraction for our purposesthan a conventional connection identifier (such as aTCP/IP 5-tuple), since they can encode a wider variety ofcommunication patterns, as well as subsuming traditionalport allocation and interface specification.For example, in the “map” phase of a MapReduce jobwe would like the application to send to, and receive from,an entire class of machines using the same communicationend-point, but nevertheless isolate the data comprising theshuffle from other data. As a second example, web serverswith a high rate of incoming TCP connections can run intoscalability problems processing connection requests [46].In Arrakis, a single filter can safely express both a listeningsocket and all subsequent connections to that socket,allowing server-side TCP connection establishment toavoid kernel mediation.Applications create a filter with a control plane operation.In the common case, a simple higher-level wrappersuffices: filter = create_filter(flags, peerlist, servicelist).flags specifies the filter direction (transmit or receive)and whether the filter refers to the Ethernet, IP, TCP, orUDP header. peerlist is a list of accepted communicationpeers specified according to the filter type, and servicelistcontains a list of accepted service addresses (e.g., portnumbers) for the filter. Wildcards are permitted.The call to create_filter returns filter, a kernelprotectedcapability conferring authority to send orreceive packets matching its predicate, and which canthen be assigned to a specific queue on a VNIC. VSAs areacquired and assigned to VSICs in a similar fashion.Finally, a rate specifier can also be assigned to a queue,either to throttle incoming traffic (in the network receivecase) or pace outgoing packets and I/O requests. Ratespecifiers and filters associated with a VIC queue canbe updated dynamically, but all such updates requiremediation from the Arrakis control plane.Our network filters are less expressive than OpenFlowmatching tables, in that they do not support priority-basedoverlapping matches. This is a deliberate choice based onhardware capabilities: NICs today only support simplematching, and to support priorities in the API would leadto unpredictable consumption of hardware resourcesbelow the abstraction. Our philosophy is therefore tosupport expressing such policies only when the hardwarecan implement them efficiently.3.5 File Name LookupA design principle in Arrakis is to separate file namingfrom implementation. In a traditional system, the fullyqualifiedfilename specifies the file system used to storethe file and thus its metadata format. To work around this,many applications build their own metadata indirectioninside the file abstraction [28]. Instead, Arrakis providesapplications direct control over VSA storage allocation:an application is free to use its VSA to store metadata,directories, and file data. To allow other applications accessto its data, an application can export file and directorynames to the kernel virtual file system (VFS). To the rest ofthe VFS, an application-managed file or directory appearslike a remote mount point—an indirection to a file systemimplemented elsewhere. Operations within the file ordirectory are handled locally, without kernel intervention.Other applications can gain access to these files in threeways. By default, the Arrakis application library managingthe VSA exports a file server interface; other applicationscan use normal POSIX API calls via user-level RPC to theembedded library file server. This library can also run asa standalone process to provide access when the originalapplication is not active. Just like a regular mounted filesystem, the library needs to implement only functionalityrequired for file access on its VSA and may choose to skipany POSIX features that it does not directly support.Second, VSAs can be mapped into multiple processes.If an application, like a virus checker or backup system,has both permission to read the application’s metadata andthe appropriate library support, it can directly access thefile data in the VSA. In this case, access control is donefor the entire VSA and not per file or directory. Finally,the user can direct the originating application to exportits data into a standard format, such as a PDF file, storedas a normal file in the kernel-provided file system.The combination of VFS and library code implementPOSIX semantics seamlessly. For example, if executerights are revoked from a directory, the VFS preventsfuture traversal of that directory’s subtree, but existingRPC connections to parts of the subtree may remain intactuntil closed. This is akin to a POSIX process retaining a7USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 7

subdirectory as the current working directory—relativetraversals are still permitted.3.6 Network Data Plane InterfaceIn Arrakis, applications send and receive network packetsby directly communicating with hardware. The dataplane interface is therefore implemented in an applicationlibrary, allowing it to be co-designed with the application[43]. The Arrakis library provides two interfaces toapplications. We describe the native Arrakis interface,which departs slightly from the POSIX standard tosupport true zero-copy I/O; Arrakis also provides a POSIXcompatibility layer that supports unmodified applications.Applications send and receive packets on queues, whichhave previously been assigned filters as described above.While filters can include IP, TCP, and UDP field predicates,Arrakis does not require the hardware to perform protocolprocessing, only multiplexing. In our implementation, Arrakisprovides a user-space network stack above the dataplane interface. This stack is designed to maximize bothlatency and throughput. We maintain a clean separation betweenthree aspects of packet transmission and reception.Firstly, packets are transferred asynchronously betweenthe network and main memory using conventional DMAtechniques using rings of packet buffer descriptors.Secondly, the application transfers ownership of a transmitpacket to the network hardware by enqueuing a chainof buffers onto the hardware descriptor rings, and acquiresa received packet by the reverse process. This is performedby two VNIC driver functions. send_packet(queue,packet_array) sends a packet on a queue; the packet isspecified by the scatter-gather array packet_array, andmust conform to a filter already associated with the queue.receive_packet(queue) = packet receives a packet froma queue and returns a pointer to it. Both operations areasynchronous. packet_done(packet) returns ownershipof a received packet to the VNIC.For optimal performance, the Arrakis stack would interactwith the hardware queues not through these calls butdirectly via compiler-generated, optimized code tailored tothe NIC descriptor format. However, the implementationwe report on in this paper uses function calls to the driver.Thirdly, we handle asynchronous notification of eventsusing doorbells associated with queues. Doorbells aredelivered directly from hardware to user programs viahardware virtualized interrupts when applications arerunning and via the control plane to invoke the schedulerwhen applications are not running. In the latter case,higher latency is tolerable. Doorbells are exposed toArrakis programs via regular event delivery mechanisms(e.g., a file descriptor event) and are fully integratedwith existing I/O multiplexing interfaces (e.g., select).They are useful both to notify an application of generalavailability of packets in receive queues, as well as alightweight notification mechanism for I/O completionand the reception of packets in high-priority queues.This design results in a protocol stack that decoupleshardware from software as much as possible using thedescriptor rings as a buffer, maximizing throughput andminimizing overhead under high packet rates, yielding lowlatency. On top of this native interface, Arrakis providesPOSIX-compatible sockets. This compatibility layerallows Arrakis to support unmodified Linux applications.However, we show that performance gains can be achievedby using the asynchronous native interface.3.7 Storage Data Plane InterfaceThe low-level storage API provides a set of commandsto asynchronously read, write, and flush hardware cachesat any offset and of arbitrary size in a VSA via a commandqueue in the associated VSIC. To do so, the caller providesan array of virtual memory ranges (address and size)in RAM to be read/written, the VSA identifier, queuenumber, and matching array of ranges (offset and size)within the VSA. The implementation enqueues thecorresponding commands to the VSIC, coalescing andreordering commands if this makes sense to the underlyingmedia. I/O completion events are reported using doorbells.On top of this, a POSIX-compliant file system is provided.We have also designed a library of persistent data structures,Caladan, to take advantage of low-latency storagedevices. Persistent data structures can be more efficientthan a simple read/write interface provided by file systems.Their drawback is a lack of backwards-compatibility to thePOSIX API. Our design goals for persistent data structuresare that (1) operations are immediately persistent, (2) thestructure is robust versus crash failures, and (3) operationshave minimal latency.We have designed persistent log and queue datastructures according to these goals and modified a numberof applications to use them (e.g., §4.4). These datastructures manage all metadata required for persistence,which allows tailoring of that data to reduce latency. Forexample, metadata can be allocated along with each datastructure entry and persisted in a single hardware writeoperation. For the log and queue, the only metadata thatneeds to be kept is where they start and end. Pointerslink entries to accommodate wrap-arounds and holes,optimizing for linear access and efficient prefetch ofentries. By contrast, a filesystem typically has separateinodes to manage block allocation. The in-memory layoutof Caladan structures is as stored, eliminating marshaling.The log API includes operations to open and close a log,create log entries (for metadata allocation), append them tothe log (for persistence), iterate through the log (for reading),and trim the log. The queue API adds a pop operationto combine trimming and reading the queue. Persistenceis asynchronous: an append operation returns immediately88 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

with a callback on persistence. This allows us to maskremaining write latencies, e.g., by optimistically preparingnetwork responses to clients, while an entry is persisted.Entries are allocated in multiples of the storagehardware’s minimum transfer unit (MTU—512 bytesfor our RAID controller, based on SCSI) and contain aheader that denotes the true (byte-granularity) size of theentry and points to the offset of the next entry in a VSA.This allows entries to be written directly from memory,without additional marshaling. At the end of each entryis a marker that is used to determine whether an entrywas fully written (empty VSA space is always zero). Byissuing appropriate cache flush commands to the storagehardware, Caladan ensures that markers are written afterthe rest of the entry (cf. [17]).Both data structures are identified by a header at thebeginning of the VSA that contains a version number, thenumber of entries, the MTU of the storage device, and apointer to the beginning and end of the structure within theVSA. Caladan repairs a corrupted or outdated header lazilyin the background upon opening, by looking for additional,complete entries from the purported end of the structure.3.8 ImplementationThe Arrakis operating system is based upon a fork of theBarrelfish [10] multicore OS code base [1]. We added33,786 lines of code to the Barrelfish code base in orderto implement Arrakis. Barrelfish lends itself well to ourapproach, as it already provides a library OS. We couldhave also chosen to base Arrakis on the Xen [9] hypervisoror the Intel Data Plane Development Kit (DPDK) [32]running on Linux; both provide user-level access to thenetwork interface via hardware virtualization. However,implementing a library OS from scratch on top of amonolithic OS would have been more time consumingthan extending the Barrelfish library OS.We extended Barrelfish with support for SR-IOV, whichrequired modifying the existing PCI device manager to recognizeand handle SR-IOV extended PCI capabilities. Weimplemented a physical function driver for the Intel 8259910G Ethernet Adapter [31] that can initialize and managea number of virtual functions. We also implemented avirtual function driver for the 82599, including support forExtended Message Signaled Interrupts (MSI-X), which areused to deliver per-VNIC doorbell events to applications.Finally, we implemented drivers for the Intel IOMMU [34]and the Intel RS3 family of RAID controllers [33]. Inaddition—to support our benchmark applications—weadded several POSIX APIs that were not implemented inthe Barrelfish code base, such as POSIX threads, manyfunctions of the POSIX sockets API, as well as the epollinterface found in Linux to allow scalable polling of a largenumber of file descriptors. Barrelfish already supportsstandalone user-mode device drivers, akin to those foundin microkernels. We created shared library versions of thedrivers, which we link to each application.We have developed our own user-level network stack,Extaris. Extaris is a shared library that interfaces directlywith the virtual function device driver and provides thePOSIX sockets API and Arrakis’s native API to theapplication. Extaris is based in part on the low-levelpacket processing code of the lwIP network stack [42]. Ithas identical capabilities to lwIP, but supports hardwareoffload of layer 3 and 4 checksum operations and doesnot require any synchronization points or serializationof packet operations. We have also developed our ownstorage API layer, as described in §3.7 and our library ofpersistent data structures, Caladan.3.9 Limitations and Future WorkDue to the limited filtering support of the 82599 NIC,our implementation uses a different MAC address foreach VNIC, which we use to direct flows to applicationsand then do more fine-grain filtering in software, withinapplications. The availability of more general-purposefilters would eliminate this software overhead.Our implementation of the virtual function driverdoes not currently support the “transmit descriptor headwriteback” feature of the 82599, which reduces thenumber of PCI bus transactions necessary for transmitoperations. We expect to see a 5% network performanceimprovement from adding this support.The RS3 RAID controller we used in our experimentsdoes not support SR-IOV or VSAs. Hence, we use itsphysical function, which provides one hardware queue,and we map a VSA to each logical disk provided by thecontroller. We still use the IOMMU for protected accessto application virtual memory, but the controller does notprotect access to logical disks based on capabilities. Ourexperience with the 82599 suggests that hardware I/Ovirtualization incurs negligible performance overheadversus direct access to the physical function. We expectthis to be similar for storage controllers.4 EvaluationWe evaluate Arrakis on four cloud application workloads:a typical, read-heavy load pattern observed in many largedeployments of the memcached distributed object cachingsystem, a write-heavy load pattern to the Redis persistentNoSQL store, a workload consisting of a large numberof individual client HTTP requests made to a farm ofweb servers via an HTTP load balancer and, finally, thesame benchmark via an IP-layer middlebox. We alsoexamine the system under maximum load in a series ofmicrobenchmarks and analyze performance crosstalkamong multiple networked applications. Using theseexperiments, we seek to answer the following questions:9USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 9

• What are the major contributors to performanceoverhead in Arrakis and how do they compare to thoseof Linux (presented in §2)?• Does Arrakis provide better latency and throughputfor real-world cloud applications? How does thethroughput scale with the number of CPU cores forthese workloads?• Can Arrakis retain the benefits of user-level applicationexecution and kernel enforcement, while providinghigh-performance packet-level network IO?• What additional performance gains are possible bydeparting from the POSIX interface?We compare the performance of the following OSconfigurations: Linux kernel version 3.8 (Ubuntu version13.04), Arrakis using the POSIX interface (Arrakis/P),and Arrakis using its native interface (Arrakis/N).We tuned Linux network performance by installing thelatest ixgbe device driver version 3.17.3 and disablingreceive side scaling (RSS) when applications execute ononly one processor. RSS spreads packets over several NICreceive queues, but incurs needless coherence overhead ona single core. The changes yield a throughput improvementof 10% over non-tuned Linux. We use the kernel-shippedMegaRAID driver version 6.600.18.00-rc1.Linux uses a number of performance-enhancingfeatures of the network hardware, which Arrakis doesnot currently support. Among these features is the useof direct processor cache access by the NIC, TCP andUDP segmentation offload, large receive offload, andnetwork packet header splitting. All of these featurescan be implemented in Arrakis; thus, our performancecomparison is weighted in favor of Linux.4.1 Server-side Packet Processing PerformanceWe load the UDP echo benchmark from §2 on the serverand use all other machines in the cluster as load generators.These generate 1 KB UDP packets at a fixed rate and recordthe rate at which their echoes arrive. Each experimentexposes the server to maximum load for 20 seconds.Shown in Table 1, compared to Linux, Arrakis eliminatestwo system calls, software demultiplexing overhead,socket buffer locks, and security checks. In Arrakis/N, weadditionally eliminate two socket buffer copies. Arrakis/Pincurs a total server-side overhead of 1.44 µs, 57% lessthan Linux. Arrakis/N reduces this overhead to 0.38 µs.The echo server is able to add a configurable delaybefore sending back each packet. We use this delay tosimulate additional application-level processing time atthe server. Figure 4 shows the average throughput attainedby each system over various such delays; the theoreticalline rate is 1.26M pps with zero processing.In the best case (no additional processing time),Arrakis/P achieves 2.3× the throughput of Linux. ByThroughput [k packets / s]120010008006004002000LinuxArrakis/PArrakis/NDriver0 1 2 4 8 16 32 64Processing time [us]Figure 4: Average UDP echo throughput for packets with 1024byte payload over various processing times. The top y-axis valueshows theoretical maximum throughput on the 10G network.Error bars in this and following figures show min/max measuredover 5 repeats of the experiment.departing from POSIX, Arrakis/N achieves 3.9× thethroughput of Linux. The relative benefit of Arrakisdisappears at 64 µs. To gauge how close Arrakis comesto the maximum possible throughput, we embedded aminimal echo server directly into the NIC device driver,eliminating any remaining API overhead. Arrakis/Nachieves 94% of the driver limit.4.2 Memcached Key-Value StoreMemcached is an in-memory key-value store used bymany cloud applications. It incurs a processing overheadof 2–3 µs for an average object fetch request, comparableto the overhead of OS kernel network processing.We benchmark memcached 1.4.15 by sending itrequests at a constant rate via its binary UDP protocol,using a tool similar to the popular memslap benchmark [2].We configure a workload pattern of 90% fetch and 10%store requests on a pre-generated range of 128 differentkeys of a fixed size of 64 bytes and a value size of 1 KB,in line with real cloud deployments [7].To measure network stack scalability for multiple cores,we vary the number of memcached server processes. Eachserver process executes independently on its own portnumber, such that measurements are not impacted by scalabilitybottlenecks in memcached itself, and we distributeload equally among the available memcached instances.On Linux, memcached processes share the kernel-levelnetwork stack. On Arrakis, each process obtains its ownVNIC with an independent set of packet queues, eachcontrolled by an independent instance of Extaris.Figure 5 shows that memcached on Arrakis/P achieves1.7× the throughput of Linux on one core, and attains nearline-rate at 4 CPU cores. The slightly lower throughputon all 6 cores is due to contention with Barrelfishsystem management processes [10]. By contrast, Linuxthroughput nearly plateaus beyond two cores. A single,multi-threaded memcached instance shows no noticeablethroughput difference to the multi-process scenario. Thisis not surprising as memcached is optimized to scale well.1010 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

Throughput [k transactions / s]120010008006004002000Linux threadsLinux procsArrakis/P1 2 4 6Number of CPU coresFigure 5: Average memcached transaction throughput andscalability. Top y-axis value = 10Gb/s.To conclude, the separation of network stack and applicationin Linux provides only limited information aboutthe application’s packet processing and poses difficulty assigningthreads to the right CPU core. The resulting cachemisses and socket lock contention are responsible for muchof the Linux overhead. In Arrakis, the application is in controlof the whole packet processing flow: assignment ofpackets to packet queues, packet queues to cores, and finallythe scheduling of its own threads on these cores. Thenetwork stack thus does not need to acquire any locks, andpacket data is always available in the right processor cache.Memcached is also an excellent example of the communicationendpoint abstraction: we can create hardwarefilters to allow packet reception and transmission onlybetween the memcached server and a designated list ofclient machines that are part of the cloud application. In theLinux case, we have to filter connections in the application.4.3 Arrakis Native Interface Case StudyAs a case study, we modified memcached to make useof Arrakis/N. In total, 74 lines of code were changed,with 11 pertaining to the receive side, and 63 to the sendside. On the receive side, the changes involve eliminatingmemcached’s receive buffer and working directly withpointers to packet buffers provided by Extaris, as wellas returning completed buffers to Extaris. The changesincrease average throughput by 9% over Arrakis/P. On thesend side, changes include allocating a number of sendbuffers to allow buffering of responses until fully sentby the NIC, which now must be done within memcacheditself. They also involve the addition of reference countsto hash table entries and send buffers to determine whenit is safe to reuse buffers and hash table entries that mightotherwise still be processed by the NIC. We gain anadditional 10% average throughput when using the sendside API in addition to the receive side API.4.4 Redis NoSQL StoreRedis [18] extends the memcached model from a cacheto a persistent NoSQL object store. Our results in Table 2show that Redis operations—while more laborious thanMemcached—are still dominated by I/O stack overheads.Throughput [k transactions / s]300250200150100500GETLinuxArrakis/PArrakis/P [15us]Linux/CaladanSETFigure 6: Average Redis transaction throughput for GET andSET operations. The Arrakis/P [15us] and Linux/Caladanconfigurations apply only to SET operations.Redis can be used in the same scenario as Memcachedand we follow an identical experiment setup, using Redisversion 2.8.5. We use the benchmarking tool distributedwith Redis and configure it to execute GET and SETrequests in two separate benchmarks to a range of 65,536random keys with a value size of 1,024 bytes, persistingeach SET operation individually, with a total concurrencyof 1,600 connections from 16 benchmark clients executingon the client machines. Redis is single-threaded, so weinvestigate only single-core performance.The Arrakis version of Redis uses Caladan. We changed109 lines in the application to manage and exchangerecords with the Caladan log instead of a file. We did noteliminate Redis’ marshaling overhead (cf. Table 2). If wedid, we would save another 2.43 µs of write latency. Dueto the fast I/O stacks, Redis’ read performance mirrors thatof Memcached and write latency improves by 63%, whilewrite throughput improves vastly, by 9×.To investigate what would happen if we had accessto state-of-the-art storage hardware, we simulate (via awrite-delaying RAM disk) a storage backend with 15 µswrite latency, such as the ioDrive2 [24]. Write throughputimproves by another 1.6×, nearing Linux read throughput.Both network and disk virtualization is needed for goodRedis performance. We tested this by porting Caladan torun on Linux, with the unmodified Linux network stack.This improved write throughput by only 5× compared toLinux, compared to 9× on Arrakis.Together, the combination of data-plane network andstorage stacks can yield large benefits in latency andthroughput for both read and write-heavy workloads.The tight integration of storage and data structure inCaladan allows for a number of latency-saving techniquesthat eliminate marshaling overhead, book-keeping ofjournals for file system metadata, and can offset storageallocation overhead. These benefits will increase furtherwith upcoming hardware improvements.4.5 HTTP Load BalancerTo aid scalability of web services, HTTP load balancersare often deployed to distribute client load over a number11USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 11

Throughput [k transactions / s]160140120100806040200LinuxLinux (SEPOLL)Arrakis/P1 2 4 6Number of CPU coresThroughput [k transactions / s]250200150100500LinuxArrakis/P1 2 4Number of CPU coresFigure 7: Average HTTP transaction throughput and scalabilityof haproxy.of web servers. A popular HTTP load balancer employedby many web and cloud services, such as Amazon EC2 andTwitter, is haproxy [3]. In these settings, many connectionsare constantly opened and closed and the OS needs tohandle the creation and deletion of the associated socketdata structures.To investigate how performance is impacted when manyconnections need to be maintained, we configure fiveweb servers and one load balancer. To minimize overheadat the web servers, we deploy a simple static web pageof 1,024 bytes, served out of main memory.These sameweb server hosts also serve as workload generators, usingApacheBench version 2.3 to conduct as many concurrentrequests for the web page as possible. Each request isencapsulated in its own TCP connection. On the load balancerhost, we deploy haproxy version 1.4.24, configuredto distribute incoming load in a round-robin fashion. Werun multiple copies of the haproxy process on the loadbalancing node, each executing on their own port number.We configure the ApacheBench instances to distributetheir load equally among the available haproxy instances.Haproxy relies on cookies, which it inserts into theHTTP stream to remember connection assignments tobackend web servers under possible client re-connects.This requires it to interpret the HTTP stream for eachclient request. Linux provides an optimization called TCPsplicing that allows applications to forward traffic betweentwo sockets without user-space involvement. This reducesthe overhead of kernel crossings when connections arelong-lived. We enable haproxy to use this feature on Linuxwhen beneficial.Finally, haproxy contains a feature known as “speculativeepoll” (SEPOLL), which uses knowledge abouttypical socket operation flows within the Linux kernel toavoid calls to the epoll interface and optimize performance.Since the Extaris implementation differs from that of theLinux kernel network stack, we were not able to use thisinterface on Arrakis, but speculate that this feature could beported to Arrakis to yield similar performance benefits. Toshow the effect of the SEPOLL feature, we repeat the Linuxbenchmark both with and without it and show both results.Figure 8: Average HTTP transaction throughput and scalabilityof the load balancing middlebox. Top y-axis value = 10Gb/s.In Figure 7, we can see that Arrakis outperforms Linux inboth regular and SEPOLL configurations on a single core,by a factor of 2.2 and 2, respectively. Both systems showsimilar scalability curves. Note that Arrakis’s performanceon 6 CPUs is affected by background activity on Barrelfish.To conclude, connection oriented workloads requirea higher number of system calls for setup (accept andsetsockopt) and teardown (close). In Arrakis, wecan use filters, which require only one control planeinteraction to specify which clients and servers maycommunicate with the load balancer service. Furthersocket operations are reduced to function calls in thelibrary OS, with lower overhead.4.6 IP-layer MiddleboxIP-layer middleboxes are ubiquitous in today’s wide areanetworks (WANs). Common middleboxes perform tasks,such as firewalling, intrusion detection, network addresstranslation, and load balancing. Due to the complexity oftheir tasks, middleboxes can benefit from the programmingand run-time convenience provided by an OS through itsabstractions for safety and resource management.We implemented a simple user-level load balancingmiddlebox using raw IP sockets [5]. Just like haproxy,the middlebox balances an incoming TCP workload toa set of back-end servers. Unlike haproxy, it is operatingcompletely transparently to the higher layer protocols.It simply rewrites source and destination IP addressesand TCP port numbers contained in the packet headers.It monitors active TCP connections and uses a hash tableto remember existing connection assignments. Responsesby the back-end web servers are also intercepted andforwarded back to the corresponding clients. This issufficient to provide the same load balancing capabilitiesas in the haproxy experiment. We repeat the experimentfrom §4.5, replacing haproxy with our middlebox.The simpler nature of the middlebox is reflected in thethroughput results (see Figure 8). Both Linux and Arrakisperform better. Because the middlebox performs lessapplication-level work than haproxy, performance factorsare largely due to OS-level network packet processing.As a consequence, Arrakis’ benefits are more prominent,1212 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

Throughput [k transactions / s]1400120010008006004002000Arrakis/P Linux Arrakis/P LinuxNo limit100Mbit/s limitFigure 9: Memcached transaction throughput over 5 instances(colors), with and without rate limiting.and its performance is 2.6× that of Linux. We also seean interesting effect: the Linux implementation does notscale at all in this configuration. The reason for this arethe raw IP sockets, which carry no connection information.Without an indication of which connections to steer towhich sockets, each middlebox instance has to look at eachincoming packet to determine whether it should handle it.This added overhead outweighs any performance gainedvia parallelism. In Arrakis, we can configure the hardwarefilters to steer packets based on packet header informationand thus scale until we quickly hit the NIC throughputlimit at two cores.We conclude that Arrakis allows us to retain the safety,abstraction, and management benefits of software developmentat user-level, while vastly improving the performanceof low level packet operations. Filters provide a versatileinterface to steer packet workloads based on arbitraryinformation stored in packet headers to effectively leveragemulti-core parallelism, regardless of protocol specifics.4.7 Performance IsolationWe show that QoS limits can be enforced in Arrakis,by simulating a simple multi-tenant scenario with 5memcached instances pinned to distinct cores, to minimizeprocessor crosstalk. One tenant has an SLA that allowsit to send up to 100Mb/s. The other tenants are not limited.We use rate specifiers in Arrakis to set the transmit ratelimit of the VNIC of the limited process. On Linux, weuse queuing disciplines [29] (specifically, HTB [20]) torate limit the source port of the equivalent process.We repeat the experiment from §4.2, plotting thethroughput achieved by each memcached instance, shownin Figure 9. The bottom-most process (barely visible) israte-limited to 100Mb/s in the experiment shown on theright hand side of the figure. All runs remained withinthe error bars shown in Figure 5. When rate-limiting, abit of the total throughput is lost for both OSes becauseclients keep sending packets at the same high rate. Theseconsume network bandwidth, even when later droppeddue to the rate limit.We conclude that it is possible to provide the same kindof QoS enforcement—in this case, rate limiting—in Arrakis,as in Linux. Thus, we are able to retain the protectionand policing benefits of user-level application execution,while providing improved network performance.5 DiscussionIn this section, we discuss how we can extend the Arrakismodel to apply to virtualized guest environments, as wellas to interprocessor interrupts.5.1 Arrakis as Virtualized GuestArrakis’ model can be extended to virtualized environments.Making Arrakis a host in this environmentis straight-forward—this is what the technology wasoriginally designed for. The best way to support Arrakis asa guest is by moving the control plane into the virtual machinemonitor (VMM). Arrakis guest applications can thenallocate virtual interface cards directly from the VMM.A simple way of accomplishing this is by pre-allocating anumber of virtual interface cards in the VMM to the guestand let applications pick only from this pre-allocated set,without requiring a special interface to the VMM.The hardware limits apply to a virtualized environmentin the same way as they do in the regular Arrakisenvironment. We believe the current limits on virtualadapters (typically 64) to be balanced with the number ofavailable processing resources.5.2 Virtualized Interprocessor InterruptsTo date, most parallel applications are designed assumingthat shared-memory is (relatively) efficient, whileinterprocessor signaling is (relatively) inefficient. A cachemiss to data written by another core is handled in hardware,while alerting a thread on another processor requireskernel mediation on both the sending and receiving side.The kernel is involved even when signaling an eventbetween two threads running inside the same application.With kernel bypass, a remote cache miss and a remoteevent delivery are similar in cost at a physical level.Modern hardware already provides the operating systemthe ability to control how device interrupts are routed. Tosafely deliver an interrupt within an application, withoutkernel mediation, requires that the hardware add accesscontrol. With this, the kernel could configure the interruptrouting hardware to permit signaling among cores runningthe same application, trapping to the kernel only whensignaling between different applications.6 Related WorkSPIN [14] and Exokernel [25] reduced shared kernelcomponents to allow each application to have customizedoperating system management. Nemesis [15] reducesshared components to provide more performance isolationfor multimedia applications. All three mediated I/O inthe kernel. Relative to these systems, Arrakis shows that13USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 13

application customization is consistent with very highperformance.Following U-Net, a sequence of hardware standardssuch as VIA [19] and Infiniband [30] addressed thechallenge of minimizing, or eliminating entirely, operatingsystem involvement in sending and receiving networkpackets in the common case. To a large extent, thesesystems have focused on the needs of parallel applicationsfor high throughout, low overhead communication.Arrakis supports a more general networking modelincluding client-server and peer-to-peer communication.Our work was inspired in part by previous work onDune [11], which used nested paging to provide supportfor user-level control over virtual memory, and ExitlessIPIs [26], which presented a technique to demultiplexhardware interrupts between virtual machines withoutmediation from the virtual machine monitor.Netmap [49] implements high throughput networkI/O by doing DMAs directly from user space. Sends andreceives still require system calls, as the OS needs to do permissionchecks on every operation. Throughput is achievedat the expense of latency, by batching reads and writes.Similarly, IX [12] implements a custom, per-applicationnetwork stack in a protected domain accessed with batchedsystem calls. Arrakis eliminates the need for batching byhandling operations at user level in the common case.Concurrently with our work, mTCP uses Intel’s DPDKinterface to implement a scalable user-level TCP [36];mTCP focuses on scalable network stack design, whileour focus is on the operating system API for general clientserverapplications. We expect the performance of Extarisand mTCP to be similar. OpenOnload [50] is a hybrid userandkernel-level network stack. It is completely binarycompatiblewith existing Linux applications; to supportthis, it has to keep a significant amount of socket state in thekernel and supports only a traditional socket API. Arrakis,in contrast, allows applications to access the networkhardware directly and does not impose API constraints.Recent work has focused on reducing the overheadsimposed by traditional file systems and block devicedrivers, given the availability of low latency persistentmemory. DFS [37] and PMFS [23] are file systemsdesigned for these devices. DFS relies on the flash storagelayer for functionality traditionally implemented inthe OS, such as block allocation. PMFS exploits thebyte-addressability of persistent memory, avoiding theblock layer. Both DFS and PMFS are implemented askernel-level file systems, exposing POSIX interfaces.They focus on optimizing file system and device driverdesign for specific technologies, while Arrakis investigateshow to allow applications fast, customized device access.Moneta-D [16] is a hardware and software platform forfast, user-level I/O to solid-state devices. The hardware andoperating system cooperate to track permissions on hardwareextents, while a user-space driver communicates withthe device through a virtual interface. Applications interactwith the system through a traditional file system. Moneta-D is optimized for large files, since each open operationrequires communication with the OS to check permissions;Arrakis does not have this issue, since applications havecomplete control over their VSAs. Aerie [53] proposesan architecture in which multiple processes communicatewith a trusted user-space file system service for filemetadata and lock operations, while directly accessing thehardware for reads and data-only writes. Arrakis providesmore flexibility than Aerie, since storage solutions can beintegrated tightly with applications rather than providedin a shared service, allowing for the development ofhigher-level abstractions, such as persistent data structures.7 ConclusionIn this paper, we described and evaluated Arrakis, a newoperating system designed to remove the kernel from theI/O data path without compromising process isolation.Unlike a traditional operating system, which mediates allI/O operations to enforce process isolation and resourcelimits, Arrakis uses device hardware to deliver I/O directlyto a customized user-level library. The Arrakis kerneloperates in the control plane, configuring the hardwareto limit application misbehavior.To demonstrate the practicality of our approach, we haveimplemented Arrakis on commercially available networkand storage hardware and used it to benchmark several typicalserver workloads. We are able to show that protectionand high performance are not contradictory: end-to-endclient read and write latency to the Redis persistent NoSQLstore is 2–5× faster and write throughput 9× higher onArrakis than on a well-tuned Linux implementation.AcknowledgmentsThis work was supported by NetApp, Google, and theNational Science Foundation. We would like to thank theanonymous reviewers and our shepherd, Emmett Witchel,for their comments and feedback. We also thank OlegGodunok for implementing the IOMMU driver, AntoineKaufmann for implementing MSI-X support, and TaesooKim for implementing interrupt support into Extaris.References[1] http://www.barrelfish.org/.[2] http://www.libmemcached.org/.[3] http://haproxy.1wt.eu.[4] Scaling in the Linux networking stack. https://www.kernel.org/doc/Documentation/networking/scaling.txt.1414 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

[5] Linux IPv4 raw sockets, May 2012. http://man7.org/linux/man-pages/man7/raw.7.html.[6] D. Abramson. Intel virtualization technology for directedI/O. Intel Technology Journal, 10(3):179–192,2006.[7] B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, andM. Paleczny. Workload analysis of a large-scalekey-value store. In SIGMETRICS, 2012.[8] G. Banga, P. Druschel, and J. C. Mogul. Resourcecontainers: A new facility for resource managementin server systems. In OSDI, 1999.[9] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris,A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xenand the art of virtualization. In SOSP, 2003.[10] A. Baumann, P. Barham, P.-E. Dagand, T. Harris,R. Isaacs, S. Peter, T. Roscoe, A. Schüpbach, andA. Singhania. The multikernel: a new OS architecturefor scalable multicore systems. In SOSP, 2009.[11] A. Belay, A. Bittau, A. Mashtizadeh, D. Terei,D. Mazières, and C. Kozyrakis. Dune: Safe user-levelaccess to privileged CPU features. In OSDI, 2012.[12] A. Belay, G. Prekas, A. Klimovic, S. Grossman,C. Kozyrakis, and E. Bugnion. IX: A protecteddataplane operating system for high throughput andlow latency. In OSDI, 2014.[13] A. Bensoussan, C. Clingen, and R. Daley. TheMultics virtual memory: Concepts and design.CACM, 15:308–318, 1972.[14] B. N. Bershad, S. Savage, P. Pardyak, E. G. Sirer,M. E. Fiuczynski, D. Becker, C. Chambers, andS. Eggers. Extensibility, safety and performance inthe SPIN operating system. In SOSP, 1995.[15] R. Black, P. T. Barham, A. Donnelly, and N. Stratford.Protocol implementation in a vertically structuredoperating system. In LCN, 1997.[16] A. M. Caulfield, T. I. Mollov, L. A. Eisner, A. De,J. Coburn, and S. Swanson. Providing safe, userspace access to fast, solid state disks. ASPLOS, 2012.[17] V. Chidambaram, T. S. Pillai, A. C. Arpaci-Dusseau,and R. H. Arpaci-Dusseau. Optimistic crashconsistency. In SOSP, 2013.[18] Citrusbyte. Redis. http://redis.io/.[19] Compaq Computer Corp., Intel Corporation, and MicrosoftCorporation. Virtual Interface ArchitectureSpecification, version 1.0 edition, December 1997.[20] M. Devera. HTB Linux queuing discipline manual– User Guide, May 2002. http://luxik.cdi.cz/~devik/qos/htb/userg.pdf.[21] A. Dragojević, D. Narayanan, M. Castro, and O. Hodson.FaRM: Fast remote memory. In NSDI, 2014.[22] P. Druschel, L. Peterson, and B. Davie. Experienceswith a high-speed network adaptor: A softwareperspective. In SIGCOMM, 1994.[23] S. R. Dulloor, S. Kumar, A. Keshavamurthy, P. Lantz,D. Reddy, R. Sankaran, and J. Jackson. Systemsoftware for persistent memory. In EuroSys, 2014.[24] Fusion-IO. ioDrive2 and ioDrive2 Duo Multi LevelCell, 2014. Product Datasheet. http://www.fusionio.com/load/-media-/2rezss/docsLibrary/FIO_DS_ioDrive2.pdf.[25] G. R. Ganger, D. R. Engler, M. F. Kaashoek, H. M.Briceño, R. Hunt, and T. Pinckney. Fast and flexibleapplication-level networking on Exokernel systems.TOCS, 20(1):49–83, Feb 2002.[26] A. Gordon, N. Amit, N. Har’El, M. Ben-Yehuda,A. Landau, A. Schuster, and D. Tsafrir. ELI:bare-metal performance for I/O virtualization. InASPLOS, 2012.[27] S. Han, S. Marshall, B.-G. Chun, and S. Ratnasamy.MegaPipe: a new programming interface for scalablenetwork I/O. In OSDI, 2012.[28] T. Harter, C. Dragga, M. Vaughn, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. A file is nota file: Understanding the I/O behavior of Appledesktop applications. In SOSP, 2011.[29] B. Hubert. Linux advanced routing & trafficcontrol HOWTO. http://www.lartc.org/howto/.[30] Infiniband Trade Organization. Introductionto Infiniband for end users. https://cw.infinibandta.org/document/dl/7268,April 2010.[31] Intel Corporation. Intel 82599 10 GbE ControllerDatasheet, December 2010. Revision 2.6. http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf.[32] Intel Corporation. Intel Data Plane DevelopmentKit (Intel DPDK) Programmer’s Guide, Aug 2013.Reference Number: 326003-003.15USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 15

[33] Intel Corporation. Intel RAID ControllersRS3DC080 and RS3DC040, Aug 2013.Product Brief. http://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/raid-controller-rs3dc-brief.pdf.[34] Intel Corporation. Intel virtualization technology fordirected I/O architecture specification. Technical ReportOrder Number: D51397-006, Intel Corporation,Sep 2013.[35] Intel Corporation. NVM Express, revision1.1a edition, Sep 2013. http://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1a.pdf.[36] E. Jeong, S. Woo, M. Jamshed, H. J. S. Ihm, D. Han,and K. Park. mTCP: A Highly Scalable User-levelTCP Stack for Multicore Systems. In NSDI, 2014.[37] W. K. Josephson, L. A. Bongo, K. Li, and D. Flynn.DFS: A file system for virtualized flash storage.Trans. Storage, 6(3):14:1–14:25, Sep 2010.[38] P. Kutch. PCI-SIG SR-IOV primer: An introductionto SR-IOV technology. Intel application note,321211–002, Jan 2011.[39] I. M. Leslie, D. McAuley, R. Black, T. Roscoe,P. Barham, D. Evers, R. Fairbairns, and E. Hyden.The design and implementation of an operating systemto support distributed multimedia applications.IEEE J.Sel. A. Commun., 14(7):1280–1297, Sep2006.[40] LSI Corporation. LSISAS2308 PCI Expressto 8-Port 6Gb/s SAS/SATA Controller,Feb 2010. Product Brief. http://www.lsi.com/downloads/Public/SAS%20ICs/LSI_PB_SAS2308.pdf.[45] V. S. Pai, P. Druschel, and W. Zwanepoel. IO-Lite: Aunified I/O buffering and caching system. In OSDI,1999.[46] A. Pesterev, J. Strauss, N. Zeldovich, and R. T.Morris. Improving network connection locality onmulticore systems. In EuroSys, 2012.[47] S. Radhakrishnan, Y. Geng, V. Jeyakumar, A. Kabbani,G. Porter, and A. Vahdat. SENIC: ScalableNIC for end-host rate limiting. In NSDI, 2014.[48] RDMA Consortium. Architectural specificationsfor RDMA over TCP/IP. http://www.rdmaconsortium.org/.[49] L. Rizzo. Netmap: A novel framework for fastpacket I/O. In USENIX ATC, 2012.[50] SolarFlare Communications, Inc. OpenOnload.http://www.openonload.org/.[51] Solarflare Communications, Inc. SolarflareSFN5122F Dual-Port 10GbE Enterprise ServerAdapter, 2010.[52] A. Trivedi, P. Stuedi, B. Metzler, R. Pletka, B. G.Fitch, and T. R. Gross. Unified high-performanceI/O: One stack to rule them all. In HotOS, 2013.[53] H. Volos, S. Nalli, S. Panneerselvam, V. Varadarajan,P. Saxena, and M. M. Swift. Aerie: Flexiblefile-system interfaces to storage-class memory. InEuroSys, 2014.[54] T. von Eicken, A. Basu, V. Buch, and W. Vogels.U-Net: A user-level network interface for paralleland distributed computing. In SOSP, 1995.[41] LSI Corporation. LSISAS3008 PCI Expressto 8-Port 12Gb/s SAS/SATA Controller,Feb 2014. Product Brief. http://www.lsi.com/downloads/Public/SAS%20ICs/LSI_PB_SAS3008.pdf.[42] lwIP. http://savannah.nongnu.org/projects/lwip/.[43] I. Marinos, R. N. M. Watson, and M. Handley.Network stack specialization for performance. InSIGCOMM, 2014.[44] D. Mosberger and L. L. Peterson. Making paths explicitin the Scout operating system. In OSDI, 1996.1616 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

Decoupling Cores, Kernels, and Operating SystemsGerd Zellweger, Simon Gerber, Kornilios Kourtis, Timothy RoscoeSystems Group, Department of Computer Science, ETH ZurichAbstractWe present Barrelfish/DC, an extension to the BarrelfishOS which decouples physical cores from a nativeOS kernel, and furthermore the kernel itself from the restof the OS and application state. In Barrelfish/DC, nativekernel code on any core can be quickly replaced, kernelstate moved between cores, and cores added and removedfrom the system transparently to applications and OSprocesses, which continue to execute.Barrelfish/DC is a multikernel with two novel ideas: theuse of boot drivers to abstract cores as regular devices, anda partitioned capability system for memory managementwhich externalizes core-local kernel state.We show by performance measurements of real applicationsand device drivers that the approach is practicalenough to be used for a number of purposes, such asonline kernel upgrades, and temporarily delivering hardreal-time performance by executing a process under aspecialized, single-application kernel.1 IntroductionThe hardware landscape is increasingly dynamic. Futuremachines will contain large numbers of heterogeneouscores which will be powered on and off individually inresponse to workload changes. Cores themselves willhave porous boundaries: some may be dynamically fusedor split to provide more energy-efficient computation. ExistingOS designs like Linux and Windows assume a staticnumber of homogeneous cores, with recent extensions toallow core hotplugging.We present Barrelfish/DC, an OS design based on theprinciple that all cores are fully dynamic. Barrelfish/DCis based on the Barrelfish research OS [5] and exploitsthe “multikernel” architecture to separate the OS statefor each core. We show that Barrelfish/DC can handledynamic cores more flexibly and with far less overheadthan Linux, and also that the approach brings additionalbenefits in functionality.A key challenge with dynamic cores is safely disposingof per-core OS state when removing a core from thesystem: this process takes time and can dominate the hardwarelatency of powering the core down, reducing anybenefit in energy consumption. Barrelfish/DC addressesthis challenge by externalizing all the per-core OS andapplication state of a system into objects called OSnodes,which can be executed lazily on another core. Whilethis general idea has been proposed before (notably, it isused in Chameleon [37] to clean up interrupt state), Barrelfish/DCtakes the concept much further in completelydecoupling the OSnode from the kernel, and this in turnfrom the physical core.While transparent to applications, this new designchoice implies additional benefits not seen in prior systems:Barrelfish/DC can completely replace the OS kernelcode running on any single core or subset of cores in thesystem at runtime, without disruption to any other OSor application code, including that running on the core.Kernels can be upgraded or bugs fixed without downtime,or replaced temporarily, for example to enable detailedinstrumentation, to change a scheduling algorithm, or toprovide a different kind of service such as performanceisolated,hard real-time processing for a bounded period.Furthermore, per-core OS state can be moved betweenslow, low-power cores and fast, energy-hungry cores.Multiple cores’ state can be temporarily aggregated onto asingle core to further trade-off performance and power, orto dedicate an entire package to running a single job for alimited period. Parts of Barrelfish/DC can be moved ontoand off cores optimized for particular workloads. Corescan be fused [26] transparently, and SMT threads [29, 34]or cores sharing functional units [12] can be selectivelyused for application threads or OS accelerators.Barrelfish/DC relies on several innovations which formthe main contributions of this paper. Barrelfish/DC treatsa CPU core as being a special case of a peripheral device,and introduces the concept of a boot driver, which canstart, stop, and restart a core while running elsewhere. We1USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 17

use a partitioned capability system for memory managementwhich allows us to completely externalize all OSstate for a core. This in turn permits a kernel to be essentiallystateless, and easily replaced while Barrelfish/DCcontinues to run. We factor the OS into per-core kernels1 and OSnodes, and a Kernel Control Block providesa kernel-readable handle on the total state of an OSnode.In the next section, we lay out the recent trends inhardware design and software requirements that motivatethe ideas in Barrelfish/DC. Following this, in Section 3we discuss in more detail the background to our work,and related systems and techniques. In Section 4 wepresent the design of Barrelfish/DC, in particular the keyideas mentioned above. In Section 5 we show by meansof microbenchmarks and real applications (a web serverand the PostgreSQL database) that the new functionalityof Barrelfish/DC incurs negligible overhead, as well asdemonstrating how Barrelfish/DC can provide worst-caseexecution time guarantees for applications by temporarilyisolating cores. Finally, we discuss Barrelfish/DC limitationsand future work in Section 6, and conclude inSection 7.2 Motivation and BackgroundBarrelfish/DC fully decouples cores from kernels (supervisoryprograms running in kernel mode), and moreoverboth of them from the per-core state of the OS as a wholeand its associated applications (threads, address spaces,communication channels, etc.). This goes considerablybeyond the core hotplug or dynamic core support in today’sOSes. Figure 1 shows the range of primitive kerneloperations that Barrelfish/DC supports transparently to applicationsand without downtime as the system executes:• A kernel on a core can be rebooted or replaced.• The per-core OS state can be moved between cores.• Multiple per-core OS components can be relocatedto temporarily “share” a core.In this section we argue why such functionality willbecome important in the future, based on recent trends inhardware and software.2.1 HardwareIt is by now commonplace to remark that core counts,both on a single chip and in a complete system, are increasing,with a corresponding increase in the complexityof the memory system – non-uniform memory access andmultiple levels of cache sharing. Systems software, and1 Barrelfish uses the term CPU driver to refer to the kernel-modecode running on a core. In this paper, we use the term “kernel” instead,to avoid confusion with boot driver.in particular the OS, must tackle the complex problem ofscheduling both OS tasks and those of applications acrossa number of processors based on memory locality.At the same time, cores themselves are becoming nonuniform:Asymmetric multicore processors (AMP) [31]mix cores of different microarchitectures (and thereforeperformance and energy characteristics) on a single processor.A key motivation for this is power reduction forembedded systems like smartphones: under high CPUload, complex, high-performance cores can completetasks more quickly, resulting in power reduction in otherareas of the system. Under light CPU load, however, it ismore efficient to run tasks on simple, low-power cores.While migration between cores can be transparent tothe OS (as is possible with, e.g., ARM’s “big.LITTLE”AMP architecture) a better solution is for the OS to managea heterogeneous collection of cores itself, poweringindividual cores on and off reactively.Alternatively, Intel’s Turbo Boost feature, which increasesthe frequency and voltage of a core when otherson the same die are sufficiently idle to keep the chipwithin its thermal envelope, is arguably a dynamic formof AMP [15].At the same time, hotplug of processors, once theprovince of specialized machines like the Tandem Non-Stop systems [6], is becoming more mainstream. Moreradical proposals for reconfiguring physical processorsinclude Core Fusion [26], whereby multiple independentcores can be morphed into a larger CPU, pooling cachesand functional units to improve the performance of sequentialprograms.Ultimately, the age of “dark silicon” [21] may welllead to increased core counts, but with a hard limit on thenumber that may be powered on at any given time. Performanceadvances and energy savings subsequently willhave to derive from specialized hardware for particularworkloads or operations [47].The implications for a future OS are that it must managea dynamic set of physical cores, and be able to adjustto changes in the number, configuration, and microarchitectureof cores available at runtime, while maintaining astable execution environment for applications.2.2 SoftwareAlongside hardware trends, there is increasing interest inmodifying, upgrading, patching, or replacing OS kernelsat runtime. Baumann et al. [9] implement dynamic kernelupdates in K42, leveraging the object-oriented designof the OS, and later extend this to interface changes usingobject adapters and lazy update [7]. More recently,Ksplice [3] allows binary patching of Linux kernels withoutreboot, and works by comparing generated object codeand replacing entire functions. Dynamic instrumentation218 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

OSnode βOSnode αk. B3k. Ckernel Akernel B1kernel B2multiplexerkernel B2core 0core 1core 2core 1update move park unparkTimeFigure 1: Shows the supported operations of a decoupled OS. Update: The entire kernel, dispatching OSnode α, is replacedat runtime. Move: OSnode α containing all per-core state, entailing applications is migrated to another core and kernel. Park:OSnode α is moved to a new core and kernel that temporarily dispatches two OSnodes. Unpark: OSnode α is transferred back toits previous core.systems like Dtrace [13] provide mechanisms that modifythe kernel at run-time to analyze program behavior.All these systems show that the key challenges in updatingan OS online are to maintain critical invariantsacross the update and to do so with minimal interruptionof service (the system should pause, if at all, for a minimalperiod). This is particularly hard in a multiprocessorkernel with shared state.In this paper, we argue for addressing all these challengesin a single framework for core and kernel managementin the OS, although the structure of Unix-likeoperating systems presents a barrier to such a unifiedframework. The rest of this paper describes the unifiedapproach we adopted in Barrelfish/DC.3 Related workOur work combines several directions in OS design andimplementation: core hotplugging, kernel update andreplacement, and multikernel architectures.3.1 CPU HotplugMost modern OS designs today support some form of corehotplug. Since the overriding motivation is reliability, unpluggingor plugging a core is considered a rare eventand the OS optimizes the common case where the coresare not being hotplugged. For example, Linux CPU hotpluguses the __stop_machine() kernel call, which haltsapplication execution on all online CPUs for typicallyhundreds of milliseconds [23], overhead that increasesfurther when the system is under CPU load [25]. We showfurther evidence of this cost in Section 5.1 where we compareLinux’ CPU hotplug with Barrelfish/DC’ core updateoperations.Recognizing that processors will be configured muchmore frequently in the future for reasons of energy usageand performance optimization, Chameleon [37] identifiesseveral bottlenecks in the existing Linux implementationdue to global locks, and argues that current OSes are illequipped for processor sets that can be reconfigured atruntime. Chameleon extends Linux to provide support forchanging the set of processors efficiently at runtime, anda scheduling framework for exploiting this new functionality.Chameleon can perform processor reconfigurationup to 100,000 times faster than Linux 2.6.Barrelfish/DC is inspired in part by this work, butadopts a very different approach. Where Chameleon targetsa single, monolithic shared kernel, Barrelfish/DCadopts a multikernel model and uses the ability to rebootindividual kernels one by one to support CPU reconfiguration.The abstractions provided are accordingly different:Chameleon abstracts hardware processors behind processorproxies and execution objects, in part to handle theproblem of per-core state (primarily interrupt handlers)on an offline or de-configured processor. In contrast, Barrelfish/DCabstracts the per-core state (typically muchlarger in a shared-nothing multikernel than in a sharedmemorymonolithic kernel) behind OSnode and kernelcontrol block abstractions.3USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 19

In a very different approach, Kozuch et al. [30] showhow commodity OS hibernation and hotplug facilities canbe used to migrate a complete OS between different machines(with different hardware configurations) withoutvirtualization.Hypervisors are typically capable of simulating hotpluggingof CPUs within a virtual machine. Barrelfish/DCcan be deployed as a guest OS to manage a variable setof virtual CPUs allocated by the hypervisor. Indeed, Barrelfish/DCaddresses a long-standing issue in virtualization:it is hard to fully virtualize the microarchitecture of aprocessor when VMs might migrate between asymmetriccores or between physical machines with different processors.As a guest, Barrelfish/DC can natively handle suchheterogeneity and change without disrupting operation.3.2 Kernel updatesThe problem of patching system software without downtimeof critical services has been a research area for sometime. For example, K42 explored update of a running kernel[7, 9], exploiting the system’s heavily object-orienteddesign. Most modern mainstream OSes support dynamicloading and unloading of kernel modules, which can beused to update or specialize limited parts of the OS.KSplice [3] patches running Linux kernels withoutthe need for reboot by replacing code in the kernel ata granularity of complete functions. It uses the Linuxstop_machine() call to ensure that no CPU is currentlyexecuting a function to be replaced, and places a branchinstruction at the start of the obsolete function to directexecution of the replacement code. Systems like KSplicereplace individual functions across all cores at the sametime. In contrast, Barrelfish/DC replaces entire kernels,but on a subset of cores at a time. KSplice makes sense foran OS where all cores must execute in the same, sharedmemorykernel and the overhead incurred by quiescingthe entire machine is unavoidable.Proteos [22] uses a similar approach to Barrelfish/DCby replacing applications in their entirety instead of applyingpatches to existing code. In contrast to Ksplice, Proteosautomatically applies state updates while preservingpointer integrity in many cases, which eases the burden onprogrammers to write complicated state transformationfunctions. In contrast to Barrelfish/DC, Proteos does notupgrade kernel-mode code but focuses on updates for OSprocesses running in user-space, in a micro-kernel environment.Much of the OS functionality in Barrelfish/DCresides in user-space as well, and Proteos would be applicablehere.Otherworld [18] also enables kernel updates withoutdisrupting applications, with a focus on recovering systemcrashes. Otherworld can microreboot the system kernelafter a critical error without clobbering running applications’state, and then attempt to restore applications thatwere running at the time of a crash by recreating applicationmemory spaces, open files and other resources.Rather than relying on a single, system-wide kernel,Barrelfish/DC exploits the multikernel environment to offerboth greater flexibility and better performance: kernelsand cores can be updated dynamically with (as we showin Section 5) negligible disruption to the rest of the OS.While their goals of security and availability differsomewhat from Barrelfish/DC, KeyKOS [24] andEROS [42] use partitioned capabilities to provide an essentiallystateless kernel. Memory in KeyKOS is persistent,and it allows updates of the OS while running,achieving continuity by restoring from disk-based checkpointsof the entire capability state. Barrelfish/DC bycontrast achieves continuity by distributing the capabilitysystem, only restarting some of the kernels at a time, andpreserving each kernel’s portion of the capability systemacross the restart.3.3 MultikernelsMultikernels such as fos [48], Akaros [40], Tessellation[33], Hive [14], and Barrelfish [8], are based onthe observation that modern hardware is a networkedsystem and so it is advantageous to model the OS as adistributed system. For example, Barrelfish runs a smallkernel on each core in the system, and the OS is built asa set of cooperating processes, each running on one ofthese kernels, sharing no memory, and communicating viamessage passing. Multikernels are motivated by both thescalability advantages of sharing no cache lines betweencores, and the goal of supporting future hardware withheterogeneous processors and little or no cache-coherentor shared physical memory.Barrelfish/DC exploits the multikernel design for a newreason: dynamic and flexible management of the coresand the kernels of the system. A multikernel can naturallyrun different versions of kernels on different cores. Theseversions can be tailored to the hardware, or specializedfor different workloads.Furthermore, since (unlike in monolithic kernels) thestate on each core is relatively decoupled from the restof the system, multikernels are a good match for systemswhere cores come and go, and intuitively should supportreconfiguration of part of the hardware without unduedisruption to software running elsewhere on the machine.Finally, the shared-nothing multikernel architecture allowsus to wrap kernel state and move it between differentkernels without worrying about potentially harmful concurrentaccesses.We chose to base Barrelfish/DC on Barrelfish, as it isreadily available, is under active development, supportsmultiple hardware platforms, and can run a variety of420 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

common applications such as databases and web servers.The features of Barrelfish/DC described in this paper willbe incorporated into a future Barrelfish release.Recently, multikernels have been combined with traditionalOS designs such as Linux [27, 36] so as to runmultiple Linux kernels on different cores of the same machineusing different partitions of physical memory, inorder to provide performance isolation between applications.Popcorn Linux [38, 43] boots a modified Linuxkernel in this fashion, and supports kernel- and user-spacecommunication channels between kernels [41], and processmigration between kernels. In principle, Popcornextended with the ideas in Barrelfish/DC could be combinedwith Chameleon in a two-level approach to dynamicprocessor support.4 DesignWe now describe how Barrelfish/DC decouples cores, kernels,and the rest of the OS. We focus entirely on mechanismin this paper, and so do not address scheduling andpolicies for kernel replacement, core power management,or application migration. Note also that our main motivationin Barrelfish/DC is adapting the OS for performanceand flexibility, and so we do not consider fault toleranceand isolation for now.We first describe how Barrelfish/DC boots a new core,and then present in stages the problem of per-core statewhen removing a core, introducing the Barrelfish/DC capabilitysystem and kernel control block. We then discussthe challenges of time and interrupts, and finish with adiscussion of the wider implications of the design.4.1 Booting a new coreCurrent CPU hotplug approaches assume a single, sharedkernel and a homogeneous (albeit NUMA) machine, witha variable number of active cores up to a fixed limit, andso a static in-kernel table of cores (whether active or inactive)suffices to represent the current hardware state.Bringing a core online is a question of turning it on, updatingthis table, and creating per-core state when needed.Previous versions of Barrelfish also adopted this approach,and booted all cores during system initialization, thoughthere has been experimental work on dynamic booting ofheterogeneous cores [35].Barrelfish/DC targets a broader hardware landscape,with complex machines comprising potentially heterogeneouscores. Furthermore, since Barrelfish/DC runsa different kernel instance on each core, there is no reasonwhy the same kernel code should run everywhere –indeed, we show one advantage of not doing this in Section5.3. We thus need an OS representation of a coreon the machine which abstracts the hardware-dependentmechanisms for bringing that core up (with some kernel)and down.Therefore, Barrelfish/DC introduces the concept of aboot driver, which is a piece of code running on a “homecore” which manages a “target core” and encapsulatesthe hardware functionality to boot, suspend, resume, andpower-down the latter. Currently boot drivers run as processes,but closely resemble device drivers and couldequally run as software objects within another process.A new core is brought online as follows:1. The new core is detected by some platform-specificmechanism (e.g., ACPI) and its appearance registeredwith the device management subsystem.2. Barrelfish/DC selects and starts an appropriate bootdriver for the new core.3. Barrelfish/DC selects a kernel binary and argumentsfor the new core, and directs the boot driver to bootthe kernel on the core.4. The boot driver loads and relocates the kernel, andexecutes the hardware protocol to start the new core.5. The new kernel initializes and uses existing Barrelfishprotocols for integrating into the running OS.The boot driver abstraction treats CPU cores much likeperipheral devices, and allows us to reuse the OS’s existingdevice and hotplug management infrastructure [50]to handle new cores and select drivers and kernels forthem. It also separates the hardware-specific mechanismfor booting a core from the policy question of what kernelbinary to boot the core with.Boot drivers remove most of the core boot process fromthe kernel: in Barrelfish/DC we have entirely replaced theexisting multiprocessor booting code for multiple architectures(which was spread throughout the system) withboot drivers, resulting in a much simpler system structure,and reduced code in the kernels themselves.Booting a core (and, indeed, shutting it down) in Barrelfish/DConly involves two processes: the boot driveron the home core, and the kernel on the target core. Forthis reason, we require no global locks or other synchronizationin the system, and the performance of these operationsis not impacted by load on other cores. We demonstratethese benefits experimentally in Section 5.1.Since a boot driver for a core requires (as with a devicedriver) at least one existing core to execute, there is apotential dependency problem as cores come and go. Forthe PC platform we focus on here, this is straightforwardsince any core can run a boot driver for any other core,but we note that in general the problem is the same as thatof allocating device drivers to cores.Boot drivers provide a convenient abstraction of hardwareand are also used to shutdown cores, but this is notthe main challenge in removing a core from the system.5USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 21

4.2 Per-core stateTaking a core out of service in a modern OS is a moreinvolved process than booting it, since modern multicoreOSes include varying amounts of per-core kernel state. Ifthey did not, removing a core would be simply requiremigrating any running thread somewhere else, updatingthe scheduler, and halting the core.The challenge is best understood by drawing a distinctionbetween the global state in an OS kernel (i.e., thestate which is shared between all running cores in thesystem) and the per-core state, which is only accessed bya single core. The kernel state of any OS is composed ofthese two categories.In, for example, older versions of Unix, all kernel statewas global and protected by locks. In practice, however, amodern OS keeps per-core state for scalability of scheduling,memory allocation, virtual memory, etc. Per-coredata structures reduce write sharing of cache lines, whichin turn reduces interconnect traffic and cache miss ratedue to coherency misses.For example, Linux and Windows use per-core schedulingqueues, and distributed memory allocators. Corey [10]allowed configurable sharing of page tables betweencores, and many Linux scaling enhancements (e.g., [11])have been of this form. K42 [2] adopted reduced sharingas a central design principle, and introduced the abstractionof clustered objects, essentially global proxies forpervasive per-core state.Multikernels like Barrelfish [8] push this idea to itslogical conclusion, sharing no data (other than messagechannels) between cores. Multikernels are an extremepoint in the design space, but are useful for precisely thisreason: they highlight the problem of consistent per-corestate in modern hardware. As core counts increase, wecan expect the percentage of OS state that is distributedin more conventional OSes to increase.Shutting down a core therefore entails disposing of thisstate without losing information or violating system-wideconsistency invariants. This may impose significant overhead.For example, Chameleon [37] devotes considerableeffort to ensuring that per-core interrupt handling stateis consistent across CPU reconfiguration. As more statebecomes distributed, this overhead will increase.Worse, how to dispose of this state depends on what itis: removing a per-core scheduling queue means migratingthreads to other cores, whereas removing a per-corememory allocator requires merging its memory pool withanother allocator elsewhere.Rather than implementing a succession of piecemealsolutions to this problem, in Barrelfish/DC we adopt aradical approach of lifting all the per-core OS state outof the kernel, so that it can be reclaimed lazily withoutdelaying the rest of the OS. This design provides theScheduler StateCap Derivation TreeTimer Offset (§4.7)IRQ State (§4.8)KCB (§4.4)...FrameFrameFrameFrameCNodePCB§4.2)CNodePCB§4.3)...FrameFrameFrameCNodeFigure 2: State in the Barrelfish/DC OSnode...NullFrameFrameFramemeans to completely decouple per-core state from both theunderlying kernel implementation and the core hardware.We find it helpful to use the term OSnode to denotethe total state of an OS kernel local to a particular core.In Linux the OSnode changes with different versions ofthe kernel; Chameleon identifies this state by manualannotation of the kernel source code. In Barrelfish, theOSnode is all the state – there is no shared global data.4.3 Capabilities in Barrelfish/DCBarrelfish/DC captures the OSnode using its capabilitysystem: all memory and other resources maintained by thecore (including interrupts and communication end-points)are represented by capabilities, and thus the OSnode isrepresented by the capability set of the core. The per-corestate of Barrelfish/DC is shown schematically in Figure 2.Barrelfish/DC’s capability system, an extension of thatin Barrelfish [44], is derived from the partitioned capabilityscheme used in seL4 [19, 20, 28].In seL4 (and Barrelfish), all regions of memory arereferred to by capabilities, and capabilities are typed to reflectwhat the memory is used for. For example, a “frame”capability refers to memory that the holder can map intotheir address space, while a “c-node” capability refers tomemory that is used to store the bit representations ofcapabilities themselves. The security of the system asa whole derives from the fact that only a small, trustedcomputing base (the kernel) holds both a frame capabilityand a c-node capability to the same memory, and cantherefore fabricate capabilities.A capability for a region can be split into two smallerregions, and also retyped according to a set of systemrules that preserve integrity. Initially, memory regionsare of type “untyped”, and must be explicitly retyped to622 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

“frame”, “c-node”, or some other type.This approach has the useful property that a process canallocate memory without being able to access its contents.This is used in seL4 to remove any dynamic memory allocationfrom the kernel, greatly simplifying both the formalspecification of the kernel and its subsequent proof [20].All kernel objects (such as process control blocks, or pagetables) are allocated by user-level processes which can,themselves, not access them directly.A key insight of Barrelfish/DC is that this approach canexternalize the kernel state entirely, as follows.4.4 Kernel Control BlocksIn developing Barrelfish/DC, we examined the Barrelfishkernel to identify all the data structures which were notdirect (optimized) derivations of information already heldin the capability tree (and which could therefore be reconstructeddynamically from the tree). We then eliminatedfrom this set any state that did not need to persist across akernel restart.For example, the runnable state and other schedulingparameters of a process 2 are held in the process’ controlblock, which is part of the capability system. However,the scheduler queues themselves do not need to persistacross a change of kernel, since (a) any scheduler willneed to recalculate them based on the current time, and(b) the new scheduler may have a completely differentpolicy and associated data structures anyway.What remained was remarkably small: it consists of:• The minimal scheduling state: the head of a linkedlist of a list of process control blocks.• Interrupt state. We discuss interrupts in Section 4.8.• The root of the capability derivation tree, from whichall the per-core capabilities can be reached.• The timer offset, discussed in Section 4.7.In Barrelfish/DC, we introduce a new memory object,the Kernel Control Block (KCB), and associated capabilitytype, holding this data in a standard format. The KCB issmall: for 64-bit x86 it is about 28 KiB in size, almostall of which is used by communication endpoints forinterrupts.4.5 Replacing a kernelThe KCB effectively decouples the per-core OS state fromthe kernel. This allows Barrelfish/DC to shut down a kernelon a core (under the control of the boot driver runningon another core) and replace it with a new one. The currentlyrunning kernel saves a small amount of persistent2 Technically, it is a Barrelfish “dispatcher”, the core-local representationof a process. A process usually consists of a set of distinct“dispatchers”, one in each OSnode.state in the KCB, and halts the core. The boot driverthen loads a new kernel with an argument supplying theaddress of the KCB. It then restarts the core (using an IPIon x86 machines), causing the new kernel to boot. Thisnew kernel then initializes any internal data structures itneeds from the KCB and the OSnode capability database.The described technique allows for arbitrary updates ofkernel-mode code. By design, the kernel does not accessstate in the OSnode concurrently. Therefore, having a quiescentstate in the OSnode before we shut-down a core isalways guaranteed. The simplest case for updates requiresno changes in any data structures reachable by the KCBand can be performed as described by simply replacingthe kernel code. Updates that require a transformationof the data structures may require a one-time adaptionfunction to execute during initialization, whose overheaddepends on the complexity of the function and the size ofthe OSnode. The worst-case scenario is one that requiresadditional memory, since the kernel by design delegatesdynamic memory allocation to userspace.As we show in Section 5, replacing a kernel can bedone with little performance impact on processes runningon the core, even device drivers.4.6 Kernel sharing and core shutdownAs we mentioned above, taking a core completely outof service involves not simply shutting down the kernel,but also disposing of or migrating all the per-core stateon the core, and this can take time. Like Chameleon,Barrelfish/DC addresses this problem by deferring it: weimmediately take the core down, but keep the OSnoderunning in order to be able to dismantle it lazily. Tofacilitate this, we created a new kernel which is capableof multiplexing several KCBs (using a simple extensionto the existing scheduler).Performance of two active OSnodes sharing a core isstrictly best-effort, and is not intended to be used for anycase where application performance matters. Rather, itprovides a way for an OSnode to be taken out of servicein the background, after the core has been shut down.Note that there is no need for all cores in Barrelfish/DCto run this multiplexing kernel, or, indeed, for any coresto run it when it is not being used – it can simply replacean existing kernel on demand. In practice, we find thatthere is no performance loss when running a single KCBabove a multiplexing kernel.Decoupling kernel state allows attaching and detachingKCBs from a running kernel. The entry point for kernelcode takes a KCB as an argument. When a new kernel isstarted, a fresh KCB is provided to the kernel code. Torestart a kernel, the KCB is detached from the running kernelcode, the core is shut down, and the KCB is providedto the newly booted kernel code.7USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 23

We rely on shared physical memory when movingOSnodes between cores. This goes against the originalmultikernel premise that assumes no shared memory betweencores. However, an OSnode is still always in useby strictly one core at the time. Therefore, the benefitsof avoiding concurrent access in OSnode state remain.We discuss support for distributed memory hardware inSection 6.The combination of state externalization via the KCBand kernel sharing on a single core has a number of furtherapplications, which we describe in Section 4.10.4.7 Dealing with timeOne of the complicating factors in starting the OSnodewith a new kernel is the passage of time. Each kernelmaintains a per-core internal clock (based on a freerunningtimer, such as the local APIC), and expects thisto increase monotonically. The clock is used for percorescheduling and other time-sensitive tasks, and is alsoavailable to application threads running on the core via asystem call.Unfortunately, the hardware timers used are rarely synchronizedbetween cores. Some hardware (for example,modern PCs) define these timers to run at the same rateon every core (regardless of power management), but theymay still be offset from each other. On other hardwareplatforms, these clocks may simply run at different ratesbetween cores.In Barrelfish/DC we address this problem with twofields in the KCB. The first holds a constant offset fromthe local hardware clock; the OS applies this offset wheneverthe current time value is read.The second field is set to the current local time whenthe kernel is shut down. When a new kernel starts withan existing KCB, the offset field is reinitialized to thedifference between this old time value and the currenthardware clock, ensuring that local time for the OSnodeproceeds monotonically.4.8 Dealing with interruptsInterrupts pose an additional challenge when moving anOSnode between cores. It is important that interrupts fromhardware devices are always routed to the correct kernel.In Barrelfish interrupts are then mapped to messages deliveredto processes running on the target core. Someinterrupts (such as those from network cards) should “follow”the OSnode to its new core, whereas others shouldnot. We identify three categories of interrupt.1. Interrupts which are used exclusively by the kernel,for example a local timer interrupt used to implementpreemptive scheduling. Handling these interruptsis internal to the kernel, and their sources aretypically per-core hardware devices like APICs orperformance counters. In this case, there is no needto take additional actions when reassigning KCBsbetween cores.2. Inter-processor interrupts (IPIs), typically used forasynchronous communication between cores. Barrelfish/DCuses an indirection table that mapsOSnode identifiers to the physical core running thecorresponding kernel. When one kernel sends anIPI to another, it uses this table to obtain the hardwaredestination address for the interrupt. Whendetaching a KCB from a core, its entry is updatedto indicate that its kernel is unavailable. Similarly,attaching a KCB to a core, updates the location tothe new core identifier.3. Device interrupts, which should be forwarded to aspecific core (e.g. via IOAPICs and PCIe bridges)running the handler for the device’s driver.When Barrelfish/DC device drivers start up they requestforwarding of device interrupts by providing twocapability arguments to their local kernel: an opaque interruptdescriptor (which conveys authorization to receivethe interrupt) and a message binding. The interrupt descriptorcontains all the architecture-specific informationabout the interrupt source needed to route the interrupt tothe right core. The kernel associates the message bindingwith the architectural interrupt and subsequently forwardsinterrupts to the message channel.For the device and the driver to continue normal operation,the interrupt needs to be re-routed to the newcore, and a new mapping is set up for the (existing) driverprocess. This could be done either transparently by thekernel, or explicitly by the device driver.We choose the latter approach to simplify the kernel.When a Barrelfish/DC kernel shuts down, it disables allinterrupts. When a new kernel subsequently resumes anOSnode, it sends a message (via a scheduler upcall) toevery process which had an interrupt registered. Eachdriver process responds to this message by re-registeringits interrupt, and then checking with the device directlyto see if any events have been missed in the meantime(ensuring any race condition is benign). In Section 5.2.1we show the overhead of this process.4.9 Application supportFrom the perspective of applications which are obliviousto the allocation of physical cores (and which dealsolely with threads), the additional functionality of Barrelfish/DCis completely transparent. However, manyapplications such as language runtimes and database systemsdeal directly with physical cores, and tailor theirscheduling of user-level threads accordingly.824 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

For these applications, Barrelfish/DC can use the existingscheduler activation [1] mechanism for processdispatch in Barrelfish to notify userspace of changes inthe number of online processors, much as it can alreadyconvey the allocation of physical cores to applications.4.10 DiscussionFrom a broad perspective, the combination of boot driversand replaceable kernels is a radically different view ofhow an OS should manage processors on a machine. Moderngeneral-purpose kernels such as Linux try to supporta broad set of requirements by implementing differentbehaviors based on build-time and run-time configuration.Barrelfish/DC offers an alternative: instead of buildingcomplicated kernels that try to do many things, build simplekernels that do one thing well. While Linux selectsa single kernel at boot time for all cores, Barrelfish/DCallows selecting not only per-core kernels, but changingthis selection on-the-fly.There are many applications for specialized kernels,including those tailored for running databases or languagerun-times, debugging or profiling, or directly executingverified user code as in Google’s native client [49].To take one example, in this paper we demonstratesupport for hard real-time applications. Despite yearsof development of real-time support features in Linuxand other general-purpose kernels [16], many users resortto specialized real-time OSes, or modified versions ofgeneral-purpose OSes [32].Barrelfish/DC can offer hard real-time support by rebootinga core with a specialized kernel, which, to eliminateOS jitter, has no scheduler (since it targets a singleapplication) and takes no interrupts. If a core is notpreallocated, it must be made available at run-time bymigrating the resident OSnode to another core that runsa multi-KCB kernel, an operation we call parking. Ifrequired, cache interference from other cores can alsobe mitigated by migrating their OSnodes to other packages.Once the hard real-time application finishes, theOSnodes can be moved back to the now-available cores.We evaluate this approach in Section 5.3.5 EvaluationWe present here a performance evaluation of Barrelfish/DC.First (Section 5.1), we measure the performanceof starting and stopping cores in Barrelfish/DCand in Linux. Second (Section 5.2), we investigate the behaviorof applications when we restart kernels, and whenwe park OSnodes. Finally, (Section 5.3), we demonstrateisolating performance via a specialized kernel. We performexperiments on the set of x86 machines shown inTable 1. Hyperthreading, TurboBoost, and SpeedSteptechnologies are disabled in machines that support them,as they complicate cycle counter measurements. Turbo-Boost and SpeedStep can change the processor frequencyin unpredictable ways, leading to high fluctuation for repeatedexperiments. The same is true for Hyperthreadingdue to sharing of hardware logic between logical cores.However, TurboBoost and Hyperthreading are both relevantfor this work as discussed in Section 6 and Section 1.packages×cores/uarch CPU model2×2 Santa-Rosa 2.8 GHz Opteron 22004×4 Shanghai 2.5 GHz Opteron 83802×10 SandyBridge 2.5 GHz Xeon E5-2670 v21×4 Haswell 3.4 GHz Xeon E3-1245 v3Table 1: Systems we use in our evaluation. The first columndescribes the topology of the machine (total number of packagesand cores per package) and the second the CPU model.5.1 Core management operationsIn this section, we evaluate the performance of managingcores in Barrelfish/DC, and also in Linux using the CPUHotplug facility [4]. We consider two operations: shuttingdown a core (down) and bringing it back up again (up).Bringing up a core in Linux is different from bringingup a core in Barrelfish/DC. In Barrelfish/DC, each coreexecutes a different kernel which needs to be loaded bythe boot driver, while in Linux all cores share the samecode. Furthermore, because cores share state in Linux,core management operations require global synchronization,resulting in stopping application execution in allcores for an extended period of time [23]. Stopping coresis also different between Linux and Barrelfish/DC. InLinux, applications executed in the halting core need tobe migrated to other online cores before the shutdown canproceed, while in Barrelfish/DC we typically would movea complete OSnode after the shutdown and not individualapplications.In Barrelfish/DC, the down time is the time it takes theboot driver to send an appropriate IPI to the core to behalted plus the propagation time of the IPI and the cost ofthe IPI handler in the receiving core. For the up operationwe take two measurements: the boot driver cost to preparea new kernel up until (and including) the point where itsends an IPI to the starting core (driver), and the cost inthe booted core from the point it wakes up until the kernelis fully online (core).In Linux, we measure the latency of starting or stoppinga core using the log entry of the smpboot module and asentinel line echoed to /dev/kmsg. For core shutdown,smboot reports when the core becomes offline, and weinsert the sentinel right before the operation is initiated.9USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 25

Barrelfish/DCLinuxidle load idle loaddownupupdowndriver core driver core down up down up(µs) (ms) (ms) (µs) (ms) (ms) (ms) (ms) (ms) (ms)2×2 Santa-Rosa 2.7 / — a 29 1.2 2.7 / — 34 ± 17 1.2 131 ± 25 20 ± 1 5049 ± 2052 26 ± 54×4 Shanghai 2.3 / 2.6 24 1.0 2.3 / 2.7 46 ± 76 1.0 104 ± 50 18 ± 3 3268 ± 980 18 ± 32×10 SandyBridge 3.5 / 3.7 10 0.8 3.6 / 3.7 23 ± 52 0.8 62 ± 46 21 ± 7 2265 ± 1656 23 ± 51×4 Haswell 0.8 / — a 7 0.5 0.8 / — 7 ± 0.1 0.5 46 ± 40 14 ± 1 2543 ± 1710 20 ± 5Results in cycles×10 3 ×10 6 ×10 6 ×10 3 ×10 6 ×10 6 ×10 6 ×10 6 ×10 6 ×10 62×2 Santa-Rosa 8/— 85 3.4 8/— 97 ± 49 3.5 367 ± 41 56 ± 2.0 14139 ± 5700 74 ± 214×4 Shanghai 6/6 63 2.6 6/7 115 ± 192 2.6 261 ± 127 44 ± 2.0 8170 ± 2452 46 ± 82×10 SandyBridge 9/10 27 2.1 9/10 59 ± 133 2.1 155 ± 116 53 ± 2.0 5663 ± 4141 57 ± 121×4 Haswell 3/— 26 1.9 2.9 / — 26 ± 0.40 2.0 156 ± 137 50 ± 0.5 8647 ± 5816 69 ± 16Table 2: Performance of core management operations for Barrelfish/DC and Linux (3.13) when the system is idle and when thesystem is under load. For the Barrelfish/DC down column, the value after the slash shows the cost of stopping a core on anothersocket with regard to the boot driver. a We do not include this number for Santa-Rosa because it lacks synchronized timestampcounters, nor for Haswell because it only includes a single package.For core boot, smpboot reports when the operation starts,so we insert the sentinel line right after the operation.For both Barrelfish/DC and Linux we consider twocases: an idle system (idle), and a system with all coresunder load (load). In Linux, we use the stress tool [45]to spawn a number of workers equal to the number ofcores that continuously execute the sync system call. InBarrelfish/DC, since the file-system is implemented as auser-space service, we spawn an application that continuouslyperforms memory management system calls oneach core of the system.Table 2 summarizes our results. We show both time(msecs and µsecs) and cycle counter units for convenience.All results are obtained by repeating the experiment 20times, and calculating the mean value. We include thestandard deviation where it is non-negligible.Stopping cores: The cost of stopping cores in Barrelfish/DCranges from 0.8 µs (Haswell) to 3.5 µs (Sandy-Bridge). Barrelfish/DC does not share state across cores,and as a result no synchronization between cores is neededto shut one down. Furthermore, Barrelfish/DC’ shutdownoperation consists of sending an IPI, which will cause thecore to stop after a minimal operation in the KCB (savingthe timer offset). In fact, the cost of stopping a core inBarrelfish/DC is small enough to observe the increasedcost of sending an IPI across sockets, leading to an increaseof 5% in stopping time on SandyBridge and 11%on Shanghai. These numbers are shown in Table 2, inthe Barrelfish/DC down columns after the slash. As thesemeasurements rely on timestamp counters being synchronizedacross packages, we are unable to present the costFigure 3: Breakdown of the cost of bringing up a core for theHaswell machine.increase of a cross-socket IPI on the Santa-Rosa machinewhose timestamp counters are only synchronized withina single package.In stark contrast, the cost of shutting down a core inLinux ranges from 46 ms to 131 ms. More importantly,the shutdown cost in Linux explodes when applying load,while it generally remains the same for Barrelfish/DC. Forexample, the average time to power down a core in Linuxon Haswell is increased by 55 times when we apply load.Starting cores: For Barrelfish/DC, the setup cost in theboot driver (driver) dominates the cost of starting a core(core). Fig. 3 shows a breakdown of the costs for bringingup a core on Haswell. Starting core corresponds to the1026 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

core Table 2 column, while the rest corresponds to operationsperformed by the boot driver: loading the imagefrom storage, allocating memory, ELF loading and relocation,etc. Loading the kernel from the file system is themost expensive operation. If multiple cores are bootedwith the same kernel, this image can be cached, significantlyimproving the time to start a core as shown in thesecond bar in Fig. 3. We note that the same costs willdominate the restart operation since shutting down a corehas negligible cost compared to bringing it up. Downtimecan be minimized by first doing the necessary preparationsin the boot driver and then halting and starting thecore.Even though Barrelfish/DC has to prepare the kernelimage, when idle, the cost of bringing up a corefor Barrelfish/DC is similar to the Linux cost (Barrelfish/DCis faster on our Intel machines, while the oppositeis true for our AMD machines). Bringing a coreup can take from 7 ms (Barrelfish/DC/Haswell) to 29 ms(Barrelfish/DC/Santa-Rosa). Load affects the cost of bootingup a core to varying degrees. In Linux such an effectis not observed in the Shanghai machine, while in theHaswell machine load increases average start time by33%. The effect of load when starting cores is generallystronger in Barrelfish/DC (e.g., in SandyBridge the costis more than doubled) because the boot driver time-sharesits core with the load generator.Overall, Barrelfish/DC has minimal overhead stoppingcores. For starting cores, results vary significantly acrossdifferent machines but the cost of bringing up cores inBarrelfish/DC is comparable to the respective Linux cost.5.2 ApplicationsIn this section, we evaluate the behavior of real applicationsunder two core management operations: restarting,where we update the core kernel as the application runs,and parking. In parking, we run the application in a corewith a normal kernel and then move its OSnode into amulti-KCB kernel running on a different core. While theapplication is parked it will share the core with anotherOSnode. We use a naive multi-KCB kernel that runs eachKCB for 20 ms, which is two times the scheduler timeslice. Finally, we move the application back to its originalcore. The application starts by running alone on its core.We execute all experiments in the Haswell machine.5.2.1 Ethernet driverOur first application is a Barrelfish NIC driver for theIntel 82574, which we modify for Barrelfish/DC to reregisterits interrupts when instructed by the kernel (seeSection 4.8). During the experiment we use ping from aclient machine to send ICMP echo requests to the NIC.We run ping as root with the -A switch, where the interpacketintervals adapt to the round-trip time. The pingmanual states: “on networks with low rtt this mode isessentially equivalent to flood mode.”Fig. 4a shows the effect of the restart operation in theround-trip time latency experienced by the client. Initially,the ping latency is 0.042 ms on average with small variation.Restarting the kernel produces two outliers (packets2307 and 2308 with an RTT of 11.1 ms and 1.07 ms, respectively).Note that 6.9 ms is the measured latency tobring up a core on this machine (Table 2).We present latency results for the parking experiment ina timeline (Fig. 4b), and in a cumulative distribution function(CDF) graph (Fig. 4c). Measurements taken whenthe driver’s OSnode runs exclusively on a core are denotedExclusive, while measurements where the OSnodeshares the core are denoted Shared. When parking begins,we observe an initial latency spike (from 0.042 msto 73.4 ms). The spike is caused by the parking operation,which involves sending a KCB capability reference fromthe boot driver to the multi-KCB kernel as a message. 3After the initial coordination, outliers are only caused byKCB time-sharing (maximum: 20 ms, mean: 5.57 ms).After unparking the driver, latency returns to its initial levels.Unparking does not cause the same spike as parkingbecause we do not use messages: we halt the multi-KCBkernel and directly pass the KCB reference to a newlybooted kernel.5.2.2 Web serverIn this experiment we examine how a web server 4 thatserves files over the network behaves when its core isrestarted and when its OSnode is parked. We initiate atransfer on a client machine in the server’s LAN usingwget and plot the achieved bandwidth for each 50 KiBchunk when fetching a 1 GiB file.Fig. 4d shows the results for the kernel restart experiment.The effect in this case is negligible on the clientside. We were unable to pinpoint the exact location ofthe update taking place from the data measured on theclient and the actual download times during kernel updateswere indistinguishable from a normal download. Asexpected, parking leads to a number of outliers caused byKCB time-sharing (Figures 4e and 4f). The average bandwidthbefore the parking is 113 MiB/s and the standarddeviation 9 MiB/s, whereas during parking the averagebandwidth is slightly lower at 111 MiB/s with a higherstandard deviation of 19 MiB/s.3 We follow the Barrelfish approach, where kernel messages arehandled by the monitor, a trusted OS component that runs in user-space.4 The Barrelfish native web server.11USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 27

(a) Ethernet driver restart (b) Ethernet driver parking (c) Ethernet driver parking CDF(d) Web server restart (e) Web server parking (f) Web server parking CDF(g) PostgreSQL restart (h) PostgreSQL parking (i) PostgreSQL parking CDFFigure 4: Application behavior when restarting kernels and parking OSnodes. For each application we include a timeline graph forrestarting, and a timeline and a CDF graph for parking.5.2.3 PostgreSQLNext, we run a PostgreSQL [39] database server in Barrelfish/DC,using TPC-H [46] data with a scaling factorof 0.01, stored in an in-memory file-system. We measurethe latency of a repeated CPU-bound query (query 9 inTPC-H) on a client over a LAN.Fig. 4g shows how restart affects client latency. Beforerebooting, average query latency is 36 ms. When a restartis performed, the first query has a latency of 51 ms. Aftera few perturbed queries, latency returns to its initial value.Figures 4h and 4i show the effect of parking theOSnode that contains the PostgreSQL server. As before,during normal operation the average latency is 36 ms.When the kernel is parked we observe two sets of outliers:one (with more points) with a latency of about 76 ms, andone with latency close to 56 ms. This happens, becausedepending on the latency, some queries wait for two KCBtime slices (20 ms each) of the co-hosted kernel, whileothers wait only for one.Overall, we argue that kernel restart incurs acceptableoverhead for online use. Parking, as expected, causes aperformance degradation, especially for latency-criticalapplications. This is, however, inherent in any form ofresource time-sharing. Furthermore, with improved KCBschedulingalgorithms the performance degradation canbe reduced or tuned (e.g., via KCB priorities).5.3 Performance isolationFinally, we illustrate the benefits of Barrelfish/DC’ supportfor restarting cores with specialized kernels usingthe case of hard-real time applications where eliminatingOS jitter is required. To ensure that the applicationwill run uninterrupted, we assign a core with a specializedkernel that does not implement scheduling and doesnot handle interrupts (see Section 4.10). We evaluatethe performance isolation that can be achieved with our1228 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

under the same configuration, we attribute the outliers toOS jitter. In Barrelfish the outliers reach up to 68k cycles(excluded from the graph). Linux performs betterthan Barrelfish, but its outliers still reach 27–28 kcycles.We ascribe the worse behavior of Barrelfish compared toLinux to OS services running in user-space.We conclude that Barrelfish/DC enables the online deploymentof a dedicated, simple to build, OS kernel thateliminates OS jitter and provides hard real-time guarantees.6 Future directions(a) Histogram for all samples(b) CDF for samples in the range of 6–7k cyclesFigure 5: Number of cycles measured for 10 3 iterations of asynthetic benchmark for Barrelfish/DC, Barrelfish, and Linuxusing real-time priorities.specialized kernel compared to the isolation provided by:(i) an unmodified Barrelfish kernel, and (ii) a Linux 3.13kernel where we set the application to run with real-timepriority. We run our experiments on the Haswell machine,ensuring that no other applications run on the same core.To measure OS jitter we use a synthetic benchmarkthat only performs memory stores to a single location.Our benchmark is intentionally simple to minimize performancevariance caused by architectural effects. Wesample the timestamp counter every 10 3 iterations, for atotal of 10 6 samples. Fig. 5a shows a histogram of sampledcycles, where for all systems, most of the valuesfall into the 6-7 thousand range (i.e., 6-7 cycles latencyper iteration). Fig. 5b presents the CDF graph for the6–7 kcycles range, showing that there are no significantdifferences for the three systems in this range.Contrarily to the Barrelfish/DC dedicated kernel whereall of the samples are in the 6-7k range, in Linux and Barrelfishwe observe significant outliers that fall outside thisrange. Since we run the experiment on the same hardware,Our ongoing work on Barrelfish/DC includes both exploringthe broader applications of the ideas, and alsoremoving some of the existing limitations of the system.On current hardware, we plan to investigate the powermanagementopportunities afforded by the ability to replacecores and migrate the running OS around the hardware.One short-term opportunity is to fully exploit Intel’sTurbo Boost feature to accelerate a serial task by temporarilyvacating (and thereby cooling) neighboring cores onthe same package.We also intend to use core replacement as a meansto improve OS instrumentation and tracing facilities, bydynamically instrumenting kernels running on particularcores at runtime, removing all instrumentation overheadin the common case. Ultimately, as kernel developers wewould like to minimize whole-system reboots as much aspossible by replacing single kernels on the fly.Barrelfish/DC currently assumes cache-coherent cores,where the OS state (i.e., the OSnode) can be easily migratedbetween cores by passing physical addresses. Thelack of cache-coherency per se can be handled with suitablecache flushes, but on hardware platforms withoutshared memory, or with different physical address spaceson different cores, the OSnode might not require considerabletransformation to migrate between cores. TheBarrelfish/DC capability system does contain all the informationnecessary to correctly swizzle pointers whencopying the OSnode between nodes, but the copy is likelyto be expensive, and dealing with shared-memory applicationstate (which Barrelfish fully supports outside theOS) is a significant challenge.A somewhat simpler case to deal with is moving anOSnode between a virtual and physical machine, allowingthe OS to switch from running natively to running in aVM container.Note that there is no requirement for the boot driverto share memory with its target core, as long as it has amechanism for loading a kernel binary into the latter’saddress space and controlling the core itself.When replacing kernels, Barrelfish/DC assumes thatthe OSnode format (in particular, the capability system)13USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 29

emains unchanged. If the in-memory format of the capabilitydatabase changes, then the new kernel must performa one-time format conversion when it boots. It is unclearhow much of a limitation this is in practice, since the capabilitysystem of Barrelfish has changed relatively littlesince its inception, but one way to mitigate the burden ofwriting such a conversion function is to exploit the factthat the format is already specified in a domain-specifichigh-level language called Hamlet [17] to derive the conversionfunction automatically.While Barrelfish/DC decouples cores, kernels, and theOS state, the topic of appropriate policies for using thesemechanisms without user intervention is an importantarea for future work. We plan to investigate policies that,based on system feedback, create new kernels to replaceothers, and move OSnodes across cores.Finally, while Barrelfish/DC applications are notifiedwhen the core set they are running on changes (via thescheduler activations mechanism), they are currently insulatedfrom knowledge about hardware core reconfigurations.However, there is no reason why this must alwaysbe the case. There may be applications (such as databases,or language runtimes) which can benefit from being notifiedabout such changes to the running system, and wesee no reason to hide this information from applicationswhich can exploit it.7 ConclusionBarrelfish/DC presents a radically different vision of howcores are exploited by an OS and the applications runningabove it, and implements it in a viable software stack:the notion that OS state, kernel code, and execution unitsshould be decoupled and freely interchangeable. Barrelfish/DCis an OS whose design assumes that all coresare dynamic.As hardware becomes more dynamic, and scalabilityconcerns increase the need to partition or replicate stateacross cores, system software will have to change itsassumptions about the underlying platform, and adaptto a new world with constantly shifting hardware. Barrelfish/DCoffers one approach to meeting this challenge.8 AcknowledgementsWe would like to thank the anonymous reviews and ourshepherd, Geoff Voelker, for their encouragement andhelpful suggestions. We would also like to acknowledgethe work of the rest of the Barrelfish team at ETH Zurichwithout which Barrelfish/DC would not be possible.References[1] ANDERSON, T. E., BERSHAD, B. N., LAZOWSKA, E. D., ANDLEVY, H. M. Scheduler activations: Effective kernel support forthe user-level management of parallelism. ACM Transactions onComputer Systems 10, 1 (1992), 53–79.[2] APPAVOO, J., DA SILVA, D., KRIEGER, O., AUSLANDER, M.,OSTROWSKI, M., ROSENBURG, B., WATERLAND, A., WIS-NIEWSKI, R. W., XENIDIS, J., STUMM, M., AND SOARES, L.Experience distributing objects in an SMMP OS. ACM Transactionson Computer Systems 25, 3 (2007).[3] ARNOLD, J., AND KAASHOEK, M. F. Ksplice: Automatic rebootlesskernel updates. In Proceedings of the EuroSys Conference(2009), pp. 187–198.[4] ASHOK RAJ. CPU hotplug support in the Linux kernel. https://www.kernel.org/doc/Documentation/cpu-hotplug.txt.[5] The Barrelfish Operating System. http://www.barrelfish.org/, 12.04.14.[6] BARTLETT, J. F. A NonStop Kernel. In Proceedings of the8th ACM Symposium on Operating Systems Principles (1981),pp. 22–29.[7] BAUMANN, A., APPAVOO, J., WISNIEWSKI, R. W., SILVA,D. D., KRIEGER, O., AND HEISER, G. Reboots are for hardware:Challenges and solutions to updating an operating system on thefly. In Proceedings of the USENIX Annual Technical Conference(2007), pp. 1–14.[8] BAUMANN, A., BARHAM, P., DAGAND, P.-E., HARRIS, T.,ISAACS, R., PETER, S., ROSCOE, T.,SCHÜPBACH, A., ANDSINGHANIA, A. The multikernel: a new OS architecture forscalable multicore systems. In Proceedings of the 22nd ACMSymposium on Operating System Principles (2009), pp. 29–44.[9] BAUMANN, A., HEISER, G., APPAVOO, J., DA SILVA, D.,KRIEGER, O., WISNIEWSKI, R. W., AND KERR, J. Providingdynamic update in an operating system. In Proceedings of theUSENIX Annual Technical Conference (2005), pp. 279–291.[10] BOYD-WICKIZER, S., CHEN, H., CHEN, R., MAO, Y.,KAASHOEK, F.,MORRIS, R., PESTEREV, A., STEIN, L., WU,M., DAI, Y., ZHANG, Y., AND ZHANG, Z. Corey: An operatingsystem for many cores. In Proceedings of the 8th Symposium onOperating Systems Design and Implementation (2008), pp. 43–57.[11] BOYD-WICKIZER, S., CLEMENTS, A. T., MAO, Y., PESTEREV,A., KAASHOEK, M.F.,MORRIS, R., AND ZELDOVICH, N. AnAnalysis of Linux Scalability to Many Cores. In Proceedings ofthe 9th USENIX Conference on Operating Systems Design andImplementation (2010), pp. 1–8.[12] BUTLER, M., BARNES, L., SARMA, D. D., AND GELINAS, B.Bulldozer: An approach to multithreaded compute performance.IEEE Micro 31, 2 (Mar. 2011), 6–15.[13] CANTRILL, B. M., SHAPIRO, M. W., AND LEVENTHAL, A. H.Dynamic instrumentation of production systems. In Proceedingsof the USENIX Annual Technical Conference (2004), pp. 15–28.[14] CHAPIN, J., ROSENBLUM, M., DEVINE, S., LAHIRI, T., TEO-DOSIU, D., AND GUPTA, A. Hive: Fault containment for sharedmemorymultiprocessors. In Proceedings of the 15th ACM Symposiumon Operating Systems Principles (1995), pp. 12–25.[15] CHARLES, J., JASSI, P., S, A. N., SADAT, A., AND FEDOROVA,A. Evaluation of the Intel Core i7 Turbo Boost feature. In Proceedingsof the IEEE International Symposium on Workload Characterization(2009).[16] CORBET, J. Deadline scheduling for 3.14. http://www.linuxfoundation.org/news-media/blogs/browse/2014/01/deadline-scheduling-314, 12.04.14.1430 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

[17] DAGAND, P.-E., BAUMANN, A., AND ROSCOE, T. Filet-o-Fish:practical and dependable domain-specific languages for OS development.In Proceedings of the 5th Workshop on ProgrammingLanguages and Operating Systems (2009).[18] DEPOUTOVITCH, A., AND STUMM, M. Otherworld: Giving Applicationsa Chance to Survive OS Kernel Crashes. In Proceedingsof the EuroSys Conference (2010), pp. 181–194.[19] DERRIN, P., ELKADUWE, D., AND ELPHINSTONE, K. seL4Reference Manual. NICTA, 2006. http://www.ertos.nicta.com.au/research/sel4/sel4-refman.pdf.[20] ELKADUWE, D., DERRIN, P., AND ELPHINSTONE, K. Kerneldesign for isolation and assurance of physical memory. InProceedings of the 1st Workshop on Isolation and Integration inEmbedded Systems (2008), pp. 35–40.[21] ESMAEILZADEH, H., BLEM, E.,ST. AMANT, R., SANKAR-ALINGAM, K., AND BURGER, D. Dark Silicon and the End ofMulticore Scaling. In Proceedings of the 38th Annual InternationalSymposium on Computer Architecture (2011), pp. 365–376.[22] GIUFFRIDA, C., KUIJSTEN, A., AND TANENBAUM, A. S. Safeand automatic live update for operating systems. In Proceedingsof the 18th International Conference on Architectural Support forProgramming Languages and Operating Systems (2013), pp. 279–292.[23] GLEIXNER, T.,MCKENNEY, P. E., AND GUITTOT, V. Cleaningup Linux’s CPU hotplug for real time and energy management.SIGBED Rev. 9, 4 (Nov. 2012), 49–52.[24] HARDY, N. KeyKOS Architecture. SIGOPS Operating SystemsReview 19, 4 (1985), 8–25.[25] CPU hotplug. https://wiki.linaro.org/WorkingGroups/PowerManagement/Archives/Hotplug, 12.04.14.[26] IPEK, E., KIRMAN, M., KIRMAN, N., AND MARTINEZ, J. F.Core Fusion: Accommodating Software Diversity in Chip Multiprocessors.In Proceedings of the 34th Annual InternationalSymposium on Computer Architecture (2007), pp. 186–197.[27] JOSHI, A. Twin-Linux: Running independent Linux Kernelssimultaneously on separate cores of a multicore system. In Proceedingsof the Linux Symposium (2010), pp. 101–108.[28] KLEIN, G., ELPHINSTONE, K., HEISER, G., ANDRONICK, J.,COCK, D., DERRIN, P., ELKADUWE, D., ENGELHARDT, K.,KOLANSKI, R., NORRISH, M., SEWELL, T.,TUCH, H., ANDWINWOOD, S. seL4: Formal verification of an OS kernel. InProceedings of the 22nd ACM Symposium on Operating SystemPrinciples (2009).[29] KONGETIRA, P.,AINGARAN, K., AND OLUKOTUN, K. Niagara:a 32-way multithreaded sparc processor. IEEE Micro 25, 2 (2005),21–29.[30] KOZUCH, M. A., KAMINSKY, M., AND RYAN, M. P. Migrationwithout virtualization. In Proceedings of the 12th Workshop onHot Topics in Operating Systems (2009), pp. 10–15.[31] KUMAR, R., FARKAS, K. I., JOUPPI, N. P., RANGANATHAN,P., AND TULLSEN, D. M. Single-ISA Heterogeneous Multi-CoreArchitectures: The Potential for Processor Power Reduction. InProceedings of the 36th Annual IEEE/ACM International Symposiumon Microarchitecture (2003), pp. 81–92.[32] Real-time Linux. https://rt.wiki.kernel.org/, 12.04.14.[33] LIU, R., KLUES, K., BIRD, S., HOFMEYR, S., ASANOVIĆ, K.,AND KUBIATOWICZ, J. Tessellation: Space-time partitioning in amanycore client OS. In Proceedings of the 1st USENIX Workshopon Hot Topics in Parallelism (2009).[34] MARR, D. T., DESKTOP, F. B., HILL, D. L., HINTON, G.,KOUFATY, D. A., MILLER, J. A., AND UPTON, M. Hyper-Threading Technology Architecture and Microarchitecture. IntelTechnology Journal (Feb 2002).[35] MENZI, D. Support for heterogeneous cores for Barrelfish. Master’sthesis, Department of Computer Science, ETH Zurich, July2011.[36] NOMURA, Y., SENZAKI, R., NAKAHARA, D., USHIO, H.,KATAOKA, T., AND TANIGUCHI, H. Mint: Booting multipleLinux kernels on a multicore processor. In Proceedings of theInternational Conference on Broadband and Wireless Computing,Communication and Applications (2011), pp. 555–560.[37] PANNEERSELVAM, S., AND SWIFT, M. M. Chameleon: Operatingsystem support for dynamic processors. In Proceedings of the17th International Conference on Architectural Support for ProgrammingLanguages and Operating Systems (2012), pp. 99–110.[38] Popcorn Linux. http://popcornlinux.org/, 12.04.14.[39] PostgreSQL. http://www.postgresql.org/, 12.04.14.[40] RHODEN, B., KLUES, K., ZHU, D., AND BREWER, E. Improvingper-node efficiency in the datacenter with new OS abstractions.In Proceedings of the 2nd ACM Symposium on Cloud Computing(2011), pp. 25:1–25:8.[41] SADINI, M., BARBALACE, A., RAVINDRAN, B., AND QUAGLIA,F. A Page Coherency Protocol for Popcorn Replicated-kernelOperating System. In Proceedings of the ManyCore ArchitectureResearch Community Symposium (MARC) (Oct. 2013).[42] SHAPIRO, J. S., SMITH, J. M., AND FARBER, D. J. EROS: AFast Capability System. In Proceedings of the 17th ACM Symposiumon Operating Systems Principles (1999), pp. 170–185.[43] SHELDON, B. H. Popcorn Linux: enabling efficient inter-corecommunication in a Linux-based multikernel operating system.Master’s thesis, Virginia Polytechnic Institute and State University,May 2013.[44] SINGHANIA, A., KUZ, I., AND NEVILL, M. Capability Managementin Barrelfish. Technical Note 013, Barrelfish Project, ETHZurich, December 2013.[45] Stress Load Generator. http://people.seas.harvard.edu/~apw/stress/, 12.04.14.[46] TPC-H. http://www.tpc.org/tpch/, 12.04.14.[47] VENKATESH, G., SAMPSON, J., GOULDING, N., GARCIA, S.,BRYKSIN, V.,LUGO-MARTINEZ, J., SWANSON, S., AND TAY-LOR, M. B. Conservation Cores: Reducing the energy of maturecomputations. In Proceedings of the 15th International Conferenceon Architectural Support for Programming Languages andOperating Systems (2010), pp. 205–218.[48] WENTZLAFF, D., GRUENWALD III, C., BECKMANN, N.,MODZELEWSKI, K., BELAY, A., YOUSEFF, L., MILLER, J.,AND AGARWAL, A. An operating system for multicore andclouds: Mechanisms and implementation. In ACM Symposium onCloud Computing (SOCC) (June 2010).[49] YEE, B., SEHR, D., DARDYK, G., CHEN, J. B., MUTH, R.,ORMANDY, T.,OKASAKA, S., NARULA, N., AND FULLAGAR,N. Native client: A sandbox for portable, untrusted x86 nativecode. In Proceedings of the 30th IEEE Symposium on Securityand Privacy (2009), pp. 79–93.[50] ZELLWEGER, G., SCHUEPBACH, A., AND ROSCOE, T. UnifyingSynchronization and Events in a Multicore OS. In Proceedings ofthe 3rd Asia-Pacific Workshop on Systems (2012).15USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 31

1. Spyro Ps1 Iso Fr
2. Spyro Psx Iso Ita
3. Spyro Collection Psx Iso

PSX on PSP / PSX2PSP eBoot ISOs

Quick Navigation:

Game Description & Reviews:

Videos and Image Media:

Upload a Screenshot/Add a Video:

Nominate for Retro Game of the Day:

Share with your Friends:

Support Emuparadise:

• Crash Bandicoot »
• Jak and Daxter: The Precursor Legacy »
• Jak 3 »
• Kya: Dark Lineage »
• MediEvil II »
• Haven: Call of the King »
• Rocket: Robot on Wheels »
• Gex 3: Deep Cover Gecko »
• Chameleon Twist 2 »
• Ratchet & Clank »
• Looney Tunes Back in Action »
• Croc 2 »
• Mort the Chicken »
• Klonoa 2: Lunatea's Veil »
• Croc: Legend of the Gobbos »
• Frogger »
• Kao the Kangaroo Round 2 »
• Kao the Kangaroo »
• 40 Winks »
• Gurumin: A Monstrous Adventure »
• Crash Bandicoot 2: Cortex Strikes Back »
• Tail Concerto »

Emulators:

Talk about Spyro the Dragon (USA):

Sony Playstation / PSX PS1 ISOs

Quick Navigation:

Game Description & Reviews:

Videos and Image Media:

• Uploaded by Rodrigo Romero Report
• Uploaded by Rodrigo Romero Report
• Uploaded by Rodrigo Romero Report

Upload a Screenshot/Add a Video:

Share with your Friends:

Support Emuparadise:

• Crash Bandicoot »
• Jak and Daxter: The Precursor Legacy »
• Jak 3 »
• Kya: Dark Lineage »
• MediEvil II »
• Haven: Call of the King »
• Rocket: Robot on Wheels »
• Gex 3: Deep Cover Gecko »
• Chameleon Twist 2 »
• Ratchet & Clank »
• Looney Tunes Back in Action »
• Croc 2 »
• Mort the Chicken »
• Klonoa 2: Lunatea's Veil »
• Croc: Legend of the Gobbos »
• Frogger »
• Kao the Kangaroo Round 2 »
• Kao the Kangaroo »
• 40 Winks »
• Gurumin: A Monstrous Adventure »
• Crash Bandicoot 2: Cortex Strikes Back »
• Tail Concerto »

Emulators:

Talk about Spyro the Dragon (E):

• Hirens Boot Cd Windows 10 32 Bit
• Hirens Boot Cd Windows 10 Blue Screen
• Hirens Boot Cd Windows 10 Alternative

1. Jun 20, 2018  Hiren’s BootCD PE includes only a single version of Windows 10 PE x64, supports new generation PCs and supports booting in UEFI mode (UEFI 64-bit, not 32-bit). If I’m not mistaken, like Bob.Omb’s Modified Win10PEx64 and Gandalf’s Windows 10PE, Windows 10 PE x64 from Hiren’s BootCD PE is also based on Win10PE SE.
2. Filename: HBCDPEx64.iso(Special thanks to our Supporters for maintaining fast and reliable mirror servers): Filesize: 1292.04 MB ( bytes) ISO MD5.

Reset Forgotten Windows 10 Password with Hiren's BootCD

1. Download the ISO image of Hiren's BootCD from its official website (about 593 MB). Burn the ISO file to your CD (or USB stick) using the freeware ISO2Disc.
2. After getting Hiren's BootCD ready, Boot your Windows 10 computer from CD (Change boot order on BIOS to set CD/DVD-ROM as the first boot device). Remember to change UEFI boot with Legacy and disable Secure Boot temporarily in BIOS.
3. You will be prompted with the Hiren's CD Menu, from there select Offline NT/2000/XP/Vista/7 Password Changer and press Enter. Don't use Kon-Boot as it doesn't support Windows 10/8 password bypass.
4. Hit Enter when you see the boot prompt.
5. The screen will list all your partitions and you need to select the partition where Windows is installed. In my screenshot you can see that 1) is the small 500Mb boot partition and 2) is the larger Windows partition. So I type 2 and press Enter.
6. Now it will ask to confirm the registry path (Windows/System32/config). The default registry path will be correct so just press Enter.
7. Painkiller: Purgatory (HD) was developed by Machineworks Northwest ( by Med-Art) for and and available on and.Soundtrack The soundtrack to the first game was written by, David Rippy, and Kevin McMullan and released in 2004. Best painkiller game.
8. We're going to reset forgotten Windows password, so choose the default choice Password reset (sam system security) by pressing Enter.
9. Choose the default choice Edit user data and passwords and press Enter again.
10. You will now see the user accounts on you computer. Type the username whose password you want to change and press Enter.
11. Type 1 and press Enter. This will clear/remove the password of your selected account.
12. Now type ! And hit Enter to quit the User Editor menu.
13. See how to set up SYNC 3 to periodically check for software updates, and learn how to install, download, and activate them using a Wi-Fi network connection. Jul 16, 2018  In this video, I explain the process for upgrading your Ford Sync 3 to version 3. While there is an option to update over WiFi, I found it slow and unreliable. The overall process is rather simple. Apr 05, 2018  In this video, we will go over a few things.What the update includes -How to turn on wifi -How to turn on auto updates with wifi There is Format Factory 4.6.2.0 Keygen Crack Key For U way to install the updates to a USB Flash drive as. Sync 3 version 3.0. Sep 22, 2017  Source: Ford Sync 3 Version 3.0 Release North America NAV and NONE NAV Versions Download - 2015+ S550 Mustang Forum (GT, GT350, Format Factory 4.6.2.0 Keygen Crack Key For U, Mach 1, Ecoboost) - Mustang6G.com. Reactions: Azampatti, tkronic, mrdouble99 and 6 others.
14. Enter q to exit chntpw Main Interactive Menu and press Enter.
15. Important step! Here must type y so it saves your password reset changes!
16. Greetings! Password has been cleared, now remove Hiren's BootCD and Press ctrl+Alt+Del button to restart. When your computer has rebooted, you can log into Windows 10 without a password.

Ready to try or buy?

conferenceproceedingsProceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation Broomfield, CO October 6–8, 201411th USENIX Symposium onOperating Systems Designand ImplementationBroomfield, COOctober 6–8, 2014Sponsored byIn cooperation with ACM SIGOPS

Thanks to Our OSDI ’14 SponsorsDiamond SponsorsGold SponsorsThanks to Our USENIX and LISA SIG SupportersUSENIX PatronsGoogle Microsoft Research NetApp VMwareSilver SponsorsUSENIX BenefactorsAkamai Facebook Hewlett-Packard IBM ResearchLinux Pro Magazine Puppet LabsUSENIX and LISA SIG PartnersCambridge Computer GoogleBronze SponsorsUSENIX PartnersCan Stock Photo EMCGeneral SponsorACM QueueADMIN magazineCRC PressDistributed ManagementTask Force (DMTF)Media Sponsors and Industry PartnersThe FreeBSD FoundationHPCwireInfoSec NewsLinux Pro MagazineLXerNo Starch PressO’Reilly MediaRaspberry Pi GeekUserFriendly.org© 2014 by The USENIX AssociationAll Rights ReservedThis volume is published as a collective work. Rights to individual papers remain with theauthor or the author’s employer. Permission is granted for the noncommercial reproduction ofthe complete work for educational or research purposes, Format Factory 4.6.2.0 Keygen Crack Key For U. Permission is granted to print, primarilyfor one person’s exclusive use, a single copy of these Proceedings. USENIX acknowledges alltrademarks herein.ISBN 978-1-931971-16-4

USENIX Format Factory 4.6.2.0 Keygen Crack Key For U of the 11th USENIX Symposiumon Operating Systems Design andImplementation (OSDI ’14)October 6–8, 2014Broomfield, CO

Conference OrganizersProgram Co-ChairsJason Flinn, University of MichiganHank Levy, University of WashingtonProgram CommitteeLorenzo Alvisi, The University of Texas at AustinDave Andersen, Carnegie Mellon UniversityRemzi Arpaci-Dusseau, University of Wisconsin—MadisonMihai Budiu, Microsoft ResearchGeorge Candea, École Polytechnique Fédérale deLausanne (EPFL)Peter Chen, University of MichiganAllen Clement, Google and Max Planck Institute forSoftware Systems (MPI-SWS)Landon Cox, Duke UniversityNick Feamster, Georgia Institute of TechnologyBryan Ford, Yale UniversityRoxana Geambasu, Columbia UniversityGernot Heiser, University of New South WalesAustralia/NICTAM. Format Factory 4.6.2.0 Keygen Crack Key For U BanabicJulian BangertPramod BhatotiaStefan BucurHaogang ChenVitaly ChipounovAustin ClementsDavid CockPatrick ColpCody CutlerRicardo DiasPedro FonsecaJoão GarciaQian GeZhenyu GuoJi HongAnuj KaliaManos KapritsosBaris KasikciVolodymyr KuznetsovDavid LazarGeoffrey LefebvreJoão LeitãoHyeontaek LimYunxin LiuYandong MaoSyed Akbar MehdiDutch MeyerExternal ReviewersHenrique MonizIulian MoraruToby MurrayMihir NanavatiNeha NarulaDaniel PeekRaluca Ada PopaDaniel PortoDan PortsZhengping QianShriram RajagopalanFranzi RoesnerChunzhi SuPhilippe SuterStephen TuJelle van den HooffJonas WagnerXi WangYang WangMing WuChao XieFan YangCristian ZamfirHuanchen ZhangTimmy Zhu

11th USENIX Symposium onOperating Systems Design and ImplementationOctober 6–8, 2014Broomfield, COMessage from the Program Chair. .ixMonday, October 6, 2014Who Put the Kernel in My OS Conference?Arrakis: The Operating System is the Control Plane. . 1Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug Woos, Arvind Krishnamurthy, and Thomas Anderson,University of Washington; Timothy Roscoe, ETH ZürichDecoupling Cores, Kernels, and Operating Systems. . 17Gerd Zellweger, Simon Gerber, Kornilios Kourtis, and Timothy Roscoe, ETH ZürichJitk: A Trustworthy In-Kernel Interpreter Infrastructure. . 33Xi Wang, David Lazar, Nickolai Zeldovich, and Adam Chlipala, MIT CSAIL; Zachary Tatlock, Universityof WashingtonIX: A Protected Dataplane Operating System for High Throughput and Low Latency. . 49Adam Belay, Stanford University; George Prekas, École Polytechnique Fédérale de Lausanne Format Factory 4.6.2.0 Keygen Crack Key For U Klimovic, Samuel Grossman, and Christos Kozyrakis, Stanford University; Edouard Bugnion, ÉcolePolytechnique Fédérale de Lausanne (EPFL)Data in the AbstractWillow: A User-Programmable SSD. 67Sudharsan Seshadri, Mark Gahagan, Sundaram Bhaskaran, Trevor Bunker, Arup De, Yanqin Jin, Yang Liu,and Steven Swanson, University of California, San DiegoPhysical Disentanglement in a Container-Based File System . 81Lanyue Lu, Yupu Zhang, Thanh Do, Samer Al-Kiswany, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, University of Wisconsin—MadisonCustomizable and Extensible Deployment for Mobile/Cloud Applications . 97Irene Zhang, Adriana Szekeres, Dana Van Aken, and Isaac Ackerman, University of Washington; Steven D.Gribble, Google and University of Washington; Arvind Krishnamurthy and Henry M. Levy, University ofWashingtonPebbles: Fine-Grained Data Management Abstractions for Modern Operating Systems . 113Riley Spahn and Jonathan Bell, Columbia University; Michael Lee, Format Factory 4.6.2.0 Keygen Crack Key For U, The University of Texas at Austin; SravanBhamidipati, Roxana Geambasu, and Gail Kaiser, Columbia UniversityMy InsecuritiesProtecting Users by Confining JavaScript with COWL. 131Deian Stefan and Edward Z. Yang, Stanford University; Petr Marchenko, Google; Alejandro Russo, ChalmersUniversity of Technology; Dave Herman, Mozilla; Brad Karp, University College London; David Mazières,Stanford UniversityCode-Pointer Integrity. . 147Volodymyr Kuznetsov, École Polytechnique Fédérale de Lausanne (EPFL); László Szekeres, Stony BrookUniversity; Mathias Payer, Purdue University; George Candea, École Polytechnique Fédérale de Lausanne(EPFL); R. Sekar, Stony Brook University; Dawn Song, University of Focus Magic 4.02 Crack Registration Code Free, Berkeley(Monday, October 6, continues on next page)

Ironclad Apps: End-to-End Security via Automated Full-System Verification. . 165Chris Hawblitzel, Jon Howell, and Jacob R. Lorch, Microsoft Research; Arjun Narayan, Universityof Pennsylvania; Bryan Parno, Format Factory 4.6.2.0 Keygen Crack Key For U, Microsoft Research; Danfeng Zhang, Cornell University; Brian Zill,Microsoft ResearchShill: A Secure Shell Scripting Language. . 183Scott Moore, Christos Dimoulas, Dan King, and Stephen Chong, Harvard UniversityVariety PackGPUnet: Networking Abstractions for GPU Programs . 201Sangman Kim, Seonggu Huh, Yige Hu, Xinya Zhang, and Emmett Witchel, The University of Texas at Austin;Amir Wated and Mark Silberstein, Technion—Israel Institute of TechnologyThe Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services . 217Michael Chow, University of Michigan; David Meisner, Facebook, Inc.; Jason Flinn, University of Michigan;Daniel Peek, Facebook, Inc.; Thomas F. Wenisch, University of MichiganEnd-to-end Performance Isolation Through Virtual Datacenters. . 233Sebastian Angel, The University of Texas at Austin; Hitesh Ballani, Thomas Karagiannis, Greg O’Shea,and Eno Thereska, Microsoft ResearchSimple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed . 249Data-Intensive SystemsDing Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain,and Michael Stumm, University of TorontoTuesday, October 7, 2014Head in the Format Factory 4.6.2.0 Keygen Crack Key For U Applications from an Untrusted Cloud with Haven. 267Andrew Baumann, Marcus Peinado, and Galen Hunt, Microsoft ResearchApollo: Scalable and Coordinated Scheduling for Cloud-Scale Computing . 285Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, and Jingren Zhou, Microsoft; Zhengping Qian, Ming Wu,and Lidong Zhou, Microsoft ResearchThe Power of Choice in Data-Aware Cluster Scheduling. . 301Shivaram Venkataraman and Aurojit Panda, University of California, Berkeley; Ganesh Ananthanarayanan,Microsoft Research; Michael J. Franklin and Ion Stoica, University of California, BerkeleyHeading Off Correlated Failures through Independence-as-a-Service . 317Ennan Zhai, Yale University; Ruichuan Chen, Format Factory 4.6.2.0 Keygen Crack Key For U, Bell Labs and Alcatel-Lucent; David Isaac Wolinsky andBryan Ford, Yale UniversityStorage Runs Hot and ColdCharacterizing Storage Workloads with Counter Stacks. . 335Jake Wires, Stephen Ingram, Zachary Drudi, Nicholas J. A. Harvey, and Andrew Warfield, Coho DataPelican: A Building Block for Exascale Cold Data Storage. . 351Shobana Balakrishnan, Richard Black, Austin Donnelly, Paul England, Adam Glass, Dave Harper, andSergey Legtchenko, Microsoft Research; Aaron Ogus, Microsoft; Eric Peterson and Antony Rowstron,Microsoft ResearchA Self-Configurable Geo-Replicated Cloud Storage System. . 367Masoud Saeida Ardekani, INRIA and Sorbonne Universités; Douglas B. Terry, Microsoft Research

f4: Facebook’s Warm BLOB Storage System. 383Subramanian Muralidhar, Facebook, Inc.; Wyatt Lloyd, University of Southern California and Facebook, Inc.;Sabyasachi Roy, Cory Hill, Ernest Lin, Weiwen Liu, Satadru Pan, Shiva Shankar, and Viswanath Sivakumar,Facebook, Inc.; Linpeng Tang, Princeton University and Facebook, Inc.; Sanjeev Kumar, Facebook, Inc.Pest ControlSAMC: Semantic-Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems. . 399Tanakorn Leesatapornwongsa and Mingzhe Hao, Format Factory 4.6.2.0 Keygen Crack Key For U, University of Chicago; Pallavi Joshi, NEC Labs America;Jeffrey F. Lukman, Surya University; Haryadi S. Gunawi, University of ChicagoSki: Exposing Kernel Concurrency Bugs through Systematic Schedule Exploration. . 415Pedro Fonseca, Max Planck Institute for Software Systems (MPI-SWS); Rodrigo Rodrigues, CITI/NOVAUniversity of Lisbon; Björn B. Brandenburg, Max Planck Institute for Software Systems (MPI-SWS)All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications. . 433Thanumalayan Sankaranarayana Pillai, Vijay Chidambaram, Ramnatthan Alagappan, Samer Al-Kiswany,Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, Format Factory 4.6.2.0 Keygen Crack Key For U, University of Wisconsin–MadisonTorturing Databases for Fun and Profit . 449Mai Zheng, The Ohio State University; Joseph Tucek, HP Labs; Dachuan Huang and Feng Qin, The Ohio StateUniversity; Mark Lillibridge, Format Factory 4.6.2.0 Keygen Crack Key For U, Elizabeth S. Yang, and Bill W. Zhao, HP Labs; Shashank Singh, The Ohio StateUniversityTransaction ActionFast Databases with Fast Durability and Recovery Through Multicore Parallelism . 465Wenting Zheng and Stephen Tu, Massachusetts Institute of Technology; Eddie Kohler, Harvard University;Barbara Liskov, Massachusetts Institute of TechnologyExtracting More Concurrency from Distributed Transactions. . 479Shuai Mu, Tsinghua University and New York University; Yang Cui and Yang Zhang, New York University;Wyatt Lloyd, University of Southern California and Facebook, Inc.; Jinyang Li, New York UniversitySalt: Combining ACID and BASE in a Distributed Database. . 495Chao Xie, Chunzhi Su, Manos Kapritsos, Yang Wang, Navid Yaghmazadeh, Lorenzo Alvisi, and Prince Mahajan,The University of Texas at AustinPhase Reconciliation for Contended In-Memory Transactions. . 511Neha Narula and Cody Cutler, MIT CSAIL; Eddie Kohler, Harvard University; Robert Morris, MIT CSAILWednesday, October 8, 2014Play It Again, SamEidetic Systems. . 525David Devecsery, Michael Chow, Xianzheng Dou, Jason Flinn, and Peter M. Chen, University of MichiganDetecting Covert Timing Channels with Time-Deterministic Replay. . 541Ang Chen, University of Pennsylvania; W. Brad Moore, Georgetown University; Hanjun Xiao, Format Factory 4.6.2.0 Keygen Crack Key For U, AndreasHaeberlen, and Linh Thi Xuan Phan, University of Pennsylvania; Micah Sherr and Wenchao Zhou, GeorgetownUniversityIdentifying Information Disclosure in Web Applications with Retroactive Auditing. . 555Haogang Chen, Taesoo Kim, Xi Wang, Nickolai Zeldovich, and M. Frans Kaashoek, MIT CSAIL(Wednesday, October 8, continues on next page)

Help Me LearnProject Adam: Building an Efficient and Scalable Deep Learning Training System. 571Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman, Microsoft ResearchScaling Distributed Machine Learning with the Parameter Server. . 583Mu Li, Carnegie Mellon University and Baidu; David G. Andersen and Jun Woo Park, Carnegie MellonUniversity; Alexander J. Smola, Carnegie Mellon University and Google, Inc.; Amr Ahmed, Vanja Josifovski,James Long, Eugene J. Shekita, Format Factory 4.6.2.0 Keygen Crack Key For U, and Bor-Yiing Su, Google, Inc.GraphX: Graph Processing in a Distributed Dataflow Framework. . 599Joseph E. Gonzalez, University of California, Berkeley; Reynold S. Xin, University of California, Berkeley, andDatabricks; Ankur Dave, Daniel Crankshaw, and Michael J. Franklin, University of California, Berkeley; IonStoica, University of California, Berkeley, and DatabricksHammers and SawsNail: A Practical Tool for Parsing and Generating Data Formats . 615Julian Bangert and Nickolai Zeldovich, MIT CSAILlprof: A Non-intrusive Request Flow Profiler for Distributed Systems. . 629Xu Zhao, Yongle Zhang, Format Factory 4.6.2.0 Keygen Crack Key For U, David Lion, Muhammad Faizan Ullah, Yu Luo, Ding Yuan, and Michael Stumm,University of TorontoPydron: Semi-Automatic Parallelization for Multi-Core and the Cloud . 645Stefan C. Format Factory 4.6.2.0 Keygen Crack Key For U, ETH Zürich and University of Applied Sciences Northwestern Switzerland; Gustavo Alonsoand Adam Amara, ETH Zürich; André Csillaghy, University of Applied Sciences Northwestern SwitzerlandUser-Guided Device Driver Synthesis. . 661Leonid Ryzhyk, University of Toronto, NICTA, and University of New South Wales; Adam Walker, NICTAand University of New South Wales; John Keys, Intel Corporation; Alexander Legg, NICTA and University ofNew South Wales; Arun Raghunath, Intel Corporation; Michael Stumm, University of Toronto; Mona Vij, IntelCorporation

Message from theOSDI ’14 Program Co-ChairsWe are delighted to welcome you to the 11th USENIX Symposium on Operating Systems Design and Implementation.This year’s program contains 42 papers representing some of the best research from academia and industry inthe area of experimental systems.This year, we received 228 submissions that met the formatting guidelines for the conference, Format Factory 4.6.2.0 Keygen Crack Key For U. We used three roundsof review, followed by electronic discussion after each round, to evaluate the submissions. Eleven papers weredesignated as “quick-accepts” based on reviews and electronic discussions. An additional 66 papers were discussedduring a day-and-a-half-long PC meeting of which 31 were selected for the program.For the first time at OSDI, the submission process included a response period in which authors could answerreviewer questions and address factual errors in the reviews. Responses had a measurable impact on PC meetingdiscussions, helping some papers and hurting others. Overall, Format Factory 4.6.2.0 Keygen Crack Key For U, we believe responses were quite useful in improvingthe fairness of the review process and the quality of the selected program.From the beginning of the process, our goal was to increase the number of papers appearing at OSDI to approximately40 accepted papers. As the chart below shows, the number of submissions to the conference has steadilyincreased over the years, while the number of accepted papers has not kept pace. At the same time, flagship conferencesin related fields (architecture, networking, programming languages, etc.) have all substantially increasedthe number of accepted papers. This harms our field in several ways: some of our top work does not appear at ourflagship conferences, our researchers are at a competitive disadvantage compared to researchers in other fields, andthe review process Format Factory 4.6.2.0 Keygen Crack Key For U excessively negative. We believe that the 42 papers in the final program reflect boththe overall strength and the breadth of research in our community.250200150100SubmittedAccepted5001996 1999 2000 2002 2004 2006 2008 2010 2012 2014We wish to thank the members of the program committee and the external review committee who produced 950reviews in approximately 2 months time. We especially thank the many members of the external review committeewho volunteered to do extra reviews when the number of submissions came in higher than we expected. We alsothank the external reviewers who helped with reviews and the USENIX staff who organized the conference.Jason Flinn, University of MichiganHank Levy, University of WashingtonOSDI ’14 Program Co-ChairsUSENIX Association11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) ix

Arrakis: The Operating System is the Control PlaneSimon Peter ∗ Jialin Li ∗ Irene Zhang ∗ Dan R. K. Ports ∗ Doug Woos ∗Arvind Krishnamurthy ∗ Thomas Anderson ∗ Timothy Roscoe †University of Washington ∗ ETH Zurich †AbstractRecent device hardware trends enable a new approach tothe design of network server operating systems. In a traditionaloperating system, the kernel mediates access todevice hardware by server applications, to enforce processisolation as well as network and disk security. We have designedand implemented a new operating system, Arrakis,that splits the traditional role of the kernel in two. Applicationshave direct access to virtualized I/O devices, allowingmost I/O operations to skip the kernel entirely, while thekernel is re-engineered to provide network and disk protectionwithout kernel mediation of every operation. Wedescribe the hardware and software changes needed totake advantage of this new abstraction, and we illustrate itspower by showing improvements of 2-5× in latency and9× in throughput for a popular persistent NoSQL storerelative Format Factory 4.6.2.0 Keygen Crack Key For U a well-tuned Linux implementation.1 IntroductionReducing the overhead of the operating system processabstraction has been a longstanding goal of systems design.This issue has become particularly salient with modernclient-server computing. The combination of high speedEthernet and low latency persistent memories is considerablyraising the efficiency bar for I/O intensive software.Many servers spend much of their time executing operatingsystem code: delivering interrupts, demultiplexing andcopying network packets, and maintaining file systemmeta-data. Server applications often perform very simplefunctions, such as key-value table lookup and storage, yettraverse the OS kernel multiple times per client request.These trends have led to a long line of research aimedat optimizing kernel code paths for various use cases:eliminating redundant copies in the kernel [45], reducingthe overhead for large numbers of connections [27],protocol specialization [44], resource containers [8, 39],direct transfers between disk and network buffers [45],interrupt steering [46], system call batching [49], hardwareTCP acceleration, etc. Much of this has been adopted inmainline commercial OSes, and yet it has been a losingbattle: we show that the Linux network and file Format Factory 4.6.2.0 Keygen Crack Key For U have latency and throughput many times worse thanthat achieved by the raw hardware.Twenty years ago, researchers proposed streamliningpacket handling for parallel computing over a network ofworkstations Format Factory 4.6.2.0 Keygen Crack Key For U mapping the network hardware directlyinto user space [19, 22, 54]. Although commerciallyunsuccessful at the time, the virtualization market has nowled hardware vendors to revive the idea [6, Format Factory 4.6.2.0 Keygen Crack Key For U, 38, 48], andalso extend it to disks [52, 53].This paper explores the OS implications of removingthe kernel from the data path for nearly all I/O operations.We argue that doing this must provide applications withthe same security model as traditional designs; it is easy toget good performance by extending the trusted computingbase to include application code, e.g., by allowingapplications unfiltered direct access to the network/disk.We demonstrate that operating system protection is notcontradictory with high performance. For our prototypeimplementation, a client request to the Redis persistentNoSQL store has 2× better read latency, 5× better write latency,and 9× better write throughput compared to Linux.We make three specific contributions:• We give an architecture for the division of labor betweenthe device hardware, kernel, and runtime for directnetwork and disk I/O by unprivileged processes, andwe show how to efficiently emulate our model for I/Odevices that do not fully support virtualization (§3).• We implement a prototype of our model as a set ofmodifications to the open source Barrelfish operatingsystem, running on commercially available multi-corecomputers and I/O device hardware (§3.8).• Format Factory 4.6.2.0 Keygen Crack Key For U use our prototype to quantify the potential benefitsof user-level I/O for several widely used networkservices, including a distributed object cache, Redis, anIP-layer middlebox, and an HTTP load balancer (§4).We show that significant gains are possible in terms ofboth latency and scalability, relative to Linux, in manycases without modifying the application programminginterface; additional gains are possible by changing thePOSIX API (§4.3).2 BackgroundWe first give a detailed breakdown of the OS and applicationoverheads in network and storage operations today,followed by a discussion of current hardware technologiesthat support user-level networking and I/O virtualization.To analyze the sources of overhead, we recordtimestamps at various stages of kernel and user-space processing.Our experiments are conducted on a six machinecluster consisting of 6-core Intel Xeon E5-2430 (SandyBridge) systems at 2.2 GHz running Ubuntu Linux 13.04.1USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 1

LinuxArrakisReceiver running CPU idle Arrakis/P Arrakis/NNetwork stackin 1.26 (37.6%) 1.24 (20.0%) 0.32 (22.3%) 0.21 (55.3%)out 1.05 (31.3%) 1.42 (22.9%) 0.27 (18.7%) 0.17 (44.7%)Scheduler 0.17 (5.0%) 2.40 (38.8%) - -Copyin 0.24 (7.1%) 0.25 (4.0%) 0.27 (18.7%) -out 0.44 (13.2%) 0.55 (8.9%) 0.58 (40.3%) -Kernel crossingreturn 0.10 (2.9%) 0.20 (3.3%) - -syscall 0.10 (2.9%) 0.13 (2.1%) - -Total 3.36 (σ =0.66) 6.19 (σ =0.82) 1.44 (σ

AppApp UserspaceCore Core Core KernelIncoming Q's Outgoing Q's NICSystem call duration [us]9080706050403020100writefsyncbtrfs 1KBbtrfs 64Bext4 1KBext4 64Bext3 1KBext3 64Bext2 1KBext2 64BFigure 1: Linux networking architecture and workflow.removed, there is an opportunity to rethink the POSIX APIfor more streamlined networking. In addition to a POSIXcompatible interface, Arrakis provides a native interface(Arrakis/N) which supports true zero-copy I/O.2.2 Storage Stack OverheadsTo illustrate the Format Factory 4.6.2.0 Keygen Crack Key For U of today’s OS storage stacks,we conduct an experiment, where we execute small writeoperations immediately followed by an fsync 1 system callin a tight loop of 10,000 iterations, measuring each operation’slatency. We store the file system on a RAM disk,so the measured latencies represent purely CPU overhead.The overheads shown in Figure 2 stem from data copyingbetween user and kernel space, parameter and accesscontrol checks, block and inode allocation, virtualization(the VFS layer), snapshot maintenance (btrfs), as well asmetadata updates, in many cases via a journal [53].While historically these CPU overheads have beeninsignificant compared to disk access time, recent hardwaretrends have drastically reduced common-case writestorage latency by introducing flash-backed DRAM ontothe device. In these systems, OS storage stack overheadbecomes a major factor. We measured average writelatency to our RAID cache to be 25 µs. PCIe-attachedflash storage adapters, like Fusion-IO’s ioDrive2, reporthardware access latencies as low as 15 µs [24]. Incomparison, OS storage stack overheads are high, addingbetween 40% and 200% for the extended file systems,depending on journal use, and up Format Factory 4.6.2.0 Keygen Crack Key For U 5× for btrfs. The largestandard deviation for btrfs stems from its highly threadeddesign, used to flush non-critical file system metadata andupdate reference counts in the background.2.3 Application OverheadsWhat do these I/O stack overheads mean to operationlatencies within a typical datacenter application? Considerthe Redis [18] NoSQL store. Redis persists each write viaan operational log (called append-only file) 2 and servesreads from an in-memory data structure.To serve a read, Redis performs a series of operations:First, epoll is called to await data for reading, followed1 We also tried fdatasync, with negligible difference in latency.2 Redis also supports snapshot persistence because of the highper-operation overhead imposed by Linux.Figure 2: Average overhead in µs of various Linux file systemimplementations, when conducting small, persistent writes.Error bars show standard deviation.by recv to receive a request. After receiving, the (textual)request is parsed and the key looked-up in memory. Oncefound, a response is prepared and then, after epoll iscalled again to check whether the socket is ready, sentto the client via send. For writes, Format Factory 4.6.2.0 Keygen Crack Key For U, Redis additionallymarshals the operation into log format, writes the logand waits for persistence (via the fsync call) beforeresponding. Redis also spends time in accounting, accesschecks, and connection handling (Other row in Table 2).Table 2 shows that a total of 76% of the latency in anaverage read hit on Linux is due to socket operations.In Arrakis, we reduce socket operation latency by 68%.Similarly, 90% of the latency of a write on Linux is due toI/O operations. In Arrakis we reduce I/O latency by 82%.We can also see that Arrakis reduces some applicationleveloverheads. This is due to better cache behavior of theuser-level I/O stacks and the control/data plane separationevading all kernel crossings. Arrakis’ write latency is stilldominated by storage access latency (25µs in our system).We expect the gap between Linux and Arrakis performanceto widen as faster storage devices appear on the market.2.4 Hardware I/O VirtualizationSingle-Root I/O Virtualization (SR-IOV) [38] is ahardware technology intended to support high-speed I/Ofor multiple virtual machines sharing a single physicalmachine. An SR-IOV-capable I/O adapter appears on thePCIe interconnect as a single “physical function” (PCIparlance for a device) which can in turn dynamically createadditional “virtual functions”. Each of these resembles aPCI device, which can be directly mapped into a differentvirtual machine and access can be protected via IOMMU(e.g. Intel’s VT-d [34]). To the guest operating system,each virtual function can be programmed as if it wasa regular physical device, with a normal device driverand an unchanged I/O stack. Hypervisor software withaccess to the physical hardware (such as Domain 0 ina Xen [9] installation) creates and deletes these virtualfunctions, and configures filters in the SR-IOV adapterto demultiplex hardware operations to different virtualfunctions and therefore different guest operating systems.3USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 3

Read hitDurable writeLinux Arrakis/P Linux Arrakis/Pepoll 2.42 (27.91%) 1.12 (27.52%) 2.64 (1.62%) 1.49 (4.73%)recv 0.98 (11.30%) 0.29 (7.13%) 1.55 (0.95%) 0.66 (2.09%)Parse input 0.85 (9.80%) 0.66 (16.22%) 2.34 (1.43%) 1.19 (3.78%)Lookup/set key 0.10 (1.15%) 0.10 (2.46%) 1.03 (0.63%) 0.43 (1.36%)Log marshaling - - 3.64 (2.23%) 2.43 (7.71%)write - - 6.33 (3.88%) 0.10 (0.32%)fsync - - 137.84 (84.49%) 24.26 (76.99%)Prepare response 0.60 (6.92%) 0.64 (15.72%) 0.59 (0.36%) 0.10 (0.32%)send 3.17 (36.56%) 0.71 (17.44%) 5.06 (3.10%) 0.33 (1.05%)Other 0.55 (6.34%) 0.46 (11.30%) 2.12 (1.30%) 0.52 (1.65%)Total 8.67 (σ =2.55) 4.07 (σ =0.44) 163.14 (σ =13.68) 31.51 (σ =1.91)99th percentile 15.21 4.25 188.67 35.76Table 2: Overheads in the Redis NoSQL store for memory reads (hits) and durable writes (legend in Table 1).In Arrakis, we use SR-IOV, the IOMMU, and supportingadapters to provide direct Format Factory 4.6.2.0 Keygen Crack Key For U access to I/Odevices. This is a modern implementation of an idea whichwas implemented twenty years ago with U-Net [54], butgeneralized to flash storage and Ethernet network adapters.To make user-level I/O stacks tractable, we need ahardware-independent device model and API that capturesthe important features of SR-IOV adapters [31, 40, 41, 51];a hardware-specific device driver matches our API to thespecifics of the particular device. We discuss this modelin the next section, along with potential improvements tothe existing hardware to better support user-level I/O.Remote Direct Memory Access (RDMA) is anotherpopular model for user-level networking [48]. RDMAgives applications the ability to read from or write to aregion of virtual memory on a remote machine directlyfrom user-space, bypassing the operating system kernel onboth sides. The intended use case is for a parallel programto be able to directly read and modify its data structureseven when Format Factory 4.6.2.0 Keygen Crack Key For U are stored on remote machines.While RDMA provides the performance benefitsof user-level networking to parallel applications, it ischallenging to apply the model to a broader class of clientserverapplications [21]. Most importantly, RDMA ispoint-to-point. Each participant receives an authenticatorproviding it permission to remotely read/write a particularregion of memory. Since clients in client-server computingare not mutually trusted, the hardware would need to keepa separate region of memory for each active connection.Therefore we do not consider RDMA operations here.3 Design and ImplementationArrakis has the following design goals:• Minimize kernel involvement for data-plane operations:Arrakis is designed to limit or remove kernel mediationfor most I/O operations. I/O requests are routedto and from the application’s address space withoutrequiring kernel involvement and without sacrificingsecurity and isolation properties.• Transparency to the application programmer: Arrakisis designed to significantly improve performancewithout requiring modifications to applications writtento the POSIX API. Additional performance gains arepossible if the developer can modify the application.• Appropriate OS/hardware abstractions: Arrakis’ abstractionsshould be sufficiently flexible to efficientlysupport a broad range of I/O patterns, scale well on multicoresystems, and support application requirements forlocality and load balance.In this section, we show how we achieve these goals inArrakis. We describe an ideal set of hardware facilities thatshould be present to take full advantage of this architecture,and we detail the design of the control plane and dataplane interfaces that we provide to the application. Finally,we describe our implementation of Arrakis based on theBarrelfish operating system.3.1 Architecture OverviewArrakis Format Factory 4.6.2.0 Keygen Crack Key For U I/O hardware with support for virtualization,and Figure 3 shows the overall architecture. In thispaper, we focus on hardware that can present multipleinstances of itself to the operating system and the applicationsrunning on the node. For each of these virtualizeddevice instances, the underlying physical device providesunique memory mapped register files, descriptor queues,and interrupts, hence allowing the control plane to mapeach device instance to a separate protection domain. Thedevice exports a management interface that is accessiblefrom the control plane in order to create or destroy virtualdevice instances, associate individual instances withnetwork flows or storage areas, and allocate shared resourcesto the different instances. Applications conduct I/O44 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

NICApplibosVNICSwitchApplibosVNICUserspaceVSICVSAVSAControlPlaneVSICVSAStorage ControllerFigure 3: Arrakis architecture. The storage controller mapsVSAs to physical storage.through their protected virtual device instance without requiringkernel intervention. In order to perform these operations,applications rely on a user-level I/O stack that is providedas a library. The user-level I/O stack can be tailored tothe application as it can assume exclusive access to a virtualizeddevice instance, allowing us to remove any featuresnot necessary for the application’s functionality. Finally,(de-)multiplexing operations and security checks are notneeded in this dedicated environment and can be removed.The user naming and protection model is unchanged.A global naming system is provided by the control plane.This is especially important for sharing stored data.Applications implement their own storage, Format Factory 4.6.2.0 Keygen Crack Key For U, while thecontrol plane manages naming and coarse-grain allocation,by associating each application with the directories andfiles it manages. Other applications can still read thosefiles by indirecting through the kernel, which hands thedirectory or read request to the appropriate application.3.2 Hardware ModelA key element of our work is Format Factory 4.6.2.0 Keygen Crack Key For U develop a hardwareindependentlayer for virtualized I/O—that is, a devicemodel providing an “ideal” set of hardware features.This device model captures the functionality requiredto implement in hardware the data plane operations of atraditional kernel. Our model resembles what is alreadyprovided by some hardware I/O adapters; we hope it willprovide guidance as to what is needed to support secureuser-level networking and storage.In particular, we assume our network devices providesupport for virtualization by presenting themselves asmultiple virtual network interface cards (VNICs) andthat they can also multiplex/demultiplex packets based oncomplex filter expressions, directly to queues that can bemanaged entirely in user space without the need for kernelintervention. Similarly, each storage controller exposesmultiple virtual storage interface controllers (VSICs)in our model. Each VSIC provides independent storagecommand queues (e.g., Format Factory 4.6.2.0 Keygen Crack Key For U, of SCSI or ATA format) that aremultiplexed by the hardware. Associated with each suchvirtual interface card (VIC) are queues and rate limiters.KernelVNICs also provide filters and VSICs provide virtualstorage areas. We discuss these components below.Queues: Each VIC contains multiple pairs of DMAqueues for user-space send and receive. The exact formof these VIC queues could depend on the specifics ofthe I/O interface card. For example, it could support ascatter/gather interface Format Factory 4.6.2.0 Keygen Crack Key For U aggregate multiple physicallydisjointmemory regions into a single data transfer. ForNICs, it could also optionally support hardware checksumoffload and TCP segmentation facilities. These featuresenable I/O to be handled more efficiently by performingadditional work in hardware. In such cases, the Arrakissystem offloads operations and further reduces overheads.Transmit and receive filters: A transmit filter is a predicateon network packet header fields that the hardwarewill use to determine whether to send the packet or discardit (possibly signaling an error either to the application orthe OS). The transmit filter prevents applications fromspoofing information such as IP addresses and VLANtags and thus eliminates kernel mediation to enforce thesesecurity checks. It can also be used to limit an applicationto communicate with only a pre-selected set of nodes.A receive filter is a similar predicate that determineswhich packets received from the network will be deliveredto a VNIC and to a specific queue associated with the targetVNIC. For example, a VNIC can be set up to receive allpackets sent to a particular port, so both connection setupand data transfers can happen at user-level. Installationof transmit and receive filters are privileged operationsperformed via the kernel control plane.Virtual storage areas: Storage controllers need to providean interface via their physical function to map virtualstorage areas (VSAs) to extents of physical drives, andassociate them with VSICs. A typical VSA will be largeenough to allow the application to ignore the underlyingmultiplexing—e.g., multiple erasure blocks on flash, orcylinder groups on disk. An application can store multiplesub-directories and files in a single VSA, providing precisecontrol over multi-object serialization constraints.A VSA is thus a persistent segment [13]. Applicationsreference blocks in the VSA using virtual offsets,converted by hardware into physical storage locations. AVSIC may have multiple VSAs, and each VSA may bemapped into multiple VSICs for interprocess sharing.Bandwidth allocators: This includes support for resourceallocation mechanisms such as rate limiters andpacing/traffic shaping of I/O. Once a frame has beenremoved from a transmit rate-limited or paced queue, thenext time another frame could be fetched from that queueis regulated by the rate limits and the inter-packet pacingcontrols associated with the queue. Installation of thesecontrols are also privileged operations.5USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 5

In addition, we assume that the I/O device driversupports an introspection interface allowing the controlplane to query for resource Format Factory 4.6.2.0 Keygen Crack Key For U (e.g., the number ofqueues) and check for the availability of hardware supportfor I/O processing (e.g., checksumming or segmentation).Network cards that support SR-IOV have the keyelements of this model: they allow the creation of multipleVNICs that f.lux Features Crack Key For U may have multiple send and receivequeues, and support at least rudimentary transmit andreceive filters. Not all NICs provide the rich filtering semanticswe desire; for example, the Intel 82599 can filter onlybased on source or destination MAC addresses and VLANtags, not arbitrary predicates on header fields. However,this capability is within reach: some network cards (e.g.,Solarflare 10Gb adapters) can already filter packets on allheader fields, and the hardware support required for moregeneral VNIC transmit and receive filtering is closely relatedto that used for techniques like Receive-Side Scaling,which is ubiquitous in high-performance network cards.Storage controllers have some parts of the technologyneeded to provide the interface we describe. For example,RAID adapters have a translation layer that is ableto provide virtual disks above physical extents, andSSDs use a flash translation layer for wear-leveling.SCSI host-bus adapters support SR-IOV technology forvirtualization [40, 41] and can expose multiple VSICs,and the NVMe standard proposes multiple commandqueues for scalability [35]. Only the required protectionmechanism is missing. We anticipate VSAs to be allocatedin large chunks and thus hardware protection mechanismscan be coarse-grained and lightweight.Finally, Format Factory 4.6.2.0 Keygen Crack Key For U number of hardware-supported VICsmight be limited. The 82599 [31] and SAS3008 [41]support 64. This number is adequate with respect to thecapabilities of the rest of the hardware (e.g., the numberof CPU cores), but we expect it to rise. The PCI workinggroup Format Factory 4.6.2.0 Keygen Crack Key For U already ratified an addendum to SR-IOV thatincreases the supported number of virtual functions to2048. Bandwidth allocation within the 82599 is limitedto weighted round-robin scheduling and rate limiting ofeach of the Format Factory 4.6.2.0 Keygen Crack Key For U transmit/receive queues. Recent researchhas demonstrated that precise rate limiting in hardwarecan scale to tens of thousands of traffic classes, enablingsophisticated bandwidth allocation policies [47].Arrakis currently assumes hardware that can filterand demultiplex flows at a level (packet headers, etc.)corresponding roughly to a traditional OS API, but nohigher. An open question is the extent to which hardwarethat can filter on application-level properties (includingcontent) would provide additional performance benefits.3.3 VSIC EmulationTo validate our model given limited support from storagedevices, we developed prototype VSIC support bydedicating a processor core to emulate the functionalitywe expect from hardware. The same technique can be usedto run Arrakis on systems without VNIC support.To handle I/O requests from the OS, our RAID controllerprovides one request and one response descriptor queueof fixed size, implemented as circular buffers along witha software-controlled register (PR) pointing to the head ofthe request descriptor queue. Request descriptors (RQDs)have a size of 256 bytes and contain a SCSI command, ascatter-gather array of system memory ranges, and a targetlogical disk number. The SCSI command specifies the typeof operation (read or write), total transfer size and on-diskbase logical block address (LBA). The scatter-gather arrayspecifies the request’s corresponding regions in systemmemory. Response descriptors refer to completed RQDsby their queue entry and contain a completion code. AnRQD can be reused only after its response is received.We replicate this setup for each VSIC by allocatingqueue pairs and register files of the same format in systemmemory mapped into applications and to a dedicatedVSIC core. Like the 82599, we limit the maximum numberof VSICs to 64. In addition, the VSIC core keeps anarray of up to 4 VSA mappings for each VSIC that isprogrammable only from the control plane. The mappingscontain the size of the VSA and an LBA offset within alogical disk, effectively specifying an extent.In the steady state, the VSIC core polls each VSIC’sPR and the latest entry of the response queue of thephysical controller in a round-robin fashion. Whena new RQD is posted via PR i on VSIC i, the VSICcore interprets the RQD’s logical disk number n asa VSA mapping entry and checks whether the correspondingtransfer fits within that VSA’s boundaries (i.e.,RQD.LBA + RQD.size ≤ VSA n .size). If so, the corecopies the RQD to the physical controller’s queue, addingVSA n .offset to RQD.LBA, and sets an unused RQD fieldto identify the corresponding RQD in the source VSICbefore updating the controller’s PR register. Upon aresponse from the controller, the VSIC core copies theresponse to the corresponding VSIC response queue.We did not consider VSIC interrupts in our prototype.They can be supported via inter-processor interrupts.To support untrusted applications, our prototype has totranslate virtual addresses. This requires it to traverse applicationpage tables for each entry in an RQD’s scatter-gatherarray. In a real system, the IOMMU carries out this task.On a microbenchmark of 10,000 fixed size write operationsof 1KB via a single VSIC to a single VSA, the averageoverhead of the emulation is 3µs. Executing virtualizationcode takes 1µs on the VSIC core; the other 2µs are due tocache overheads that we did not quantify further. To measurethe expected VSIC performance with direct hardwaresupport, we map the single RAID hardware VSIC directlyinto the application memory; we report those results in §4.66 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

3.4 Control Plane InterfaceThe interface between an application and the Arrakiscontrol plane is used to request resources from the systemand direct I/O flows to and from user programs. Thekey abstractions presented by this interface are VICs,doorbells, filters, VSAs, and rate specifiers.An application can create and delete VICs, and associatedoorbells with particular events on particular VICs. Adoorbell is an IPC end-point used to notify the applicationthat an event (e.g. packet arrival or I/O completion) hasoccurred, and is discussed below. VICs are hardwareresources and so Arrakis must allocate them amongapplications according to an OS policy. Currently thisis done on a first-come-first-served basis, followed byspilling to software emulation (§3.3).Filters have a type (transmit or receive) and a predicatewhich corresponds to a convex sub-volume of thepacket header space (for example, obtained with a setof mask-and-compare operations), Format Factory 4.6.2.0 Keygen Crack Key For U. Filters can be usedto specify ranges of IP addresses and port numbersassociated with valid packets transmitted/received at eachVNIC. Filters are a better abstraction for our purposesthan a conventional connection identifier (such as aTCP/IP 5-tuple), since they can encode a wider variety ofcommunication patterns, as well as subsuming traditionalport allocation and interface specification.For example, in the “map” phase of a MapReduce jobwe would like the application to send to, and receive from,an entire class of machines using the same communicationend-point, but nevertheless isolate the data comprising theshuffle from other data. As a second example, web serverswith a high rate of incoming TCP connections can run intoscalability problems processing connection requests [46].In Arrakis, a single filter can safely Format Factory 4.6.2.0 Keygen Crack Key For U both a listeningsocket and all subsequent connections to that socket,allowing server-side TCP connection establishment toavoid kernel mediation.Applications create a filter with a control plane operation.In the common case, a simple higher-level wrappersuffices: filter = create_filter(flags, peerlist, servicelist).flags specifies the filter direction (transmit or receive)and whether the filter refers to the Ethernet, IP, TCP, orUDP header. peerlist is a list of accepted communicationpeers specified according to the filter type, and servicelistcontains a list of accepted service addresses (e.g., portnumbers) for the filter. Wildcards are permitted.The call to create_filter returns filter, a kernelprotectedcapability conferring authority to send orreceive packets matching its predicate, and which canthen be assigned to a specific queue on a VNIC. VSAs areacquired and assigned to VSICs in a similar fashion.Finally, a rate specifier can also be assigned to a queue,either to throttle incoming traffic (in the network receivecase) or pace outgoing packets and I/O requests. Ratespecifiers and filters associated with a VIC queue canbe updated dynamically, but all such updates requiremediation from the Arrakis control plane.Our network filters are less expressive than OpenFlowmatching tables, in that they do not support priority-basedoverlapping matches. This is a deliberate choice based onhardware capabilities: NICs today only support simplematching, and to support priorities in the API would leadto unpredictable consumption of hardware resourcesbelow the abstraction. Our philosophy is therefore tosupport expressing such policies only when the hardwarecan implement them efficiently.3.5 File Name LookupA design principle in Arrakis is to separate file namingfrom implementation. In a traditional system, the fullyqualifiedfilename specifies the file system used to storethe file and thus its metadata format. To work around this,many applications build their own metadata indirectioninside the file abstraction [28]. Instead, Arrakis providesapplications direct control over VSA storage allocation:an application is free to use its VSA to store metadata,directories, and file data. To allow other applications accessto its data, an application can export file and directorynames to the kernel virtual file system (VFS). To the rest ofthe VFS, an application-managed file or directory appearslike a remote mount point—an indirection to a file systemimplemented elsewhere. Operations within the file ordirectory are handled locally, without kernel intervention.Other applications can gain access to these files in threeways. By default, the Arrakis application library managingthe VSA exports a file server interface; other applicationscan use normal POSIX API calls via user-level RPC to theembedded library file server. This library can also run asa standalone process to provide access when the originalapplication is not active. Just like a regular mounted filesystem, the library needs to implement only functionalityrequired for file access on its VSA and may choose to skipany POSIX features that it does not directly support.Second, VSAs can be mapped into multiple processes.If an application, like a virus checker or backup system,has both permission to read the application’s metadata andthe appropriate library support, it can directly access thefile data in the VSA. In this case, access control is donefor the entire VSA and not per file or directory. Finally,the user can direct the originating application to exportits data into a standard format, such as a PDF file, storedas a normal file in the kernel-provided file system.The combination of VFS and library code implementPOSIX semantics seamlessly. For example, Format Factory 4.6.2.0 Keygen Crack Key For U executerights are revoked from a directory, Format Factory 4.6.2.0 Keygen Crack Key For U, the VFS preventsfuture traversal of that directory’s subtree, but existingRPC connections to parts of the subtree may remain intactuntil closed. This is akin to a POSIX process retaining a7USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 7

subdirectory as the current working directory—relativetraversals are still permitted.3.6 Network Data Plane InterfaceIn Arrakis, applications send and receive network packetsby directly communicating with hardware. The dataplane interface is therefore implemented in an applicationlibrary, allowing it to be co-designed with the application[43]. The Arrakis library provides two interfaces toapplications. We describe the native Arrakis interface,which departs slightly from the POSIX standard tosupport true zero-copy I/O; Arrakis also provides a POSIXcompatibility layer that supports unmodified applications.Applications send and receive packets on queues, whichhave previously been assigned filters as described above.While filters can include IP, TCP, and UDP field predicates,Arrakis does not require the hardware to perform protocolprocessing, only multiplexing. In our implementation, Arrakisprovides a user-space network stack above the dataplane interface. This stack is designed to maximize bothlatency and throughput. We maintain a clean separation betweenthree aspects of packet transmission and reception.Firstly, packets are transferred asynchronously betweenthe network and main memory using conventional DMAtechniques using rings of packet buffer descriptors.Secondly, the application transfers ownership of a transmitpacket to the network hardware by enqueuing a chainof buffers onto the hardware descriptor rings, and acquiresa received packet by the reverse process. This is performedby two VNIC driver functions. send_packet(queue,packet_array) sends a packet on a queue; the packet isspecified Format Factory 4.6.2.0 Keygen Crack Key For U the scatter-gather array packet_array, andmust conform to a filter already associated with the queue.receive_packet(queue) = packet receives a packet froma queue and returns a pointer to it. Both operations areasynchronous. packet_done(packet) returns ownershipof a received packet to the VNIC.For optimal performance, the Arrakis stack would interactwith the hardware queues not through these calls butdirectly via compiler-generated, optimized code tailored tothe NIC descriptor format. However, the implementationwe report on in this paper uses function calls to the driver.Thirdly, we handle asynchronous notification of eventsusing doorbells associated with queues. Doorbells aredelivered directly from hardware to user programs viahardware virtualized interrupts when applications arerunning and via the control plane to invoke the schedulerwhen applications are not running. In the latter case,higher latency is tolerable. Doorbells are exposed toArrakis programs via regular event delivery mechanisms(e.g., a file descriptor event) and are fully integratedwith existing I/O multiplexing interfaces (e.g., select).They are useful both to notify an application of generalavailability of packets in receive queues, as well as alightweight notification mechanism for I/O completionand the reception of packets in high-priority queues.This design results in a protocol stack that decoupleshardware from software as much as possible using thedescriptor rings as a buffer, maximizing throughput andminimizing overhead under high packet rates, yielding lowlatency. On top of this native interface, Arrakis providesPOSIX-compatible sockets. This compatibility layerallows Arrakis to support unmodified Linux applications.However, we show that performance gom player codec Crack Key For U can be achievedby using the asynchronous native interface.3.7 Storage Data Plane InterfaceThe low-level storage API provides a set of commandsto asynchronously read, write, and flush hardware cachesat any offset and of arbitrary size in a VSA via a commandqueue in the associated VSIC. To do so, the caller providesan array of virtual memory ranges (address and size)in RAM to be read/written, the VSA identifier, queuenumber, Format Factory 4.6.2.0 Keygen Crack Key For U, and matching array of ranges (offset and size)within the VSA. The implementation enqueues thecorresponding commands to the VSIC, coalescing andreordering commands if this makes sense to the underlyingmedia. I/O completion events are reported using doorbells.On top of this, a POSIX-compliant file system is provided.We have also designed a library of persistent data structures,Caladan, to take advantage of low-latency storagedevices. Persistent data structures can be more efficientthan a simple read/write interface provided by file systems.Their drawback is a lack of backwards-compatibility to thePOSIX API. Our design goals for persistent data structuresare that (1) operations are immediately persistent, (2) thestructure is robust versus crash failures, and (3) operationshave minimal latency.We have designed persistent log and queue datastructures according to these goals and modified a numberof applications to use them (e.g., §4.4). These datastructures manage all metadata required for persistence,which allows tailoring of that data to reduce latency. Forexample, metadata can be allocated along with each datastructure entry and persisted in a single hardware writeoperation. For the log and queue, the only metadata thatneeds to be kept is where they start and end. Pointerslink entries to accommodate wrap-arounds and holes,optimizing for linear access and efficient prefetch ofentries. By contrast, a filesystem typically has separateinodes to manage block allocation. The in-memory layoutof Caladan structures is as stored, eliminating marshaling.The log API includes operations to open and close a log,create log entries (for metadata allocation), append them tothe log (for persistence), iterate through the log (for reading),and trim the log. The queue API adds a pop operationto combine trimming and reading the queue. Persistenceis asynchronous: an append operation returns immediately88 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

with a callback on persistence. This allows us to maskremaining write latencies, e.g., by optimistically preparingnetwork responses to clients, while an entry is persisted.Entries are allocated in multiples of the storagehardware’s minimum transfer unit (MTU—512 bytesfor our RAID controller, based on SCSI) and contain aheader that denotes the true (byte-granularity) size of theentry and points to the offset of the next entry in a VSA.This allows entries to be written directly from memory,without additional marshaling. At the end of each entryis a marker that is used to determine whether an entrywas fully written (empty VSA space is always zero). Byissuing appropriate cache flush commands to the storagehardware, Caladan ensures that markers are written afterthe rest of the entry (cf. [17]).Both data structures are identified by a header at thebeginning of the VSA that contains a version number, thenumber of entries, the MTU of the storage device, and apointer to the beginning and end of the structure within theVSA. Caladan repairs a corrupted or outdated header lazilyin the background upon opening, by looking for additional,complete entries from the purported end of the structure.3.8 ImplementationThe Arrakis operating Format Factory 4.6.2.0 Keygen Crack Key For U is based upon a fork of theBarrelfish [10] multicore OS code base [1]. We added33,786 lines of code to the Format Factory 4.6.2.0 Keygen Crack Key For U code base in orderto implement Arrakis. Barrelfish lends itself well to ourapproach, as it already provides a library OS. We couldhave also chosen to base Arrakis on the Xen [9] hypervisoror the Intel Data Plane Development Kit (DPDK) [32]running on Linux; both provide user-level access to thenetwork interface via hardware virtualization. However,implementing a library OS from scratch on top of amonolithic OS would have been more time consumingthan extending the Barrelfish library OS.We extended Barrelfish with support for SR-IOV, whichrequired modifying the existing PCI device manager to recognizeand handle SR-IOV extended PCI capabilities. Weimplemented a physical function driver for the Intel 8259910G Ethernet Adapter [31] that can initialize and managea number of virtual functions. We also implemented avirtual function driver for the 82599, including support forExtended Message Signaled Interrupts (MSI-X), which areused to deliver per-VNIC doorbell events to applications.Finally, we implemented drivers for the Intel IOMMU [34]and the Intel RS3 family of RAID controllers [33]. Inaddition—to support our benchmark applications—weadded several POSIX APIs that were not implemented inthe Barrelfish code base, such as POSIX threads, manyfunctions of the POSIX sockets API, as well as the epollinterface found in Linux to allow scalable polling of a largenumber of file descriptors. Barrelfish already supportsstandalone user-mode device drivers, akin to those foundin microkernels. We created shared library versions of thedrivers, which we link to each application.We have developed our Format Factory 4.6.2.0 Keygen Crack Key For U user-level network stack,Extaris. Extaris is a shared library that interfaces directlywith the virtual function device driver and provides thePOSIX sockets API and Arrakis’s native API to theapplication. Extaris is based in part on the low-levelpacket processing code of the lwIP network stack [42]. Ithas identical capabilities to lwIP, but supports hardwareoffload of layer 3 and 4 checksum operations and doesnot require any synchronization points or serializationof packet operations. We have also developed our ownstorage API layer, as described in §3.7 and our library ofpersistent data structures, Caladan.3.9 Limitations and Future WorkDue to the limited filtering support of the 82599 NIC,our implementation uses a different MAC address foreach VNIC, which we use to direct flows to applicationsand then do more fine-grain filtering in software, withinapplications. The availability of more general-purposefilters would eliminate this software overhead.Our implementation of the virtual function driverdoes not currently support the “transmit descriptor headwriteback” feature of the 82599, which reduces thenumber of PCI bus transactions necessary for transmitoperations. We expect to see a 5% network performanceimprovement from adding this support.The RS3 RAID controller we media player classic always on top Crack Key For U in our experimentsdoes not support SR-IOV or VSAs. Hence, we use itsphysical function, which provides one hardware queue,and we map a VSA to each logical disk provided by thecontroller. We still use the IOMMU for protected accessto application virtual memory, but the controller does notprotect access to logical disks based on capabilities. Ourexperience with the 82599 suggests that hardware I/Ovirtualization incurs negligible performance overheadversus direct access to the physical function, Format Factory 4.6.2.0 Keygen Crack Key For U. We expectthis to be similar for storage controllers.4 EvaluationWe evaluate Arrakis on four cloud application Format Factory 4.6.2.0 Keygen Crack Key For U typical, read-heavy load pattern observed in many largedeployments of the memcached distributed object cachingsystem, a write-heavy load pattern to the Redis persistentNoSQL store, a workload consisting of a large numberof individual client HTTP requests made Format Factory 4.6.2.0 Keygen Crack Key For U a farm ofweb servers via an HTTP load balancer and, finally, thesame benchmark via an IP-layer middlebox. We alsoexamine the system under maximum load in a series ofmicrobenchmarks and Format Factory 4.6.2.0 Keygen Crack Key For U performance crosstalkamong multiple networked applications. Using theseexperiments, we seek to answer the following questions:9USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 9

• What are the major contributors to performanceoverhead in Arrakis and how do they compare to thoseof Linux (presented in §2)?• Does Arrakis provide better latency and throughputfor real-world cloud applications? How does thethroughput scale with the number of CPU cores forthese workloads?• Can Arrakis retain the benefits of user-level applicationexecution and kernel enforcement, while providinghigh-performance packet-level network IO?• What additional performance gains are possible bydeparting from the POSIX interface?We compare the performance of the following OSconfigurations: Linux kernel version 3.8 (Ubuntu version13.04), Arrakis using the POSIX interface (Arrakis/P),and Arrakis using its native interface (Arrakis/N).We tuned Linux network performance by installing thelatest ixgbe device driver version 3.17.3 and disablingreceive side scaling (RSS) when applications execute ononly one processor. RSS spreads packets over several NICreceive queues, but incurs needless coherence overhead ona single core. The changes yield a throughput improvementof 10% over non-tuned Linux. We use the kernel-shippedMegaRAID driver version 6.600.18.00-rc1.Linux uses a number of performance-enhancingfeatures of the network hardware, which Arrakis doesnot currently support. Among these features is the useof direct processor cache access by the NIC, TCP andUDP segmentation offload, large receive offload, andnetwork packet header splitting. All of these featurescan be implemented in Arrakis; thus, our performancecomparison is weighted in favor of Linux.4.1 Server-side Packet Processing PerformanceWe load the UDP echo benchmark from §2 on the serverand use all Format Factory 4.6.2.0 Keygen Crack Key For U machines in the cluster as load generators.These generate 1 KB UDP packets at a fixed rate and recordthe rate at which their echoes arrive. Each experimentexposes the server to maximum load for 20 seconds.Shown in Table 1, compared to Linux, Arrakis eliminatestwo system calls, software demultiplexing overhead,socket buffer locks, and security checks. In Arrakis/N, weadditionally eliminate two socket buffer copies. Arrakis/Pincurs a total server-side overhead of 1.44 µs, 57% lessthan Linux. Arrakis/N reduces this overhead to 0.38 µs.The echo server is able to add a configurable delaybefore sending back each packet. We use this delay tosimulate additional application-level processing time atthe server. Figure 4 shows the average throughput attainedby each system over various such delays; the theoreticalline rate is 1.26M pps with zero processing.In the best case (no additional processing time),Arrakis/P achieves 2.3× the throughput of Linux. ByThroughput [k packets / s]120010008006004002000LinuxArrakis/PArrakis/NDriver0 1 2 4 8 16 32 64Processing time [us]Figure 4: Average UDP echo throughput for packets with 1024byte payload over various processing times. The top y-axis valueshows theoretical maximum throughput on the 10G network.Error bars in this and following figures show min/max measuredover 5 repeats of the experiment.departing from POSIX, Arrakis/N achieves 3.9× thethroughput of Linux. The relative benefit of Arrakisdisappears at 64 µs. To gauge how close Arrakis comesto the maximum possible throughput, we embedded aminimal echo server directly into the NIC device driver,eliminating any remaining API overhead. Arrakis/Nachieves 94% of the driver limit.4.2 Memcached Key-Value StoreMemcached is an in-memory key-value store used bymany cloud applications. It incurs a processing overheadof 2–3 µs for an average object fetch request, comparableto the overhead of OS kernel network processing.We benchmark memcached 1.4.15 by sending itrequests at a constant rate via its binary UDP protocol,using a tool similar to the popular memslap benchmark [2].We configure a workload pattern of 90% fetch and 10%store requests on a pre-generated range of 128 differentkeys of a fixed size of 64 bytes and a value size of 1 KB,in line with real cloud deployments [7].To measure network stack scalability for multiple cores,we vary the number of memcached server processes. Eachserver process executes independently on its own portnumber, such that measurements are not impacted by scalabilitybottlenecks in memcached itself, and we distributeload equally among the available memcached instances.On Linux, memcached processes share the kernel-levelnetwork stack. On Arrakis, each process obtains its ownVNIC with an independent set of packet queues, eachcontrolled by an independent instance of Extaris.Figure 5 shows that memcached on Arrakis/P achieves1.7× the throughput of Linux on one core, and attains nearline-rate at 4 CPU cores. The slightly lower throughputon all 6 cores is due to contention with Barrelfishsystem management processes [10]. By contrast, Linuxthroughput nearly plateaus beyond two cores. A single,multi-threaded memcached instance shows no noticeablethroughput difference to the multi-process scenario. Thisis not surprising as memcached is optimized to scale well.1010 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

Throughput [k transactions / s]120010008006004002000Linux threadsLinux procsArrakis/P1 2 4 6Number of CPU coresFigure 5: Average memcached transaction throughput andscalability, Format Factory 4.6.2.0 Keygen Crack Key For U. Top y-axis value = 10Gb/s.To conclude, the separation of network stack and applicationin Linux provides only limited information aboutthe application’s packet processing and poses difficulty assigningthreads to the right CPU core. The resulting cachemisses and socket lock contention are responsible for muchof the Linux overhead. In Arrakis, the application is in controlof the whole packet processing flow: assignment ofpackets to packet queues, packet queues to cores, and finallythe scheduling of its own threads on these cores. Thenetwork stack thus does not need to acquire any locks, andpacket data is always available in the right processor cache.Memcached is also an excellent example of the communicationendpoint abstraction: we can create hardwarefilters to allow packet reception and transmission onlybetween the memcached server and a designated list ofclient machines that are part of the cloud application. In theLinux case, we have to filter connections in the application.4.3 Arrakis Native Interface Case StudyAs a case study, we modified memcached to make useof Arrakis/N. In total, 74 lines of code were changed,with 11 pertaining to the receive side, and 63 to the sendside. On the receive side, the changes involve eliminatingmemcached’s receive buffer and working directly withpointers to packet buffers provided by Extaris, as wellas returning completed buffers to Extaris. The changesincrease average throughput by 9% over Arrakis/P. On thesend side, changes include allocating a number of sendbuffers to allow buffering of responses until fully sentby the NIC, which now must be done within memcacheditself. They also involve the addition of reference countsto hash table entries and send buffers to determine whenit is safe to reuse buffers and hash table entries that mightotherwise still be processed by the NIC. We gain anadditional 10% average throughput when using the sendside API in addition to the receive side API.4.4 Redis NoSQL StoreRedis [18] extends the memcached model from a cacheto a persistent NoSQL object store. Our results in Table 2show that Redis operations—while more laborious thanMemcached—are still dominated by I/O stack overheads.Throughput [k transactions / s]300250200150100500GETLinuxArrakis/PArrakis/P [15us]Linux/CaladanSETFigure 6: Average Redis transaction throughput for GET andSET operations. The Arrakis/P [15us] and Linux/Caladanconfigurations apply only to SET operations.Redis can be used in the same scenario as Memcachedand we follow an identical experiment setup, using Redisversion 2.8.5. We use the benchmarking tool distributedwith Redis and configure free home design software south africa to execute GET and SETrequests in two separate benchmarks to a range of 65,536random keys with a value size of 1,024 bytes, persistingeach SET operation individually, with a total concurrencyof 1,600 connections from 16 benchmark clients executingon the client machines. Redis is single-threaded, so weinvestigate only single-core performance.The Arrakis version of Redis uses Caladan, Format Factory 4.6.2.0 Keygen Crack Key For U. We changed109 lines in the application to manage and exchangerecords with the Caladan log instead of a file. We did noteliminate Redis’ marshaling overhead (cf. Table 2). If wedid, we would save another 2.43 µs of write latency. Dueto the fast I/O stacks, Redis’ read performance mirrors thatof Memcached and write latency improves by 63%, whilewrite throughput improves Format Factory 4.6.2.0 Keygen Crack Key For U, by 9×.To investigate what would happen if we had accessto state-of-the-art storage hardware, we simulate (via awrite-delaying RAM disk) a storage backend with 15 µswrite latency, such as the ioDrive2 [24]. Write throughputimproves by another 1.6×, nearing Linux read throughput.Both network and disk virtualization is needed for goodRedis performance. We tested this by porting Caladan torun on Linux, with the unmodified Linux network stack.This improved write throughput by only 5× compared toLinux, compared to 9× on Arrakis.Together, the combination of data-plane network andstorage stacks can yield large benefits in latency andthroughput for both read and write-heavy workloads.The tight integration of storage and data structure inCaladan allows for a number of latency-saving techniquesthat eliminate marshaling overhead, book-keeping ofjournals for file system metadata, and can offset storageallocation overhead. These benefits will increase furtherwith upcoming hardware improvements.4.5 HTTP Load BalancerTo aid scalability of web services, HTTP load balancersare often deployed to distribute client load over a number11USENIX Association 11th USENIX Symposium on Mobaxterm professional edition portable crack Free Activators Systems Design and Implementation (OSDI ’14) 11

Throughput [k transactions / s]160140120100806040200LinuxLinux (SEPOLL)Arrakis/P1 2 4 6Number of CPU coresThroughput [k transactions / s]250200150100500LinuxArrakis/P1 2 4Number of CPU coresFigure 7: Average HTTP transaction throughput and scalabilityof haproxy.of web servers. A popular HTTP load balancer employedby many web and cloud services, such as Amazon EC2 andTwitter, is haproxy [3]. In these settings, many connectionsare constantly opened and closed and the OS needs tohandle the creation and deletion of the associated socketdata structures.To investigate how performance is impacted when manyconnections need to be maintained, we configure fiveweb servers and one load balancer. To minimize overheadat the web servers, we deploy a simple static web pageof 1,024 bytes, served out of main memory.These sameweb server hosts also serve as workload generators, Format Factory 4.6.2.0 Keygen Crack Key For U, usingApacheBench version 2.3 to conduct as many concurrentrequests for the web page as possible. Each request isencapsulated in its own TCP connection. On the load balancerhost, we deploy haproxy version 1.4.24, configuredto distribute incoming load in a round-robin fashion. Werun multiple copies of the haproxy process on the loadbalancing node, each executing on their own port number.We configure the ApacheBench instances to distributetheir load equally among the available haproxy instances.Haproxy relies on cookies, which it inserts into theHTTP stream to remember connection assignments tobackend web servers under possible client re-connects.This requires it to interpret the HTTP stream for eachclient request. Linux provides an optimization called TCPsplicing that allows applications to forward traffic betweentwo sockets without user-space involvement. This reducesthe overhead of Format Factory 4.6.2.0 Keygen Crack Key For U crossings when connections arelong-lived. We enable haproxy to use this feature on Linuxwhen beneficial.Finally, haproxy contains a feature known as “speculativeepoll” (SEPOLL), which uses knowledge abouttypical socket operation flows within the Linux kernel toavoid Grammarly 1.5.49 License Key Crack Key For U to the epoll interface and optimize performance.Since the Extaris implementation differs from that of theLinux kernel network stack, we were not able to use thisinterface on Arrakis, but speculate that this feature could beported to Arrakis to yield similar performance benefits. Toshow the effect of the SEPOLL feature, we repeat the Linuxbenchmark both with and without it and show both results.Figure 8: Average HTTP transaction throughput and scalabilityof the load balancing middlebox. Top y-axis value = 10Gb/s.In Figure 7, we can see that Arrakis outperforms Linux inboth regular and SEPOLL configurations on a single core,by a factor of 2.2 and 2, respectively. Both systems showsimilar scalability curves. Note that Arrakis’s performanceon 6 CPUs is affected by background activity on Barrelfish.To conclude, Format Factory 4.6.2.0 Keygen Crack Key For U, connection oriented workloads requirea higher number of system calls for setup (accept andsetsockopt) and teardown (close). In Arrakis, wecan use filters, Format Factory 4.6.2.0 Keygen Crack Key For U, which require only one control planeinteraction to specify which clients and servers maycommunicate with the load balancer service. Furthersocket operations are reduced to function calls Format Factory 4.6.2.0 Keygen Crack Key For U thelibrary OS, with lower overhead.4.6 IP-layer MiddleboxIP-layer middleboxes are ubiquitous in today’s wide areanetworks (WANs). Common middleboxes perform tasks,such as firewalling, intrusion detection, network addresstranslation, and load balancing. Due to the complexity oftheir tasks, middleboxes can benefit from the programmingand run-time convenience provided by an OS through itsabstractions for safety and resource management.We implemented a simple user-level load balancingmiddlebox using raw IP sockets [5]. Dvdfab enlarger ai like haproxy,the middlebox balances an incoming TCP workload toa set of back-end servers. Unlike haproxy, it is operatingcompletely transparently to the higher layer protocols.It simply rewrites source and destination IP addressesand TCP port numbers contained in the packet headers.It monitors active TCP connections and uses a hash tableto remember existing connection assignments. Responsesby the back-end web servers are also intercepted andforwarded back to the corresponding clients. This issufficient to provide the same load balancing capabilitiesas in the haproxy experiment. We repeat the experimentfrom §4.5, replacing haproxy with our middlebox.The simpler nature of the middlebox is reflected in thethroughput results (see Figure 8). Both Linux and Arrakisperform better. Because the middlebox performs lessapplication-level work than haproxy, performance factorsare largely due to OS-level network packet processing.As a consequence, Arrakis’ benefits are more prominent,1212 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

Throughput [k transactions / s]1400120010008006004002000Arrakis/P Linux Arrakis/P LinuxNo limit100Mbit/s limitFigure 9: Memcached transaction throughput over 5 instances(colors), with and without rate limiting.and its performance is 2.6× that of Linux. We also seean interesting effect: the Linux implementation does notscale at all in this configuration. The reason for this arethe raw IP sockets, which carry no connection information.Without an indication of which connections to steer towhich sockets, each middlebox instance has to look at eachincoming packet to determine whether it should handle it.This added overhead outweighs any performance gainedvia parallelism. In Arrakis, we can configure the hardwarefilters to steer packets based on packet header informationand thus scale until we quickly hit the NIC throughputlimit at two cores.We conclude that Arrakis allows us to retain the safety,abstraction, and management benefits of software developmentat user-level, while vastly improving the performanceof low level packet operations. Filters provide a versatileinterface to steer packet workloads based on arbitraryinformation stored in packet headers to effectively leveragemulti-core parallelism, regardless of protocol specifics.4.7 Performance IsolationWe show that QoS limits can be enforced in Arrakis,by simulating a simple multi-tenant scenario with 5memcached instances pinned to distinct cores, to minimizeprocessor crosstalk. One tenant has an SLA that allowsit to send up to 100Mb/s. The other tenants are not limited.We use rate specifiers in Arrakis to set the transmit ratelimit of the VNIC of the limited process. On Linux, weuse queuing disciplines [29] (specifically, HTB [20]) torate limit the source port of the equivalent process.We repeat the experiment from §4.2, plotting thethroughput achieved by each memcached instance, shownin Figure 9. The bottom-most process (barely visible) israte-limited to 100Mb/s in the experiment shown on theright hand side of the figure. All runs remained withinthe error bars shown in Figure 5. When rate-limiting, abit of the total throughput is lost for both OSes becauseclients keep sending packets at the same high rate. Theseconsume network bandwidth, even when later droppeddue to the rate limit.We conclude that it is possible to provide the same kindof QoS enforcement—in this case, rate limiting—in Arrakis,as in Linux. Thus, we are able to retain the protectionand policing benefits of user-level application execution,while providing improved network performance.5 DiscussionIn this section, we discuss how we can extend the Arrakismodel to apply to virtualized guest environments, as wellas to interprocessor interrupts.5.1 Arrakis as Virtualized GuestArrakis’ model can be extended to virtualized environments.Making Arrakis a host in this environmentis straight-forward—this is what the technology wasoriginally designed for, Format Factory 4.6.2.0 Keygen Crack Key For U. The best way to support Arrakis asa guest is by moving the control plane into the virtual machinemonitor (VMM). Arrakis guest applications can thenallocate virtual interface cards directly from the VMM.A simple way of accomplishing this is by pre-allocating anumber of virtual interface cards in the VMM to the guestand let applications pick only from this pre-allocated set,without requiring a special interface to the VMM.The hardware limits apply to a virtualized environmentin the same way as they do in the regular Arrakisenvironment. We believe the current limits on virtualadapters (typically 64) to be balanced with the number ofavailable processing resources.5.2 Virtualized Interprocessor InterruptsTo date, most parallel applications are designed assumingthat shared-memory is (relatively) efficient, whileinterprocessor signaling is (relatively) inefficient. A cachemiss to data written by another core is handled in hardware,while alerting a thread on another processor requireskernel mediation on both the sending and receiving side.The kernel is involved even when signaling an eventbetween two threads running inside the same application.With kernel bypass, a remote cache miss and a remoteevent delivery are similar in cost at a physical level.Modern hardware already provides the operating systemthe ability to control how device interrupts are routed. Tosafely deliver an interrupt within an application, withoutkernel mediation, requires that the hardware add accesscontrol. With this, the kernel could configure the interruptrouting hardware to permit signaling among cores runningthe same application, trapping to the kernel only whensignaling between different applications.6 Related WorkSPIN [14] and Exokernel [25] reduced shared kernelcomponents to allow each application to have customizedoperating system management. Nemesis [15] reducesshared components to provide more performance isolationfor multimedia applications. Format Factory 4.6.2.0 Keygen Crack Key For U three mediated I/O inthe kernel. Relative to these systems, Arrakis shows that13USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 13

application customization is consistent with very highperformance.Following U-Net, a sequence of hardware standardssuch as VIA [19] and Infiniband [30] addressed thechallenge of minimizing, or eliminating entirely, operatingsystem involvement in sending and receiving networkpackets in the common case. To a large extent, thesesystems have focused on the needs of parallel applicationsfor high throughout, low overhead communication.Arrakis supports a more general networking modelincluding client-server and peer-to-peer communication.Our work was inspired in part by previous work onDune [11], which used nested paging to provide supportfor user-level control over virtual memory, and ExitlessIPIs [26], which presented a technique to demultiplexhardware interrupts between virtual machines withoutmediation from the virtual machine monitor.Netmap [49] implements high throughput networkI/O by doing DMAs directly from user space. Sends andreceives still require system calls, as the OS needs to do permissionchecks on every operation. Throughput is achievedat the expense of latency, by batching reads and writes.Similarly, IX [12] implements a custom, per-applicationnetwork stack in a protected domain accessed with batchedsystem calls. Arrakis eliminates the need for batching byhandling operations at user level in the common case.Concurrently with our work, mTCP Format Factory 4.6.2.0 Keygen Crack Key For U Intel’s DPDKinterface to implement a scalable user-level TCP [36];mTCP focuses on scalable network stack design, whileour focus is on the operating system API for general clientserverapplications, Format Factory 4.6.2.0 Keygen Crack Key For U. We expect the performance of Extarisand mTCP to be similar. OpenOnload [50] is a hybrid userandkernel-level network stack. It is completely binarycompatiblewith existing Linux applications; to supportthis, it has to keep a significant amount of socket state in thekernel and supports only a traditional socket API. Arrakis,in contrast, allows applications to access the networkhardware directly and does not impose API constraints.Recent work has focused on reducing the overheadsimposed by traditional file systems and block devicedrivers, given the availability of low latency persistentmemory. DFS [37] and PMFS [23] are file systemsdesigned for these devices. DFS relies on the flash storagelayer for functionality traditionally implemented inthe OS, such as block allocation. PMFS exploits thebyte-addressability of persistent memory, avoiding theblock layer. Both DFS and PMFS are implemented askernel-level file systems, exposing POSIX interfaces.They focus on optimizing file system and device driverdesign for specific technologies, while Arrakis investigateshow to allow applications fast, customized device access.Moneta-D [16] is a hardware and software platform forfast, user-level I/O to solid-state devices. The hardware andoperating system cooperate to track permissions on hardwareextents, while a user-space driver communicates withthe device through a virtual interface. Applications Format Factory 4.6.2.0 Keygen Crack Key For U the system through a traditional file system. Moneta-D is optimized for large files, since each open operationrequires communication with the OS to check permissions;Arrakis does not have this issue, since applications havecomplete control over their VSAs. Aerie [53] proposesan architecture in which multiple processes communicatewith a trusted user-space file system service for filemetadata and lock operations, while directly accessing thehardware for reads and data-only writes. Arrakis providesmore flexibility than Aerie, since storage solutions can beintegrated tightly with applications rather than providedin a shared service, allowing for the development ofhigher-level abstractions, such as persistent data structures.7 ConclusionIn this paper, we described and evaluated Arrakis, a newoperating system designed to remove the kernel from theI/O data path without compromising process isolation.Unlike a traditional operating system, which mediates allI/O operations to enforce process isolation and resourcelimits, Arrakis uses device hardware to deliver I/O directlyto a customized user-level library. The Arrakis kerneloperates in the control plane, configuring the hardwareto limit application misbehavior.To demonstrate the practicality of our approach, we haveimplemented Arrakis on commercially available networkand storage hardware and used it to benchmark several typicalserver workloads. We are able to show that protectionand high performance are not contradictory: end-to-endclient read and write latency to the Redis persistent NoSQLstore is 2–5× faster and write throughput 9× higher onArrakis than on a well-tuned Linux implementation.AcknowledgmentsThis work was supported by NetApp, Google, and theNational Science Foundation. We would like to thank theanonymous reviewers and our shepherd, Emmett Witchel,for their comments and feedback. We also thank OlegGodunok for implementing the IOMMU driver, AntoineKaufmann for implementing MSI-X support, and TaesooKim for implementing interrupt support into Extaris.References[1] http://www.barrelfish.org/.[2] http://www.libmemcached.org/.[3] http://haproxy.1wt.eu.[4] Scaling in the Linux networking stack. https://www.kernel.org/doc/Documentation/networking/scaling.txt.1414 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

[5] Linux IPv4 raw sockets, May 2012. http://man7.org/linux/man-pages/man7/raw.7.html.[6] D. Abramson. Intel virtualization technology for directedI/O. Intel Technology Journal, 10(3):179–192,2006.[7] B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang, andM. Paleczny. Workload analysis of a large-scalekey-value store. In SIGMETRICS, 2012.[8] G. Banga, P. Druschel, and J. C. Mogul. Resourcecontainers: A new facility for resource managementin server systems. In OSDI, 1999.[9] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris,A. Ho, R. Neugebauer, Format Factory 4.6.2.0 Keygen Crack Key For U, I. Pratt, and A. Warfield. Xenand the art of virtualization. In SOSP, 2003.[10] A. Baumann, P. Barham, P.-E. Dagand, T. Harris,R. Isaacs, S. Peter, T. Roscoe, A. Schüpbach, andA. Singhania. The multikernel: a new OS architecturefor scalable multicore systems. In SOSP, 2009.[11] A. Belay, A. Bittau, A. Mashtizadeh, D. Terei,D. Mazières, and C. Kozyrakis. Dune: Safe user-levelaccess to privileged CPU features. In OSDI, 2012.[12] A. Belay, G. Prekas, A. Klimovic, S. Grossman,C. Kozyrakis, and E. Bugnion. IX: A protecteddataplane operating system for high throughput andlow latency. In OSDI, 2014.[13] A. Bensoussan, C. Clingen, and R. Daley. TheMultics virtual memory: Concepts and design.CACM, 15:308–318, 1972.[14] B. N, Format Factory 4.6.2.0 Keygen Crack Key For U. Bershad, S. Savage, P, Format Factory 4.6.2.0 Keygen Crack Key For U. Pardyak, E. G. Sirer,M. E. Fiuczynski, D. Becker, C. Chambers, andS. Eggers. Extensibility, safety and performance inthe SPIN operating system. In SOSP, 1995.[15] R. Black, P. T. Barham, A. Donnelly, and N. Stratford.Protocol implementation in a vertically structuredoperating system. In LCN, 1997.[16] A. M. Caulfield, T. I. Mollov, L. A. Eisner, A. De,J. Coburn, and S. Swanson. Providing safe, userspace access to fast, solid state disks. ASPLOS, 2012.[17] V. Chidambaram, T. S. Pillai, A. C. Arpaci-Dusseau,and R. H. Arpaci-Dusseau. Optimistic crashconsistency. In SOSP, 2013.[18] Citrusbyte. Redis. http://redis.io/.[19] Compaq Computer Corp., Intel Corporation, and MicrosoftCorporation. Virtual Interface ArchitectureSpecification, version 1.0 edition, December 1997.[20] M. Devera. HTB Linux queuing discipline manual– User Guide, May 2002. http://luxik.cdi.cz/~devik/qos/htb/userg.pdf.[21] A. Dragojević, D. Narayanan, M. Castro, and O. Hodson.FaRM: Fast remote memory. In NSDI, 2014.[22] P. Druschel, L. Peterson, and B. Davie. Experienceswith a high-speed network adaptor: A softwareperspective. In SIGCOMM, 1994.[23] S. R. Dulloor, S. Kumar, A. Keshavamurthy, P. Lantz,D. Reddy, R. Sankaran, and J. Jackson. Systemsoftware for persistent memory. In EuroSys, 2014.[24] Fusion-IO. ioDrive2 and ioDrive2 Duo Multi LevelCell, 2014. Product Datasheet. http://www.fusionio.com/load/-media-/2rezss/docsLibrary/FIO_DS_ioDrive2.pdf.[25] G. R. Ganger, D. R. Engler, M. F. Kaashoek, H. M.Briceño, R. Hunt, and T. Pinckney. Fast and flexibleapplication-level networking on Exokernel systems.TOCS, 20(1):49–83, Feb 2002.[26] A. Gordon, N. Amit, N. Har’El, M. Ben-Yehuda,A. Landau, A. Schuster, and D. Tsafrir. Format Factory 4.6.2.0 Keygen Crack Key For U performance for I/O virtualization. InASPLOS, 2012.[27] S. Han, S. Marshall, B.-G. Chun, and S. Ratnasamy.MegaPipe: a new programming interface for scalablenetwork I/O. In OSDI, 2012.[28] T. Harter, C. Dragga, M. Vaughn, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. A file is nota file: Understanding the I/O behavior of Appledesktop applications. In SOSP, 2011.[29] B. Hubert. Linux advanced routing & trafficcontrol HOWTO. http://www.lartc.org/howto/.[30] Infiniband Trade Organization. Introductionto Infiniband for end users. https://cw.infinibandta.org/document/dl/7268,April 2010.[31] Intel Corporation. Intel 82599 10 GbE ControllerDatasheet, December 2010. Revision 2.6. http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf.[32] Intel Corporation. Intel Data Plane DevelopmentKit (Intel DPDK) Programmer’s Guide, Aug 2013.Reference Number: 326003-003.15USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 15

[33] Format Factory 4.6.2.0 Keygen Crack Key For U Corporation. Intel RAID ControllersRS3DC080 and RS3DC040, Aug 2013.Product Brief. http://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/raid-controller-rs3dc-brief.pdf.[34] Intel Corporation. Intel virtualization technology fordirected I/O architecture specification. Technical ReportOrder Number: D51397-006, Intel Corporation,Sep 2013.[35] Intel Corporation. NVM Express, revision1.1a edition, Sep 2013. http://www.nvmexpress.org/wp-content/uploads/NVM-Express-1_1a.pdf.[36] E. Jeong, S. Woo, M. Jamshed, H. J. S. Ihm, D. Han,and K. Park. mTCP: A Highly Scalable User-levelTCP Stack for Multicore Systems. In NSDI, 2014.[37] W. K. Josephson, L. A. Bongo, K. Li, and D. Flynn.DFS: A file system for virtualized flash storage.Trans. Storage, 6(3):14:1–14:25, Sep 2010.[38] P. Kutch. PCI-SIG SR-IOV primer: An introductionto SR-IOV technology. Intel application note,321211–002, Jan 2011.[39] I. M. Leslie, D. McAuley, R. Black, T. Roscoe,P. Barham, D. Evers, R. Fairbairns, and E. Hyden.The design and implementation of an operating systemto support distributed multimedia applications.IEEE J.Sel. A. Commun., 14(7):1280–1297, Sep2006.[40] LSI Corporation. LSISAS2308 PCI Expressto 8-Port 6Gb/s SAS/SATA Controller,Feb 2010. Product Brief. http://www.lsi.com/downloads/Public/SAS%20ICs/LSI_PB_SAS2308.pdf.[45] V. S. Pai, P. Druschel, and W, Format Factory 4.6.2.0 Keygen Crack Key For U. Zwanepoel. IO-Lite: Aunified I/O buffering and caching system. In OSDI,1999.[46] A. Pesterev, J. Strauss, N. Zeldovich, and R. T.Morris. Improving network connection locality onmulticore systems. In EuroSys, 2012.[47] S. Radhakrishnan, Y. Geng, V. Jeyakumar, Format Factory 4.6.2.0 Keygen Crack Key For U, A. Kabbani,G. Porter, and A. Vahdat. SENIC: ScalableNIC for end-host rate limiting. In NSDI, 2014.[48] RDMA Consortium. Architectural specificationsfor RDMA over TCP/IP. http://www.rdmaconsortium.org/.[49] L. Rizzo. Netmap: A novel framework for fastpacket I/O. In USENIX ATC, 2012.[50] SolarFlare Communications, Inc. OpenOnload.http://www.openonload.org/.[51] Solarflare Communications, Inc. SolarflareSFN5122F Dual-Port 10GbE Enterprise ServerAdapter, 2010.[52] A. Trivedi, P. Stuedi, B. Metzler, R. Pletka, B. G.Fitch, and T. R. Gross. Unified high-performanceI/O: One stack to rule them all. In HotOS, 2013.[53] H. Volos, S. Nalli, S. Panneerselvam, V. Varadarajan,P. Saxena, and M. M. Swift. Aerie: Flexiblefile-system interfaces to storage-class memory. InEuroSys, 2014.[54] T. von Eicken, A. Basu, V. Buch, and W. Vogels.U-Net: A user-level network interface for paralleland distributed computing. In SOSP, 1995.[41] LSI Corporation. LSISAS3008 PCI Expressto 8-Port 12Gb/s SAS/SATA Controller,Feb 2014. Product Brief. http://www.lsi.com/downloads/Public/SAS%20ICs/LSI_PB_SAS3008.pdf.[42] lwIP, Format Factory 4.6.2.0 Keygen Crack Key For U. http://savannah.nongnu.org/projects/lwip/.[43] I. Marinos, R. N. M. Watson, and M. Handley.Network stack specialization for performance. InSIGCOMM, 2014.[44] D. Mosberger and L. L. Peterson. Format Factory 4.6.2.0 Keygen Crack Key For U paths explicitin the Scout operating system. In OSDI, 1996.1616 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

Decoupling Cores, Kernels, and Operating SystemsGerd Zellweger, Simon Gerber, Kornilios Kourtis, Timothy RoscoeSystems Group, Department of Computer Science, ETH ZurichAbstractWe present Barrelfish/DC, an extension to the BarrelfishOS which decouples physical cores from a nativeOS kernel, and furthermore the kernel itself from the restof the OS and application state. In Barrelfish/DC, nativekernel code on any core can be quickly replaced, kernelstate moved between cores, and cores added and removedfrom the system transparently to applications and OSprocesses, which continue to execute.Barrelfish/DC is a multikernel with two novel ideas: theuse of boot drivers to abstract cores as regular devices, anda partitioned capability system for memory managementwhich externalizes core-local kernel state.We show by performance measurements of real applicationsand device drivers that the approach is practicalenough to be used for a number of purposes, such asonline kernel upgrades, and temporarily delivering hardreal-time performance by executing a process under aspecialized, single-application kernel.1 IntroductionThe hardware landscape is increasingly dynamic. Futuremachines will contain large numbers of heterogeneouscores which will be powered on and off individually inresponse to workload changes. Cores themselves willhave porous boundaries: some may be dynamically fusedor split to provide more energy-efficient computation. ExistingOS designs like Linux and Windows assume a staticnumber of homogeneous cores, with recent extensions toallow core hotplugging.We present Barrelfish/DC, an OS design based on theprinciple that all cores are fully dynamic. Barrelfish/DCis based on the Barrelfish research OS [5] and exploitsthe “multikernel” architecture to separate the OS statefor each core. We show that Barrelfish/DC can handledynamic cores more flexibly and with far less overheadthan Linux, and also that the approach brings additionalbenefits in functionality.A key challenge with dynamic cores is safely disposingof per-core OS state when removing a core from thesystem: this process takes time and can dominate the hardwarelatency of powering the core down, reducing anybenefit in energy consumption. Barrelfish/DC addressesthis challenge by externalizing all the per-core OS andapplication state of a system into objects called OSnodes,which can be executed lazily on another core. Whilethis general idea has been proposed before (notably, it isused in Chameleon [37] to clean up interrupt state), Barrelfish/DCtakes the concept much further in completelydecoupling the OSnode from the kernel, and this in turnfrom the physical core.While transparent to applications, this new designchoice implies additional benefits not seen in prior systems:Barrelfish/DC can completely replace the OS kernelcode running on any single Format Factory 4.6.2.0 Keygen Crack Key For U or subset of cores in thesystem at runtime, without disruption mocha pro 2019 free download Crack Key For U any other OSor application code, Format Factory 4.6.2.0 Keygen Crack Key For U, including that running on the core.Kernels can be upgraded or bugs fixed without downtime,or replaced temporarily, for example to enable detailedinstrumentation, to change a scheduling algorithm, or toprovide a different kind of service such as performanceisolated,hard real-time processing for a bounded period.Furthermore, per-core OS state can be moved betweenslow, low-power cores and fast, energy-hungry cores.Multiple cores’ state can be temporarily aggregated onto asingle core to further trade-off performance and power, orto dedicate an entire package to FontCreator Pro Torrent Key a single job for alimited period. Parts of Barrelfish/DC can be moved ontoand off cores optimized for particular workloads. Corescan be fused [26] transparently, and SMT threads [29, 34]or cores sharing functional units [12] can be selectivelyused for application threads or OS accelerators.Barrelfish/DC relies on several innovations which formthe main contributions of this paper. Barrelfish/DC treatsa CPU core as being a special case of a peripheral device,and introduces the concept of a boot driver, which canstart, stop, and restart a core while running elsewhere. We1USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 17

use a partitioned capability system for memory managementwhich allows us to completely externalize all OSstate for a core. This in turn permits a kernel to be essentiallystateless, and easily replaced while Barrelfish/DCcontinues to run, Format Factory 4.6.2.0 Keygen Crack Key For U. We factor the OS into per-core kernels1 and OSnodes, and a Kernel Control Block providesa kernel-readable handle on the total state of an OSnode.In the next section, we lay out the recent trends inhardware design and software requirements that motivatethe ideas in Barrelfish/DC, Format Factory 4.6.2.0 Keygen Crack Key For U. Following this, in Section 3we discuss in more detail the background to our work,and related systems and techniques. In Section 4 wepresent the design of Barrelfish/DC, in particular the keyideas mentioned above. In Section 5 we show by meansof microbenchmarks and real applications (a web serverand the PostgreSQL database) that the new functionalityof Barrelfish/DC incurs negligible overhead, as well asdemonstrating how Barrelfish/DC can provide worst-caseexecution time guarantees for applications by temporarilyisolating cores. Finally, we Format Factory 4.6.2.0 Keygen Crack Key For U Barrelfish/DC limitationsand future work in Section 6, Format Factory 4.6.2.0 Keygen Crack Key For U, and conclude inSection 7.2 Motivation and BackgroundBarrelfish/DC fully decouples cores from kernels (supervisoryprograms running in kernel mode), and moreoverboth of them from the per-core state of the OS as a wholeand its associated applications (threads, address spaces,communication channels, etc.). This goes considerablybeyond the core hotplug or dynamic core support in today’sOSes. Figure 1 shows the range of primitive kerneloperations that Barrelfish/DC supports transparently to applicationsand without downtime as the system executes:• A kernel on a core can be rebooted or replaced.• The per-core OS state can be moved between cores.• Multiple per-core OS components can be relocatedto temporarily “share” a core.In this section we argue why such functionality willbecome important in the future, based on recent trends inhardware and software.2.1 HardwareIt is by now commonplace to remark that core counts,both on a single chip and in a complete system, are increasing,with a corresponding increase in the complexityof the memory system – non-uniform memory access andmultiple levels of cache sharing. Systems software, and1 Barrelfish uses the term CPU driver to refer to the kernel-modecode running on a core. In this paper, Format Factory 4.6.2.0 Keygen Crack Key For U, we use the term “kernel” instead,to avoid confusion with boot driver.in particular the OS, must tackle the complex problem ofscheduling both OS tasks and those of applications acrossa number of processors based on memory locality.At the same time, cores themselves are goodsync discount code nonuniform:Asymmetric multicore processors (AMP) [31]mix cores of different microarchitectures (and thereforeperformance and energy characteristics) on a single processor.A key motivation for this is power reduction forembedded systems like smartphones: under high CPUload, complex, high-performance cores can completetasks more quickly, resulting in power reduction in otherareas of the system. Under light CPU load, however, it ismore efficient to run tasks on simple, low-power cores.While migration between cores can be transparent tothe OS (as is possible with, e.g., ARM’s “big.LITTLE”AMP architecture) a better solution is for the OS to managea heterogeneous collection of cores itself, poweringindividual cores on and off reactively.Alternatively, Intel’s Turbo Boost feature, which increasesthe frequency and voltage of a core when otherson the same die are sufficiently idle to keep the chipwithin its thermal envelope, is arguably a dynamic formof AMP [15].At the same time, hotplug of processors, once theprovince of specialized machines like the Tandem Non-Stop systems [6], is becoming more mainstream. Moreradical proposals for reconfiguring physical processorsinclude Core Fusion [26], whereby multiple independentcores can be morphed into a larger CPU, pooling cachesand functional units to improve the performance of sequentialprograms.Ultimately, the age of “dark silicon” [21] may welllead to increased core counts, but with a hard limit on thenumber that may be powered on at any given time. Performanceadvances and energy savings subsequently willhave to derive from specialized hardware for particularworkloads or operations [47].The implications for a future OS are that it must managea dynamic set of physical cores, and be able to adjustto changes in the number, configuration, and microarchitectureof cores available at runtime, while maintaining astable execution environment for applications.2.2 SoftwareAlongside hardware trends, there is increasing interest inmodifying, upgrading, patching, or replacing OS kernelsat runtime. Baumann et al. [9] implement dynamic kernelupdates in K42, leveraging the object-oriented designof the OS, and later extend this to interface changes usingobject adapters and lazy update [7]. More recently,Ksplice [3] allows binary patching of Linux kernels withoutreboot, and works by comparing generated object codeand replacing entire functions. Dynamic instrumentation218 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

OSnode βOSnode αk. B3k. Ckernel Akernel B1kernel B2multiplexerkernel B2core 0core 1core 2core 1update move park unparkTimeFigure 1: Shows the supported operations of a decoupled OS. Update: The entire kernel, dispatching OSnode α, is replacedat runtime. Move: OSnode α containing all per-core state, entailing applications is migrated to another core and kernel. Park:OSnode α is moved to a new core and kernel that temporarily dispatches two OSnodes. Unpark: OSnode α is transferred back toits previous core.systems like Dtrace [13] provide mechanisms that modifythe kernel at run-time to analyze program behavior.All these systems show that the key challenges in updatingan OS online are to maintain critical invariantsacross the update and to do so with minimal interruptionof service (the system should pause, if at all, for a minimalperiod). This is particularly hard in a multiprocessorkernel with shared state.In this paper, we argue for addressing all these challengesin a single framework for core and kernel managementin the OS, although the structure of Unix-likeoperating systems presents a barrier to such a unifiedframework. The rest of this paper describes the unifiedapproach we adopted in Barrelfish/DC.3 Related workOur work combines several directions in OS design andimplementation: core hotplugging, kernel update andreplacement, and multikernel architectures.3.1 CPU HotplugMost modern OS designs today Format Factory 4.6.2.0 Keygen Crack Key For U some form of corehotplug. Since the overriding motivation is reliability, unpluggingor plugging a core is considered a rare eventand the OS optimizes the common case where the coresare not being hotplugged. For example, Linux CPU hotpluguses the __stop_machine() kernel call, which haltsapplication execution on all online CPUs for typicallyhundreds of milliseconds [23], overhead that increasesfurther when the system is under CPU load [25]. We showfurther evidence of this cost in Section 5.1 where we compareLinux’ CPU hotplug with Barrelfish/DC’ core updateoperations.Recognizing that processors will be configured muchmore frequently in the future for reasons of energy usageand performance optimization, Chameleon [37] identifiesseveral bottlenecks in the existing Linux implementationdue to global locks, and argues that current OSes are illequipped for processor sets that can be reconfigured atruntime. Chameleon extends Linux to provide support forchanging the set of processors efficiently at runtime, anda scheduling framework for exploiting this new functionality.Chameleon can perform processor reconfigurationup to 100,000 times faster than Linux 2.6.Barrelfish/DC is inspired in part by this work, butadopts a very different approach. Where Chameleon targetsa single, monolithic shared kernel, Barrelfish/DCadopts a multikernel model and uses the ability to rebootindividual kernels one by one to support CPU reconfiguration.The abstractions provided are accordingly different:Chameleon abstracts hardware processors behind processorproxies and execution objects, in part to handle theproblem of per-core state (primarily interrupt handlers)on an offline or de-configured processor. In contrast, Barrelfish/DCabstracts the per-core state (typically muchlarger in a shared-nothing multikernel than in a sharedmemorymonolithic kernel) behind OSnode and kernelcontrol block abstractions.3USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 19

In a very different approach, Kozuch et al. [30] showhow commodity OS hibernation and hotplug facilities canbe used to migrate a complete OS between different machines(with different hardware configurations) withoutvirtualization.Hypervisors are typically capable of simulating hotpluggingof CPUs within a virtual machine. Barrelfish/DCcan be deployed as a guest OS to manage a variable setof virtual CPUs allocated by the Format Factory 4.6.2.0 Keygen Crack Key For U. Indeed, Barrelfish/DCaddresses a long-standing issue in virtualization:it is hard to fully virtualize the microarchitecture of aprocessor when VMs might migrate between asymmetriccores or between physical machines with different processors.As a guest, Barrelfish/DC can natively handle suchheterogeneity and change without disrupting operation.3.2 Kernel updatesThe problem of patching system software without downtimeof critical services has been a research area for sometime. For example, K42 explored update of a running kernel[7, 9], exploiting the system’s heavily object-orienteddesign. Most modern mainstream OSes support dynamicloading and unloading of kernel modules, which can beused to update or specialize limited parts of the OS.KSplice [3] patches running Linux kernels withoutthe need for reboot by replacing code in the MediaHuman YouTube Downloader Keygen ata granularity of complete functions. It uses the Linuxstop_machine() call to ensure that no CPU is currentlyexecuting a function to be replaced, and places a branchinstruction at the start of the obsolete function to directexecution of the replacement code, Format Factory 4.6.2.0 Keygen Crack Key For U. Systems like KSplicereplace individual functions across all cores at the sametime. In contrast, Barrelfish/DC replaces entire kernels,but on a subset of cores at a time. KSplice makes sense foran OS where all cores must execute in the same, sharedmemorykernel and the overhead incurred by quiescingthe entire machine is unavoidable.Proteos [22] uses a similar approach to Barrelfish/DCby replacing applications in their entirety instead of applyingpatches to existing code. In contrast to Ksplice, Proteosautomatically applies state updates while preservingpointer integrity in many cases, which eases the burden onprogrammers to write complicated state transformationfunctions. In contrast to Barrelfish/DC, Proteos does notupgrade kernel-mode code but focuses on updates for OSprocesses running in user-space, in a micro-kernel environment.Much of the OS functionality in Barrelfish/DCresides in user-space as well, and Proteos would be applicablehere.Otherworld [18] also enables kernel updates withoutdisrupting applications, with a focus on recovering systemcrashes. Otherworld can microreboot the system kernelafter a critical error without clobbering running applications’state, and then attempt to restore applications thatwere running at the time of a crash by recreating applicationmemory spaces, open files and other resources.Rather than relying on a single, system-wide kernel,Barrelfish/DC exploits the multikernel environment to offerboth greater flexibility and better performance: kernelsand cores can be updated dynamically with (as we showin Section 5) negligible disruption to the rest of the OS.While their goals of security and availability differsomewhat from Barrelfish/DC, KeyKOS [24] andEROS [42] use partitioned capabilities to provide an essentiallystateless kernel. Memory in KeyKOS is persistent,and it allows updates of the OS while running,achieving continuity by restoring from disk-based checkpointsof the entire capability state. Barrelfish/DC bycontrast achieves continuity by distributing the capabilitysystem, only restarting some of the kernels at a time, andpreserving each kernel’s portion of the capability systemacross the restart.3.3 MultikernelsMultikernels such as fos [48], Akaros [40], Tessellation[33], Hive [14], and Barrelfish [8], are based onthe observation that modern hardware is a networkedsystem and so it is advantageous to model the OS as adistributed system. For example, Barrelfish runs a smallkernel on each core in the system, and the OS is built asa set of cooperating processes, each running on one ofthese kernels, sharing no memory, and communicating viamessage passing. Multikernels are motivated by both thescalability advantages of sharing no cache lines betweencores, and the goal of supporting future hardware withheterogeneous processors and little or no cache-coherentor shared physical memory.Barrelfish/DC exploits the multikernel design for a newreason: dynamic and flexible management of the coresand the kernels of the system. A multikernel can naturallyrun different versions of kernels on different cores. Theseversions can be tailored to the hardware, or specializedfor different workloads.Furthermore, since (unlike in monolithic kernels) thestate on each core CyberLink AudioDirector Ultra Crack 11.0.2304.0 & Serial Key [Latest] relatively decoupled from the restof the system, multikernels are a good match for systemswhere cores come and go, and intuitively should supportreconfiguration of part of the hardware without unduedisruption to software running elsewhere on the machine.Finally, the shared-nothing multikernel architecture allowsus to wrap kernel state and move it between differentkernels without worrying about potentially harmful concurrentaccesses.We chose to base Barrelfish/DC on Barrelfish, as it isreadily available, is under active development, supportsmultiple hardware platforms, and can run a variety of420 11th USENIX Symposium on Operating Systems Design and Format Factory 4.6.2.0 Keygen Crack Key For U (OSDI ’14) USENIX Association

common applications such as databases and web servers.The features of Barrelfish/DC magix movie edit pro 2019 premium crack Activators Patch in this paper willbe incorporated into a future Barrelfish release.Recently, multikernels have been combined with traditionalOS designs such as Linux [27, 36] so as to runmultiple Linux kernels on different cores of the same machineusing different partitions of physical memory, inorder to provide performance isolation between applications.Popcorn Linux [38, 43] boots a modified Linuxkernel in this fashion, and supports kernel- and user-spacecommunication channels between kernels [41], and processmigration between kernels. In principle, Popcornextended with the ideas in Barrelfish/DC could be combinedwith Chameleon in a two-level approach to dynamicprocessor support.4 DesignWe now describe how Barrelfish/DC decouples cores, kernels,and the rest of the OS. We focus entirely on mechanismin this paper, and so do not address scheduling andpolicies for kernel replacement, core power management,or application migration. Note also that our main motivationin Barrelfish/DC is adapting the OS for performanceand flexibility, and so we do not consider fault toleranceand isolation for now.We first describe how Barrelfish/DC boots a new core,and then present in stages the problem of per-core statewhen removing a core, introducing the Barrelfish/DC capabilitysystem and kernel control block. We then discussthe challenges of time and interrupts, and finish with adiscussion of the wider implications of the design.4.1 Booting a new coreCurrent CPU hotplug approaches assume a single, sharedkernel and a homogeneous (albeit NUMA) machine, witha variable number of active cores up to a fixed limit, andso a static in-kernel table of cores (whether active or inactive)suffices to represent the current hardware state.Bringing a core online is a question of turning it on, updatingthis table, and creating per-core state when needed.Previous versions of Barrelfish also adopted this approach,and booted all cores during system initialization, thoughthere has been experimental work on dynamic booting ofheterogeneous cores [35].Barrelfish/DC targets a broader hardware landscape,with complex machines comprising potentially heterogeneouscores. Furthermore, since Barrelfish/DC runsa different kernel instance on each core, there is no reasonwhy the same kernel code should run everywhere –indeed, we show one advantage of not doing this in Section5.3. We thus need an OS representation of a coreon the machine which abstracts the hardware-dependentmechanisms for bringing that core up (with some kernel)and down.Therefore, Barrelfish/DC introduces the concept of aboot driver, which is a piece of code running on a “homecore” which manages a “target core” and encapsulatesthe hardware functionality to boot, suspend, resume, andpower-down the latter. Currently boot drivers run as processes,but closely resemble device drivers and couldequally run as software objects within another process.A new core is brought online as follows:1. The new core is detected by some platform-specificmechanism (e.g., ACPI) and its appearance registeredwith the device management subsystem.2. Barrelfish/DC selects and starts an appropriate bootdriver for the new core.3. Barrelfish/DC selects a kernel binary and argumentsfor the new core, and directs the boot driver to bootthe kernel on the core.4. The boot driver loads and relocates the kernel, andexecutes the hardware protocol Format Factory 4.6.2.0 Keygen Crack Key For U start the new core.5. The new kernel initializes and uses existing Barrelfishprotocols for integrating into the running OS.The boot driver abstraction treats CPU cores much hitmanpro 3.8.0 build 292 crack Crack Key For U devices, and allows us to reuse the OS’s existingdevice and hotplug management infrastructure Format Factory 4.6.2.0 Keygen Crack Key For U handle new cores and select drivers and kernels forthem. It also separates the hardware-specific mechanismfor booting a core from the policy question of what kernelbinary to boot the core with.Boot drivers remove most of the core boot process fromthe kernel: in Barrelfish/DC we have entirely replaced theexisting multiprocessor booting code for multiple architectures(which was spread throughout the system) withboot drivers, resulting in a much simpler system structure,and reduced code in the kernels themselves.Booting a core (and, indeed, shutting it down) in Barrelfish/DConly involves two processes: the boot driveron the home core, Format Factory 4.6.2.0 Keygen Crack Key For U, and the kernel on the target core. Forthis reason, we require no global locks or other synchronizationin the system, and the performance of these operationsis not impacted by load on other cores. We demonstratethese benefits experimentally in Section 5.1.Since a boot driver for a core requires (as with a devicedriver) at least one existing core to execute, Format Factory 4.6.2.0 Keygen Crack Key For U, there is apotential dependency problem as cores come and go. Forthe PC platform we focus on here, this is straightforwardsince any core can run a boot driver for any other core,but we note that in general the problem is the same as thatof allocating device drivers to cores.Boot drivers provide a convenient abstraction of hardwareand are also used to shutdown cores, but this is notthe main challenge in removing a core from the system.5USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 21

4.2 Per-core stateTaking a core out of service in a modern OS is a moreinvolved process than booting it, since modern multicoreOSes include varying amounts of per-core kernel state. Ifthey did not, removing a core would be simply requiremigrating any running thread somewhere else, updatingthe scheduler, and halting the core.The challenge is best understood by drawing a distinctionbetween the global state in an OS kernel (i.e., thestate which is shared between all running cores in thesystem) and the per-core state, which is only accessed bya single core. The kernel state of any OS is composed ofthese two categories.In, for example, older versions of Unix, all kernel statewas global and protected by locks. In practice, however, amodern OS Filmora 8.5.0 Serial Key per-core state for scalability of scheduling,memory allocation, virtual memory, etc. Per-coredata structures reduce write sharing of cache lines, whichin turn reduces interconnect traffic and cache miss ratedue to coherency misses.For example, Linux and Windows use per-core schedulingqueues, and distributed memory allocators. Corey [10]allowed configurable sharing of page tables betweencores, and many Linux scaling enhancements (e.g., [11])have been of this form. K42 [2] adopted reduced sharingas a central design principle, and introduced the abstractionof clustered objects, essentially global proxies forpervasive per-core state.Multikernels like Barrelfish [8] push this idea to itslogical conclusion, sharing no data (other than messagechannels) between cores. Multikernels are an extremepoint in the design space, but are useful for precisely thisreason: they highlight the problem of consistent per-corestate in modern hardware. As core counts increase, wecan expect the percentage of OS state that is distributedin more conventional OSes to increase.Shutting down a core therefore entails disposing of thisstate without losing information or violating system-wideconsistency invariants. This may impose significant overhead.For example, Chameleon [37] devotes considerableeffort to ensuring that per-core interrupt handling stateis consistent across CPU reconfiguration. As more statebecomes distributed, Format Factory 4.6.2.0 Keygen Crack Key For U, this overhead will increase.Worse, how to dispose of this state depends on what itis: removing a per-core scheduling queue means migratingthreads to other cores, whereas removing a per-corememory allocator requires merging its memory pool withanother allocator elsewhere.Rather than implementing a succession of piecemealsolutions to this problem, in Barrelfish/DC we adopt aradical approach of lifting all the per-core OS state outof the kernel, so that it can be reclaimed lazily withoutdelaying the rest of the OS. This design provides theScheduler StateCap Derivation TreeTimer Offset (§4.7)IRQ State (§4.8)KCB (§4.4).FrameFrameFrameFrameCNodePCB§4.2)CNodePCB§4.3).FrameFrameFrameCNodeFigure 2: State in the Barrelfish/DC OSnode.NullFrameFrameFramemeans to completely decouple per-core state from both theunderlying kernel implementation and the core hardware.We find it helpful to use the term OSnode to denotethe total state of an OS kernel local to a particular core.In Linux the OSnode changes with different versions ofthe kernel; Chameleon identifies this state by manualannotation of the kernel source code. In Barrelfish, theOSnode is all the state – there is no shared global data.4.3 Capabilities in Barrelfish/DCBarrelfish/DC captures the OSnode using its capabilitysystem: all memory and other resources maintained by thecore (including interrupts and communication end-points)are represented by capabilities, and thus the OSnode isrepresented by the capability set of the core. The per-corestate of Barrelfish/DC is shown schematically in Figure 2.Barrelfish/DC’s capability system, an extension of thatin Barrelfish [44], is derived from the partitioned capabilityscheme used in seL4 [19, 20, 28].In seL4 (and Barrelfish), all regions of memory arereferred to by capabilities, and capabilities are typed to reflectwhat the memory is used for. For example, a “frame”capability refers to memory that the holder can map intotheir address space, while a “c-node” capability refers tomemory that is used to store the bit representations ofcapabilities themselves. The security of the system asa whole derives from the fact that only a small, trustedcomputing base (the kernel) holds both a frame capabilityand a c-node capability to the same memory, and cantherefore fabricate capabilities.A capability for a region can be split into two Format Factory 4.6.2.0 Keygen Crack Key For U, and also retyped according to a set of systemrules that preserve integrity. Initially, memory regionsare of type “untyped”, and must be explicitly retyped to622 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

“frame”, “c-node”, or some other type.This approach has the useful property that a process canallocate memory without being able to access its movavi photo manager activation key is used in seL4 to remove any dynamic memory allocationfrom the kernel, greatly simplifying both the formalspecification of the kernel Format Factory 4.6.2.0 Keygen Crack Key For U its subsequent proof [20].All kernel objects (such as process control blocks, or pagetables) are allocated by user-level processes which can,themselves, not access them directly.A key insight of Barrelfish/DC is that this approach canexternalize the kernel state entirely, as follows.4.4 Kernel Control BlocksIn developing Barrelfish/DC, Format Factory 4.6.2.0 Keygen Crack Key For U, we examined the Barrelfishkernel to identify all the data structures which were notdirect (optimized) derivations of information already heldin the capability tree (and which could therefore be reconstructeddynamically from the tree). We then eliminatedfrom this set any state that did not need to persist across akernel restart.For example, the runnable state and other schedulingparameters of a process 2 are held in the process’ controlblock, which is part of the capability system. However,the scheduler queues themselves do not need to persistacross a change of kernel, since (a) any scheduler willneed to recalculate them based on the current time, and(b) the new scheduler may have a completely differentpolicy and associated data structures anyway.What remained was remarkably small: it consists of:• The minimal scheduling state: the head of a linkedlist of a list of process control blocks.• Interrupt state. We discuss interrupts in Section 4.8.• The root of the capability derivation tree, from whichall the per-core capabilities can be reached.• The timer offset, discussed in Section 4.7.In Barrelfish/DC, we introduce a new memory object,the Kernel Control Block (KCB), and associated capabilitytype, holding this data in a standard format. The KCB issmall: for 64-bit x86 it is about 28 KiB in size, almostall of which is used by communication endpoints forinterrupts.4.5 Replacing a kernelThe KCB effectively decouples the per-core OS state fromthe kernel. This allows Barrelfish/DC to shut down a kernelon a core (under the control of the boot driver Format Factory 4.6.2.0 Keygen Crack Key For U another core) and replace it with a new one. The currentlyrunning kernel saves a small amount of persistent2 Technically, it is a Barrelfish “dispatcher”, the core-local representationof a process. A process usually consists of a set of distinct“dispatchers”, one in free pro Activators Patch OSnode.state in the KCB, and halts the core. The boot driverthen loads a new kernel with an argument supplying theaddress of the KCB. It then restarts the core (using an IPIon x86 machines), causing the new kernel to boot. Thisnew kernel then initializes any internal data structures itneeds from the KCB and the OSnode capability database.The described free home design software south africa allows for arbitrary updates ofkernel-mode code. By design, the kernel does not accessstate in the OSnode concurrently. Therefore, having a quiescentstate in the OSnode before we shut-down a core isalways guaranteed. The simplest case for updates requiresno changes in any data structures reachable by the KCBand can be performed as described by simply replacingthe kernel code. Updates that require a transformationof the data structures may require a one-time adaptionfunction to execute during initialization, whose overheaddepends on the complexity of the function and the size ofthe OSnode. The worst-case scenario is one that requiresadditional memory, since the kernel by design delegatesdynamic memory allocation to userspace.As we show in Section 5, replacing a kernel can bedone with little performance impact on processes runningon the core, even device drivers.4.6 Kernel sharing and core shutdownAs we mentioned above, taking a core completely outof service involves not simply shutting down the kernel,but also disposing of or migrating all the per-core stateon the core, and this can take time. Like Chameleon,Barrelfish/DC addresses this problem by deferring it: weimmediately take the core down, but keep the OSnoderunning in order to be able to dismantle it lazily. Tofacilitate this, Format Factory 4.6.2.0 Keygen Crack Key For U created a new kernel which is capableof multiplexing several KCBs (using a simple extensionto the existing scheduler).Performance of two active OSnodes sharing a core isstrictly best-effort, and is not intended to be used for anycase where application performance matters. Rather, itprovides a way for an OSnode to be taken out of servicein the background, Format Factory 4.6.2.0 Keygen Crack Key For U, after the core has been shut down.Note that there is no need for all cores in Barrelfish/DCto run this multiplexing kernel, or, indeed, for any coresto run it when it is not being used – it can simply replacean existing kernel on demand. In practice, we find thatthere is no performance loss when running a single KCBabove a multiplexing kernel.Decoupling kernel state allows attaching and detachingKCBs from a running kernel. The entry point for kernelcode takes a KCB as an argument. When a new kernel isstarted, a fresh KCB is provided to the kernel code. Torestart a kernel, the KCB is detached from the running kernelcode, the core is shut down, and the KCB is providedto the newly booted kernel code.7USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) Format Factory 4.6.2.0 Keygen Crack Key For U rely on shared physical memory when movingOSnodes between cores. This goes against the originalmultikernel premise that assumes no shared memory betweencores. However, an OSnode is still always in useby strictly one core at the time. Therefore, the benefitsof avoiding concurrent access in OSnode state remain.We discuss support for distributed memory hardware inSection 6.The combination of state externalization via the KCBand kernel sharing on a single core has a number of furtherapplications, which we describe in Section 4.10.4.7 Dealing with timeOne of the complicating factors in starting the OSnodewith a new kernel is the passage of time. Each kernelmaintains a per-core internal clock (based on a freerunningtimer, such as the local APIC), and expects thisto increase monotonically. The clock is used for percorescheduling and other time-sensitive tasks, and is alsoavailable to application threads running on the core via asystem call.Unfortunately, the hardware timers used are rarely synchronizedbetween cores. Some hardware (for example,modern PCs) define these timers to run at the same rateon every core (regardless of power management), but theymay still be offset from each other. On other hardwareplatforms, these clocks may simply run at different ratesbetween cores.In Barrelfish/DC we address this problem with twofields in the KCB. The first holds a constant offset fromthe local hardware clock; the OS applies this offset wheneverthe current time value is read.The second field is set to the current local time whenthe kernel is shut down. When a new kernel starts withan existing KCB, the offset field is reinitialized to thedifference between this old time value and the currenthardware clock, ensuring that local time for the OSnodeproceeds monotonically.4.8 Dealing with interruptsInterrupts pose an additional challenge when moving anOSnode between cores. It is important that interrupts fromhardware devices are always routed to the correct kernel.In Barrelfish interrupts are then mapped to messages deliveredto processes running on the target core. Someinterrupts (such as those from network cards) should “follow”the OSnode to its new core, whereas others shouldnot. We identify three categories of interrupt.1. Interrupts which are used exclusively by the kernel,for example a local timer interrupt used to implementpreemptive scheduling. Handling these interruptsis internal to the kernel, and their sources aretypically per-core hardware devices like APICs orperformance counters. In this case, there is no needto take additional actions when reassigning KCBsbetween cores.2. Inter-processor interrupts (IPIs), typically used forasynchronous communication between cores. Barrelfish/DCuses an indirection table that mapsOSnode identifiers to the physical core running thecorresponding kernel. When one kernel sends anIPI to another, it uses this table to obtain the hardwaredestination address for the interrupt. Whendetaching a KCB from a core, its entry is updatedto indicate that its kernel is unavailable. Similarly,attaching a KCB to a core, updates the location tothe new core identifier.3. Device interrupts, which should be forwarded to aspecific core (e.g. via IOAPICs and PCIe bridges)running the handler for the device’s driver.When Barrelfish/DC device drivers start up they requestforwarding of device interrupts by providing twocapability arguments to their local kernel: an opaque interruptdescriptor (which conveys authorization to receivethe interrupt) and a message binding. The interrupt descriptorcontains all the architecture-specific informationabout the interrupt source needed to route the interrupt tothe right core. The kernel associates the message bindingwith the architectural interrupt and subsequently forwardsinterrupts to the message channel.For the device and the driver to continue normal operation,the interrupt needs to be re-routed to the newcore, and a new mapping is set up for the (existing) driverprocess. This could be done either transparently by thekernel, or explicitly by the device driver.We choose the latter approach to simplify the kernel.When a Barrelfish/DC kernel shuts down, it disables allinterrupts. When a new kernel subsequently resumes anOSnode, it sends a message (via a scheduler upcall) toevery process which had an interrupt registered. Eachdriver process responds to this message by re-registeringits interrupt, and then checking with the device directlyto see if any events have been missed in the meantime(ensuring any race Format Factory 4.6.2.0 Keygen Crack Key For U is benign). In Section 5.2.1we show the overhead of this process.4.9 Application supportFrom the perspective of applications which are obliviousto the allocation of physical cores (and which dealsolely with threads), the additional functionality of Barrelfish/DCis completely transparent. However, manyapplications such as language runtimes and database systemsdeal directly with physical cores, and tailor theirscheduling of user-level threads accordingly.824 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

For these applications, Barrelfish/DC can use the existingscheduler activation [1] mechanism for processdispatch in Barrelfish to notify userspace of changes inthe number of online processors, much as it can alreadyconvey the allocation of physical cores to applications.4.10 DiscussionFrom a broad perspective, the combination of boot driversand replaceable kernels is a radically different view ofhow an OS should manage processors on a machine. Moderngeneral-purpose kernels such as Linux try to supporta broad set of requirements by implementing differentbehaviors based on build-time and run-time configuration.Barrelfish/DC offers an alternative: instead of buildingcomplicated kernels that try to do many things, build simplekernels that do one thing well. While Linux selectsa single kernel at boot time for all cores, Format Factory 4.6.2.0 Keygen Crack Key For U, Barrelfish/DCallows selecting not only per-core kernels, but changingthis selection on-the-fly.There are many applications for specialized kernels,including those tailored for running databases or languagerun-times, debugging or profiling, or directly executingverified user code as in Google’s native client [49].To take one example, in this paper we demonstratesupport for hard real-time applications. Despite yearsof development of real-time support features in Linuxand other general-purpose kernels [16], many users resortto specialized real-time OSes, or modified versions ofgeneral-purpose OSes [32].Barrelfish/DC can offer hard real-time support by rebootinga core with a specialized kernel, which, to eliminateOS jitter, has no scheduler (since it targets a singleapplication) and takes no interrupts. If a core is notpreallocated, it must be made available at run-time bymigrating the resident OSnode to another core that runsa multi-KCB kernel, an operation we call parking. Ifrequired, cache interference from other cores can alsobe mitigated by migrating their OSnodes to other packages.Once the hard real-time application finishes, theOSnodes can be moved back to the now-available cores.We evaluate this approach in Section 5.3.5 EvaluationWe present here a performance evaluation of Barrelfish/DC.First (Section 5.1), we measure the performanceof starting and stopping cores in Barrelfish/DCand in Linux. Second (Section 5.2), we investigate the behaviorof applications when we restart kernels, and whenwe park OSnodes. Finally, (Section 5.3), we demonstrateisolating performance via a specialized kernel. We performexperiments on the set of x86 machines shown inTable 1. Hyperthreading, TurboBoost, and SpeedSteptechnologies are disabled in machines that support them,as they complicate cycle counter measurements. Turbo-Boost and SpeedStep can change the processor frequencyin unpredictable ways, leading to high fluctuation Format Factory 4.6.2.0 Keygen Crack Key For U repeatedexperiments. The same is true for Hyperthreadingdue to sharing of hardware logic between logical cores.However, Format Factory 4.6.2.0 Keygen Crack Key For U, TurboBoost and Hyperthreading are both relevantfor this work as discussed in Section 6 and Section 1.packages×cores/uarch CPU model2×2 Santa-Rosa 2.8 GHz Opteron 22004×4 Shanghai 2.5 GHz Opteron 83802×10 SandyBridge 2.5 GHz Xeon E5-2670 v21×4 Haswell 3.4 GHz Xeon E3-1245 v3Table 1: Systems we use in our evaluation. The first columndescribes the topology of the machine (total number of packagesand cores per package) and the second the CPU model.5.1 Core management operationsIn this section, we evaluate the performance of managingcores in Barrelfish/DC, and also in Linux using the CPUHotplug facility [4]. We consider two operations: shuttingdown a core (down) and bringing it back up again (up).Bringing up a core in Linux is different from bringingup a core in Barrelfish/DC. In Barrelfish/DC, each coreexecutes a different kernel which needs to be loaded bythe boot driver, while in Linux all cores share the samecode. Furthermore, because cores share state in Linux,core management operations require global synchronization,resulting in stopping application execution in allcores for an extended period of time [23]. Stopping coresis also different between Linux and Barrelfish/DC. InLinux, applications executed in the halting core need tobe migrated to other online cores before the shutdown canproceed, while in Barrelfish/DC we typically would movea complete OSnode after the shutdown and not individualapplications.In Barrelfish/DC, the down time is the time it takes theboot driver to send an appropriate IPI to the core to behalted plus the propagation time of the IPI and the cost ofthe IPI handler in the receiving core. For the up operationwe take two measurements: the boot driver cost to preparea new kernel up until (and including) the point where itsends an IPI to the starting core (driver), and the cost inthe booted core from the point it wakes up until the kernelis fully online (core).In Linux, we measure the latency Format Factory 4.6.2.0 Keygen Crack Key For U starting or stoppinga core using the log entry of the smpboot module and asentinel line echoed to /dev/kmsg. For core shutdown,smboot reports when the core becomes offline, and weinsert the sentinel right before the operation is initiated.9USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 25

Barrelfish/DCLinuxidle load idle loaddownupupdowndriver core driver core down up down up(µs) (ms) (ms) (µs) (ms) (ms) (ms) (ms) (ms) (ms)2×2 Santa-Rosa 2.7 / — a 29 1.2 2.7 / — 34 ± 17 1.2 131 ± 25 20 ± 1 5049 ± 2052 26 ± 54×4 Shanghai 2.3 / 2.6 24 1.0 2.3 / 2.7 46 ± 76 1.0 104 ± 50 18 ± 3 3268 ± 980 18 ± 32×10 SandyBridge 3.5 / 3.7 10 0.8 3.6 / 3.7 23 ± 52 0.8 62 ± 46 21 ± 7 2265 ± 1656 23 ± 51×4 Haswell 0.8 / — a 7 0.5 0.8 / — 7 ± 0.1 0.5 46 ± 40 14 ± 1 2543 ± 1710 20 ± 5Results in cycles×10 3 ×10 6 ×10 6 ×10 3 ×10 6 ×10 6 ×10 6 ×10 6 ×10 6 ×10 62×2 Santa-Rosa 8/— 85 3.4 8/— 97 ± 49 3.5 367 ± 41 56 ± 2.0 14139 ± 5700 74 ± 214×4 Shanghai 6/6 63 2.6 6/7 115 ± 192 2.6 261 ± 127 44 ± 2.0 8170 ± 2452 46 ± 82×10 SandyBridge 9/10 27 2.1 9/10 59 ± 133 2.1 155 ± 116 53 ± 2.0 5663 ± 4141 57 ± 121×4 Haswell 3/— 26 1.9 2.9 / — 26 ± 0.40 2.0 156 ± 137 50 ± 0.5 8647 ± 5816 69 ± 16Table 2: Performance of core management operations for Barrelfish/DC and Linux (3.13) when the system is idle Format Factory 4.6.2.0 Keygen Crack Key For U when thesystem is under load. For the Barrelfish/DC down column, the value after the slash shows the cost of stopping a core on anothersocket with regard to the boot driver. a We do not include this number for Santa-Rosa because it lacks synchronized timestampcounters, nor for Haswell because it only includes a single package.For core boot, smpboot reports when the operation starts,so we insert the sentinel line right after the operation.For both Barrelfish/DC and Linux we consider twocases: an idle system (idle), and a system with all coresunder load (load). In Linux, we use the stress tool [45]to spawn a number of workers equal to the number ofcores that continuously execute the sync system call. InBarrelfish/DC, since the file-system is implemented as auser-space service, we spawn an application that continuouslyperforms memory management system calls oneach core of the system.Table 2 summarizes our results. We show both time(msecs and µsecs) and cycle counter units for convenience.All results are obtained by repeating the experiment 20times, and calculating the mean value. We include thestandard deviation where it is non-negligible.Stopping cores: The cost of stopping cores in Barrelfish/DCranges from 0.8 µs (Haswell) to 3.5 µs (Sandy-Bridge). Barrelfish/DC does not share state across cores,and as a result no synchronization between cores is neededto shut one down. Furthermore, Barrelfish/DC’ shutdownoperation consists of sending an IPI, which will cause thecore to stop after a minimal operation in the KCB (savingthe timer offset). In fact, the cost of stopping a core inBarrelfish/DC is small enough to observe the increasedcost of sending an IPI across sockets, leading to an increaseof 5% in stopping time on SandyBridge and 11%on Shanghai. These numbers are shown in Table 2, inthe Barrelfish/DC down columns after the slash. As thesemeasurements rely on timestamp counters being synchronizedacross packages, we are unable to present the costFigure 3: Breakdown of the cost of bringing up a core for theHaswell machine.increase of a cross-socket IPI on the Santa-Rosa machinewhose timestamp counters are only synchronized withina single package.In stark contrast, the cost of shutting down a core inLinux ranges from 46 ms to 131 ms. More importantly,the shutdown cost in Linux explodes when applying load,while it generally remains the same for Barrelfish/DC. Forexample, the average time to power down a core in Linuxon Haswell is increased by 55 times when we apply load.Starting cores: For Barrelfish/DC, the setup cost in theboot driver (driver) dominates the cost of starting a core(core). Fig. 3 shows a breakdown of the costs for bringingup a core on Haswell. Starting core corresponds to the1026 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

core Table 2 column, Format Factory 4.6.2.0 Keygen Crack Key For U, while the rest corresponds to operationsperformed by the boot driver: loading the imagefrom storage, allocating memory, ELF loading and relocation,etc. Loading the kernel from the file system is themost expensive operation. If multiple cores are bootedwith the same kernel, this image can be cached, significantlyimproving the time to start a core as shown in thesecond bar in Fig. 3. We note that the same costs willdominate the restart operation since shutting down a corehas negligible cost compared to bringing it up. Downtimecan be minimized by first doing the necessary preparationsin the boot driver and then halting and starting thecore.Even though Barrelfish/DC has to prepare the kernelimage, when idle, the cost of bringing up a corefor Barrelfish/DC is similar to the Linux cost (Barrelfish/DCis faster on our Intel machines, while the oppositeis true for our AMD machines). Bringing a coreup can take from 7 ms (Barrelfish/DC/Haswell) to 29 ms(Barrelfish/DC/Santa-Rosa). Load affects the cost of bootingup a core to varying degrees. In Linux such an effectis not observed in the Shanghai machine, while in theHaswell machine load increases average start time by33%. The effect of load when starting cores is generallystronger in Barrelfish/DC (e.g., in SandyBridge the costis more than doubled) because the boot driver time-sharesits core with the load generator.Overall, Barrelfish/DC has minimal overhead stoppingcores. For starting cores, results vary significantly acrossdifferent machines but the cost of bringing up cores inBarrelfish/DC is comparable to the respective Linux cost.5.2 ApplicationsIn this section, we evaluate the behavior of real applicationsunder two core management operations: restarting,where we update the core kernel as the application runs,and parking. In parking, we run the application in a corewith a normal kernel and then move its OSnode into amulti-KCB kernel running on a different core. While theapplication is parked it will share the core with anotherOSnode. We use a naive multi-KCB kernel that runs eachKCB for 20 ms, which is two times the scheduler timeslice. Finally, we move the application back to its originalcore. The application starts by running alone on its core.We execute all experiments in the Haswell machine.5.2.1 Ethernet driverOur first application is a Barrelfish NIC driver for theIntel 82574, which we modify for Barrelfish/DC to reregisterits interrupts when instructed by the kernel (seeSection 4.8). During the experiment we use ping from aclient machine to send ICMP echo requests to the NIC.We run ping as root with the -A switch, where the interpacketintervals adapt to the round-trip time. The pingmanual states: “on networks with low rtt this mode isessentially equivalent to flood mode.”Fig. 4a shows the effect of the restart operation in theround-trip time latency experienced by the client. Initially,the ping latency is 0.042 ms on average with small variation.Restarting the kernel produces two outliers (packets2307 and 2308 with an RTT of 11.1 ms and 1.07 ms, respectively).Note that 6.9 ms is the measured latency tobring up a core on this machine (Table 2).We present latency results for the parking experiment ina timeline (Fig. 4b), and in a cumulative distribution function(CDF) graph (Fig. 4c). Measurements taken whenthe driver’s OSnode runs exclusively on a core are denotedExclusive, while measurements where the OSnodeshares the core are denoted Shared, Format Factory 4.6.2.0 Keygen Crack Key For U. When parking begins,we observe an initial latency spike (from 0.042 msto 73.4 ms). The spike is caused by the parking operation,which involves sending a KCB capability reference fromthe boot driver to the multi-KCB kernel as a message. 3After the initial coordination, outliers are only caused byKCB time-sharing (maximum: 20 ms, mean: 5.57 ms).After unparking the driver, latency returns to its initial levels.Unparking does not cause the same spike as parkingbecause we do not use messages: we halt the multi-KCBkernel and directly pass the KCB reference to a newlybooted kernel.5.2.2 Web serverIn this experiment we examine how a web server 4 thatserves files over the network behaves when its core isrestarted and when its OSnode is parked. We initiate atransfer on a client machine in the server’s LAN usingwget and plot the achieved bandwidth for each 50 KiBchunk when fetching a 1 GiB file.Fig. 4d shows the results for the kernel restart experiment.The effect in this case is negligible on the clientside. We were unable to pinpoint the exact location ofthe update taking place from the data measured on theclient and the actual download times during kernel updateswere indistinguishable from a normal download. Asexpected, parking leads to a number of outliers caused byKCB time-sharing (Figures 4e and 4f). The average bandwidthbefore the parking is 113 MiB/s and the standarddeviation 9 MiB/s, whereas during parking the averagebandwidth is slightly lower at 111 MiB/s with a higherstandard deviation of 19 MiB/s.3 We follow the Barrelfish approach, where kernel messages arehandled by the monitor, a trusted OS component that runs in user-space.4 The Barrelfish native web server.11USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 27

(a) Ethernet driver restart (b) Ethernet driver parking (c) Ethernet driver parking CDF(d) Web server restart (e) Web server parking (f) Web server parking CDF(g) PostgreSQL restart (h) PostgreSQL parking (i) PostgreSQL parking CDFFigure 4: Application behavior when restarting kernels and parking OSnodes. For each application we include a timeline graph forrestarting, and a timeline and a CDF graph for parking.5.2.3 PostgreSQLNext, we run a PostgreSQL [39] database server in Barrelfish/DC,using TPC-H [46] data with a scaling factorof 0.01, stored in an in-memory file-system. We measurethe latency of a repeated CPU-bound query (query 9 inTPC-H) on a client over a LAN.Fig. 4g shows how restart affects client latency. Beforerebooting, average query latency is 36 ms. When a restartis performed, the first query has a latency of 51 ms. Aftera few perturbed queries, latency returns to its initial value.Figures 4h and 4i show the effect of parking theOSnode that contains the PostgreSQL server. As before,during normal operation the average latency is 36 ms.When the kernel is parked we observe two sets of outliers:one (with more points) with a latency of about 76 ms, andone with latency close to 56 ms. This happens, becausedepending on the latency, some queries wait for two KCBtime slices (20 ms each) of the co-hosted kernel, whileothers wait only for one.Overall, Format Factory 4.6.2.0 Keygen Crack Key For U, we argue that kernel restart incurs acceptableoverhead for online use. Parking, as expected, causes aperformance degradation, especially for latency-criticalapplications. This is, however, inherent in any form ofresource time-sharing. Furthermore, with improved KCBschedulingalgorithms the performance degradation canbe reduced or tuned (e.g., via KCB priorities).5.3 Performance isolationFinally, we illustrate mcafee total protection 2018 crack Crack Key For U benefits of Barrelfish/DC’ supportfor restarting cores with specialized kernels usingthe case of hard-real time applications where eliminatingOS jitter is required, Format Factory 4.6.2.0 Keygen Crack Key For U. To ensure that the applicationwill run uninterrupted, we assign a core with a specializedkernel that does not implement scheduling and doesnot handle interrupts (see Section 4.10), Format Factory 4.6.2.0 Keygen Crack Key For U. We evaluatethe performance isolation that can be achieved with our1228 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

under the same configuration, we attribute the outliers toOS jitter. In Barrelfish the outliers reach up to 68k cycles(excluded from the graph). Linux performs betterthan Barrelfish, but its outliers still reach 27–28 kcycles.We ascribe the worse behavior of Barrelfish compared toLinux to OS services running in user-space.We conclude that Barrelfish/DC enables the online deploymentof a dedicated, simple to build, OS kernel thateliminates OS jitter and provides hard real-time guarantees.6 Future directions(a) Histogram for all samples(b) CDF for samples in the range of 6–7k cyclesFigure 5: Number of cycles measured for 10 3 iterations of asynthetic benchmark for Barrelfish/DC, Barrelfish, and Format Factory 4.6.2.0 Keygen Crack Key For U real-time priorities.specialized kernel compared to the isolation provided by:(i) an unmodified Barrelfish kernel, and (ii) a Linux 3.13kernel where we set the application to run with real-timepriority. We run our experiments on the Format Factory 4.6.2.0 Keygen Crack Key For U machine,ensuring that no other applications run on the same core.To measure OS jitter we use a synthetic benchmarkthat only performs memory stores to a single location.Our benchmark is intentionally simple to minimize performancevariance caused by architectural effects. Wesample the timestamp counter every 10 3 iterations, for atotal of 10 6 samples. Fig. 5a shows a histogram of sampledcycles, where for all systems, most of the valuesfall into the 6-7 thousand range (i.e., 6-7 cycles latencyper iteration). Fig. 5b presents the CDF graph for the6–7 kcycles range, showing that there Format Factory 4.6.2.0 Keygen Crack Key For U no significantdifferences for the three systems in this range.Contrarily to the Barrelfish/DC dedicated kernel whereall of the samples are in the 6-7k range, in Linux and Barrelfishwe observe significant outliers that fall outside thisrange. Since we run the experiment on the same hardware,Our ongoing work on Barrelfish/DC includes both exploringthe broader applications of the ideas, and alsoremoving some of the existing limitations of the system.On current hardware, we plan to investigate the powermanagementopportunities afforded by the ability to replacecores and migrate the running OS around the hardware.One short-term opportunity is to fully exploit Intel’sTurbo Boost feature to accelerate a serial task by temporarilyvacating (and thereby cooling) neighboring cores onthe same package.We also intend to use core replacement as a meansto improve OS instrumentation and tracing facilities, bydynamically instrumenting kernels running on particularcores at runtime, removing all instrumentation overheadin the common case. Ultimately, as kernel developers wewould like to minimize whole-system reboots as much aspossible by replacing single kernels on the fly.Barrelfish/DC currently assumes cache-coherent cores,where the OS state (i.e., the OSnode) can be easily migratedbetween cores by passing physical addresses. Thelack of cache-coherency per se can be handled with suitablecache flushes, Format Factory 4.6.2.0 Keygen Crack Key For U, but on hardware platforms withoutshared memory, or with different physical address spaceson different cores, the OSnode might not require considerabletransformation to migrate between cores. TheBarrelfish/DC capability system does contain all the informationnecessary to correctly swizzle pointers whencopying the OSnode between nodes, but the copy is likelyto be expensive, and dealing with shared-memory applicationstate (which Barrelfish fully supports outside theOS) is a significant challenge.A somewhat simpler case to deal with is moving anOSnode between a virtual and physical machine, allowingthe OS to switch from running natively to running in aVM container.Note that there is no requirement for the boot driverto share memory with its target core, as long as it has amechanism for loading a kernel binary into the latter’saddress space and controlling the core itself.When replacing kernels, Barrelfish/DC assumes thatthe OSnode format (in particular, the capability system)13USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 29

emains unchanged. If the in-memory format of the capabilitydatabase changes, then the new kernel must performa one-time format conversion when it boots. It is unclearhow much of a limitation this is in practice, since the capabilitysystem of Barrelfish has changed relatively littlesince its inception, but one way to mitigate the burden ofwriting such a conversion function is to exploit the factthat the format is already specified in a domain-specifichigh-level language called Hamlet [17] to derive the conversionfunction automatically.While Barrelfish/DC decouples cores, kernels, and theOS state, the topic of appropriate policies for using thesemechanisms without user intervention is an importantarea for future work. We plan to investigate policies that,based on system feedback, create new kernels to replaceothers, and move OSnodes across cores.Finally, while Barrelfish/DC applications are notifiedwhen the core set they are running on changes (via thescheduler activations mechanism), they are currently insulatedfrom knowledge about hardware core reconfigurations.However, there is no reason why this must alwaysbe the case. There may be applications (such as databases,or language runtimes) which can benefit from being notifiedabout such changes to the running system, and wesee no reason to hide this information from applicationswhich can exploit it.7 ConclusionBarrelfish/DC presents a radically different vision of howcores are exploited by an OS and the applications runningabove it, and implements it in a viable software stack:the notion that OS state, kernel code, and execution unitsshould be decoupled and freely interchangeable. Barrelfish/DCis an OS whose design assumes that all coresare dynamic.As hardware becomes more dynamic, and scalabilityconcerns increase the need to partition or replicate stateacross cores, system software will have to change itsassumptions about the underlying platform, and adaptto a new world with constantly shifting hardware. Barrelfish/DCoffers one approach to meeting this challenge.8 AcknowledgementsWe would like to thank the anonymous reviews and ourshepherd, Geoff Voelker, for their encouragement andhelpful suggestions. We would also like to acknowledgethe work of the rest of the Barrelfish team at ETH Zurichwithout which Barrelfish/DC would not be possible.References[1] ANDERSON, T, Format Factory 4.6.2.0 Keygen Crack Key For U. E., BERSHAD, B. N., LAZOWSKA, E. D., ANDLEVY, H. M, Format Factory 4.6.2.0 Keygen Crack Key For U. Scheduler activations: Effective kernel support forthe user-level management of parallelism. ACM Transactions onComputer Systems 10, 1 (1992), 53–79.[2] APPAVOO, J., DA SILVA, D., KRIEGER, O., AUSLANDER, M.,OSTROWSKI, M., ROSENBURG, B., WATERLAND, A., WIS-NIEWSKI, R. W., XENIDIS, J., STUMM, M., AND SOARES, L.Experience distributing objects in an SMMP OS. ACM Transactionson Computer Systems 25, 3 (2007).[3] ARNOLD, J., AND KAASHOEK, M. F. Ksplice: Automatic rebootlesskernel updates. In Proceedings of the EuroSys Conference(2009), pp. 187–198.[4] ASHOK RAJ. CPU hotplug support in the Linux kernel. https://www.kernel.org/doc/Documentation/cpu-hotplug.txt.[5] The Barrelfish Operating System. http://www.barrelfish.org/, 12.04.14.[6] BARTLETT, J. F. A NonStop Kernel. In Proceedings of the8th ACM Symposium on Operating Systems Principles (1981),pp. 22–29.[7] BAUMANN, A., APPAVOO, J., WISNIEWSKI, R. W., SILVA,D. D., KRIEGER, O., AND HEISER, G. Reboots are for hardware:Challenges and solutions to updating an operating system on thefly, Format Factory 4.6.2.0 Keygen Crack Key For U. In Proceedings of the USENIX Annual Technical Conference(2007), pp. 1–14.[8] BAUMANN, A., BARHAM, P., DAGAND, P.-E., HARRIS, T.,ISAACS, R., PETER, S., ROSCOE, T.,SCHÜPBACH, A., ANDSINGHANIA, A. The multikernel: a new OS architecture forscalable multicore systems. In Proceedings of the 22nd ACMSymposium on Operating System Principles (2009), pp. 29–44.[9] BAUMANN, A., HEISER, G., APPAVOO, J., DA SILVA, D.,KRIEGER, O., WISNIEWSKI, R. W., AND KERR, J. Providingdynamic update in an operating system. In Proceedings of theUSENIX Annual Technical Conference (2005), pp, Format Factory 4.6.2.0 Keygen Crack Key For U. 279–291.[10] BOYD-WICKIZER, S., CHEN, H., CHEN, R., MAO, Y.,KAASHOEK, F.,MORRIS, R., PESTEREV, A., STEIN, L., WU,M., DAI, Y., ZHANG, Y., AND ZHANG, Z. Corey: An operatingsystem for many cores. In Proceedings of the 8th Symposium onOperating Systems Design and Implementation (2008), pp. 43–57.[11] BOYD-WICKIZER, S., CLEMENTS, Format Factory 4.6.2.0 Keygen Crack Key For U, A. T., MAO, Y., PESTEREV,A., KAASHOEK, M.F.,MORRIS, R., AND ZELDOVICH, N. AnAnalysis of Linux Scalability to Many Cores. In Proceedings ofthe 9th USENIX Conference on Operating Systems Design andImplementation (2010), pp. 1–8.[12] BUTLER, M., BARNES, L., SARMA, D. D., AND GELINAS, B.Bulldozer: An approach to multithreaded compute performance.IEEE Micro 31, 2 (Mar. 2011), 6–15.[13] CANTRILL, Format Factory 4.6.2.0 Keygen Crack Key For U, B. M., SHAPIRO, M. W., AND LEVENTHAL, A. H.Dynamic instrumentation of production systems. In Proceedingsof the USENIX Annual Technical Conference (2004), pp. 15–28.[14] CHAPIN, J., ROSENBLUM, M., DEVINE, S., Format Factory 4.6.2.0 Keygen Crack Key For U, LAHIRI, T., TEO-DOSIU, D., AND GUPTA, A. Hive: Fault containment for sharedmemorymultiprocessors. In Proceedings of the 15th ACM Symposiumon Operating Systems Principles (1995), pp. 12–25.[15] CHARLES, J., JASSI, P., S, A. N., SADAT, A., AND FEDOROVA,A. Evaluation of the Intel Core i7 Turbo Boost feature. In Proceedingsof the IEEE International Symposium on Workload Characterization(2009).[16] CORBET, J. Deadline scheduling for 3.14. http://www.linuxfoundation.org/news-media/blogs/browse/2014/01/deadline-scheduling-314, 12.04.14.1430 11th Format Factory 4.6.2.0 Keygen Crack Key For U Symposium on Operating Systems Design and Implementation (OSDI ’14) USENIX Association

[17] DAGAND, P.-E., BAUMANN, A., AND ROSCOE, T. Filet-o-Fish:practical and dependable domain-specific languages for OS development.In Proceedings of the 5th Workshop on ProgrammingLanguages and Operating Systems (2009).[18] DEPOUTOVITCH, A., AND STUMM, M. Otherworld: Giving Applicationsa Chance to Survive OS Kernel Crashes. In Proceedingsof the EuroSys Conference (2010), pp. 181–194.[19] DERRIN, P., ELKADUWE, D., AND ELPHINSTONE, K. seL4Reference Manual. NICTA, 2006. http://www.ertos.nicta.com.au/research/sel4/sel4-refman.pdf.[20] ELKADUWE, D., DERRIN, P., AND ELPHINSTONE, K. Kerneldesign for isolation and assurance of physical memory. InProceedings of the 1st Workshop on Isolation and Integration inEmbedded Systems (2008), pp. 35–40.[21] ESMAEILZADEH, H., BLEM, E.,ST. AMANT, R., SANKAR-ALINGAM, K., AND BURGER, D. Dark Silicon and the End ofMulticore Scaling. In Proceedings of the 38th Annual InternationalSymposium on Computer Architecture (2011), pp. 365–376.[22] GIUFFRIDA, C., KUIJSTEN, A., Format Factory 4.6.2.0 Keygen Crack Key For U, AND TANENBAUM, A. S, Format Factory 4.6.2.0 Keygen Crack Key For U. Safeand automatic live update for operating systems. In Proceedingsof the 18th International Conference on Architectural Support forProgramming Languages and Operating Systems (2013), pp. 279–292.[23] GLEIXNER, T.,MCKENNEY, P. E., AND GUITTOT, V. Cleaningup Linux’s CPU hotplug for real time and energy management.SIGBED Rev. 9, 4 (Nov. 2012), 49–52.[24] HARDY, N. KeyKOS Architecture. SIGOPS Operating SystemsReview 19, 4 (1985), 8–25.[25] CPU hotplug. https://wiki.linaro.org/WorkingGroups/PowerManagement/Archives/Hotplug, 12.04.14.[26] IPEK, E., KIRMAN, M., KIRMAN, N., AND MARTINEZ, J. F.Core Fusion: Accommodating Software Diversity in Chip Multiprocessors.In Proceedings of the 34th Annual InternationalSymposium on Computer Architecture (2007), pp. 186–197.[27] JOSHI, A. Twin-Linux: Running independent Linux Kernelssimultaneously on separate cores of a multicore system. In Proceedingsof the Linux Symposium (2010), Format Factory 4.6.2.0 Keygen Crack Key For U, pp. 101–108.[28] KLEIN, G., ELPHINSTONE, K., HEISER, G., ANDRONICK, J.,COCK, D., DERRIN, P., ELKADUWE, D., Format Factory 4.6.2.0 Keygen Crack Key For U, K.,KOLANSKI, R., NORRISH, M., SEWELL, T.,TUCH, H., ANDWINWOOD, S. seL4: Formal verification of an OS kernel. InProceedings of the 22nd ACM Symposium on Operating SystemPrinciples (2009).[29] KONGETIRA, P.,AINGARAN, K., AND OLUKOTUN, K. Niagara:a 32-way multithreaded sparc processor. IEEE Micro 25, 2 (2005),21–29.[30] KOZUCH, M. A., Format Factory 4.6.2.0 Keygen Crack Key For U, KAMINSKY, M., AND RYAN, M. P. Migrationwithout virtualization. In Proceedings of the 12th Workshop onHot Topics in Operating Systems (2009), pp. 10–15.[31] KUMAR, R., FARKAS, K. I., JOUPPI, N. P., RANGANATHAN,P., AND TULLSEN, D. M. Single-ISA Heterogeneous Multi-CoreArchitectures: The Potential for Processor Power Reduction. InProceedings of the 36th Annual IEEE/ACM International Symposiumon Microarchitecture (2003), pp. 81–92.[32] Real-time Linux. https://rt.wiki.kernel.org/, 12.04.14.[33] LIU, R., KLUES, K., BIRD, S., HOFMEYR, S., ASANOVIĆ, K.,AND KUBIATOWICZ, J. Tessellation: Space-time partitioning in amanycore client OS. In Proceedings Format Factory 4.6.2.0 Keygen Crack Key For U the 1st USENIX Workshopon Hot Topics in Parallelism (2009).[34] MARR, D. T., DESKTOP, F. B., HILL, D. L., HINTON, G.,KOUFATY, D. A., MILLER, J. A., AND UPTON, M. Hyper-Threading Technology Architecture and Microarchitecture. IntelTechnology Journal (Feb 2002).[35] MENZI, D. Support for heterogeneous cores for Barrelfish. Master’sthesis, Department of Computer Science, ETH Zurich, July2011.[36] NOMURA, Y., SENZAKI, R., NAKAHARA, D., USHIO, H.,KATAOKA, T., AND TANIGUCHI, H. Mint: Booting multipleLinux kernels on a multicore processor. In Proceedings of theInternational Conference on Broadband and Wireless Computing,Communication and Applications (2011), Format Factory 4.6.2.0 Keygen Crack Key For U, pp. 555–560.[37] PANNEERSELVAM, S., AND SWIFT, M. M. Chameleon: Operatingsystem support for dynamic processors. In Proceedings of the17th International Format Factory 4.6.2.0 Keygen Crack Key For U on Architectural Support for ProgrammingLanguages and Operating Systems (2012), Format Factory 4.6.2.0 Keygen Crack Key For U, pp. 99–110.[38] Popcorn Linux. http://popcornlinux.org/, 12.04.14.[39] PostgreSQL, Format Factory 4.6.2.0 Keygen Crack Key For U. http://www.postgresql.org/, 12.04.14.[40] RHODEN, B., KLUES, K., ZHU, D., AND BREWER, E. Improvingper-node efficiency in the datacenter with new OS abstractions.In Proceedings of the 2nd ACM Symposium on Cloud Computing(2011), pp. 25:1–25:8.[41] SADINI, M., BARBALACE, A., RAVINDRAN, B., AND QUAGLIA,F. A Page Coherency Protocol for Popcorn Replicated-kernelOperating System. In Proceedings of the ManyCore ArchitectureResearch Community Symposium (MARC) (Oct. 2013).[42] SHAPIRO, J. S., SMITH, J. M., AND FARBER, D. J. EROS: AFast Capability System. In Proceedings of the 17th ACM Symposiumon Operating Systems Principles (1999), pp. 170–185.[43] SHELDON, B. H. Popcorn Linux: enabling efficient inter-corecommunication in a Linux-based multikernel operating system.Master’s thesis, Virginia Polytechnic Institute and State University,May 2013.[44] SINGHANIA, A., KUZ, I., Format Factory 4.6.2.0 Keygen Crack Key For U, AND NEVILL, M. Capability Managementin Barrelfish. Technical Note 013, Barrelfish Project, ETHZurich, December 2013.[45] Stress Load Generator. http://people.seas.harvard.edu/~apw/stress/, 12.04.14.[46] TPC-H. http://www.tpc.org/tpch/, 12.04.14.[47] VENKATESH, G., SAMPSON, J., GOULDING, N., GARCIA, S.,BRYKSIN, V.,LUGO-MARTINEZ, J., SWANSON, S., AND TAY-LOR, M. B. Conservation Cores: Reducing the energy of maturecomputations. In Proceedings of the 15th International Conferenceon Architectural Support for Programming Languages andOperating Systems (2010), pp. 205–218.[48] WENTZLAFF, D., GRUENWALD III, C., BECKMANN, N.,MODZELEWSKI, K., BELAY, A., YOUSEFF, L., MILLER, J.,AND AGARWAL, A. An operating system for multicore andclouds: Mechanisms and implementation. In ACM Symposium onCloud Computing (SOCC) (June 2010).[49] YEE, B., SEHR, D., DARDYK, G., CHEN, J. B., MUTH, R.,ORMANDY, T.,OKASAKA, S., NARULA, N., AND FULLAGAR,N. Native client: A sandbox for portable, untrusted x86 nativecode. In Proceedings of the 30th IEEE Symposium on Securityand Privacy (2009), pp. 79–93.[50] ZELLWEGER, G., SCHUEPBACH, A., AND ROSCOE, T. UnifyingSynchronization and Events in a Multicore OS. In Proceedings ofthe 3rd Asia-Pacific Workshop on Systems (2012).15USENIX Association 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) 31

{"id": "OPENVAS:1361412562310869834", "type": "openvas", "bulletinFamily": "scanner", "title": "Fedora Update for openssh FEDORA-2015-12054", "description": "The remote host is missing an update for the ", "published": "2015-08-11T00:00:00", "modified": "2019-03-15T00:00:00", "cvss": {"score": 8.5, "vector": "AV:N/AC:L/Au:N/C:P/I:N/A:C"}, "href": "http://plugins.openvas.org/nasl.php?oid=1361412562310869834", "reporter": "Copyright (C) 2015 Greenbone Networks GmbH", "references": ["2015-12054", "https://lists.fedoraproject.org/pipermail/package-announce/2015-August/163045.PRM", "9309.PRM", "ALA_ALAS-2015-625.NASL", "CENTOS_RHSA-2015-0425.NASL", "CENTOS_RHSA-2015-2088.NASL", "CENTOS_RHSA-2016-0466.NASL", "DEBIAN_DLA-288.NASL", "F5_BIGIP_SOL17113.NASL", "FEDORA_2015-11981.NASL", "FEDORA_2015-13469.NASL", "FREEBSD_PKG_5B74A5BC348F11E5BA05C80AA9043978.NASL", "GENTOO_GLSA-201512-04.NASL", "MACOSX_10_10_5.NASL", "NEWSTART_CGSL_NS-SA-2019-0146_OPENSSH-LATEST.NASL", "OPENSSH_70.NASL", "OPENSSH_MAXAUTHTRIES_BRUTEFORCE.NASL", "ORACLELINUX_ELSA-2015-0425.NASL", "ORACLELINUX_ELSA-2015-2088.NASL", "ORACLELINUX_ELSA-2016-0466.NASL", "ORACLELINUX_ELSA-2016-3531.NASL", "ORACLEVM_OVMSA-2016-0038.NASL", "ORACLEVM_OVMSA-2016-0070.NASL", "PFSENSE_SA-15_08.NASL", "REDHAT-RHSA-2015-0425.NASL", "REDHAT-RHSA-2015-2088.NASL", "REDHAT-RHSA-2016-0466.NASL", "SL_20150305_OPENSSH_ON_SL7_X.NASL", "SL_20151119_OPENSSH_ON_SL7_X.NASL", "SL_20160321_OPENSSH_ON_SL6_X.NASL", "SUSE_SU-2015-1544-1.NASL", "SUSE_SU-2015-1547-1.NASL", "SUSE_SU-2015-1547-2.NASL", "SUSE_SU-2015-1581-1.NASL", "SUSE_SU-2015-1695-1.NASL", "SUSE_SU-2015-1840-1.NASL", "UBUNTU_USN-2710-1.NASL", "UBUNTU_USN-2710-2.NASL", Format Factory 4.6.2.0 Keygen Crack Key For U, "REDHAT-RHSA-2015-0425.NASL", "SL_20151119_OPENSSH_ON_SL7_X.NASL", "SUSE_SU-2015-1547-2.NASL"]}, {"type": "openvas", "idList": ["OPENVAS:1361412562310122744", "OPENVAS:1361412562310869875"]}, {"type": "oracle", "idList": ["ORACLE:CPUJUL2018-4258247"]}, Format Factory 4.6.2.0 Keygen Crack Key For U, {"type": media player classic always on top Crack Key For U, "idList": ["ELSA-2015-0425"]}, {"type": "redhat", "idList": ["RHSA-2015:0425"]}, {"type": "securityvulns", "idList": ["SECURITYVULNS:VULN:14614"]}, {"type": "suse", "idList": ["SUSE-SU-2015:1581-1"]}, {"type": "symantec", "idList": ["SMNTC-1337"]}, {"type": "ubuntu", "idList": ["USN-2710-2"]}, {"type": "ubuntucve", "idList": ["UB:CVE-2015-5600"]}]}, "exploitation": null, "vulnersScore": -0.2}, "pluginID": "1361412562310869834", "sourceData": "###############################################################################\n# OpenVAS Vulnerability Test\n#\n# Fedora Update for openssh FEDORA-2015-12054\n#\n# Authors:\n# System Generated Check\n#\n# Copyright:\n# Copyright (C) 2015 Greenbone Networks GmbH, http://www.greenbone.net\n#\n# This program is free software; you can redistribute it and/or modify\n# it under the terms of the GNU General Public License version 2\n# (or any later version), as published by the Free Software Foundation.\n#\n# This program is distributed in the hope that it will be useful,\n# but WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the\n# GNU General Public License for more details.\n#\n# You should have received a copy of the GNU General Public License\n# along with this program; if not, write to the Free Software\n# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA.\n###############################################################################\n\nif(description)\n{\n script_oid(\"1.3.6.1.4.1.25623.1.0.869834\");\n script_version(\"$Revision: 14223 $\");\n script_tag(name:\"last_modification\", value:\"$Date: 2019-03-15 14:49:35 +0100 (Fri, 15 Mar 2019) $\");\n script_tag(name:\"creation_date\", value:\"2015-08-11 11:56:26 +0530 (Tue, 11 Aug 2015)\");\n script_cve_id(\"CVE-2015-5600\", \"CVE-2014-9278\");\n script_tag(name:\"cvss_base\", value:\"8.5\");\n script_tag(name:\"cvss_base_vector\", value:\"AV:N/AC:L/Au:N/C:P/I:N/A:C\");\n script_tag(name:\"qod_type\", value:\"package\");\n script_name(\"Fedora Update for openssh FEDORA-2015-12054\");\n script_tag(name:\"summary\", value:\"The remote host is missing an update for the 'openssh'\n package(s) announced via the referenced advisory.\");\n script_tag(name:\"vuldetect\", value:\"Checks if a vulnerable version is present on the target host.\");\n script_tag(name:\"affected\", value:\"openssh on Fedora 21\");\n script_tag(name:\"solution\", value:\"Please install the updated package(s).\");\n script_xref(name:\"FEDORA\", value:\"2015-12054\");\n script_xref(name:\"URL\", value:\"https://lists.fedoraproject.org/pipermail/package-announce/2015-August/163045.html\");\n script_tag(name:\"solution_type\", value:\"VendorFix\");\n script_category(ACT_GATHER_INFO);\n script_copyright(\"Copyright (C) 2015 Greenbone Networks GmbH\");\n script_family(\"Fedora Local Security Checks\");\n script_dependencies(\"gather-package-list.nasl\");\n script_mandatory_keys(\"ssh/login/fedora\", \"ssh/login/rpms\", re:\"ssh/login/release=FC21\");\n\n exit(0);\n}\n\ninclude(\"revisions-lib.inc\");\ninclude(\"pkg-lib-rpm.inc\");\n\nrelease = rpm_get_ssh_release();\nif(!release)\n exit(0);\n\nres = \"\";\n\nif(release == \"FC21\")\n{\n\n if ((res = isrpmvuln(pkg:\"openssh\", rpm:\"openssh~6.6.1p1~15.fc21\", rls:\"FC21\")) != NULL)\n {\n security_message(data:res);\n exit(0);\n }\n\n if (__pkg_match) exit(99);\n exit(0);\n}\n", "naslFamily": "Fedora Local Security Checks", "immutableFields": [], "cvss2": {}, "cvss3": {}, Format Factory 4.6.2.0 Keygen Crack Key For U, "_state": {"dependencies": 1659988328, "score": 1659976568}, "_internal": {"score_hash": "71706e8a14c989466683de0d3ac16902"}}