FiST: Stackable File System Language and Templates

File System development is very difficult and time consuming. Even small changes to existing file systems require deep understanding of kernel internals, making the barrier to entry for new developers high. Moreover, porting file system code from one operating system to another is almost as difficult as the first port. Past proposals to provide extensible (stackable) file system interfaces would have simplified the development of new file systems. These proposals, however, advocated massive changes to existing operating system interfaces and existing file systems; operating system vendors and maintainers resist making any large changes to their kernels because of stability and performance concerns. As a result, file system development is still a difficult, long, and non-portable process.

The FiST (File System Translator) system combines two methods to solve the above problems in a novel way: a set of stackable file system templates for each operating system, and a high-level language that can describe stackable file systems in a cross-platform portable fashion. Using FiST, stackable file systems need only be described once. FiST's code generation tool, fistgen, compiles a single file system description into loadable kernel modules for several operating systems (currently Solaris, Linux, and FreeBSD). The project demonstrates that with FiST, code size and development time are reduced significantly, while imposing a small performance overhead of only 1-2%. These benefits are achieved, as well as portability, without changing existing operating systems or file system.

Public Software:

My stackable file systems software package (fistgen) is available from my ftp server. These distributions include these file systems and more:
- unionfs: a Unioning file system
- cryptfs: encrypts file names and data pages using Blowfish (Open Source, exportable).
- gzipfs: compresses files based on the Deflate algorithm
- uuencodefs: uuencodes files
- copyfs: turns on size-changing algorithm support (SCA) used in gzipfs and uuencodes, but then only copies data without changing its size; useful to measure the overhead of the SCA code. SCA file systems are currently available only for Linux 2.4.
- base0fs: null layer ala lofs
- base1fs: null layer, but copies data pages between layers
- base2fs null layer, but copies file names between layers
- base3fs: null layer, but copies both file names and data pages between layers.
- wrapfs: same as base3fs
- rot13fs: cheap encryption file system using rot13
- rot1fs: same as rot13fs, but using an asymmetric algorithm.
- snoopfs: tells when users snoop your files
- aclfs: adds simple ACLs per directory
If you want stacking support (for lofs, wrapfs, cryptfs, gzipfs, etc.) for the Linux kernels prior to 2.4, then apply the appropriate linux kernel patch. No patches are required for Linux 2.4.
Browse the main four fistgen file systems for fistgen-0.0.7 under linux-2.4.24 using LXR: base0fs, base1fs, base2fs, base3fs (aka Wrapfs).
Browse the main four fistgen file systems for fistgen-cvs (as of March 17, 2004) under linux-2.4.24 using LXR: base0fs, base1fs, base2fs, base3fs (aka Wrapfs).
Browse the main four fistgen file systems for fistgen-0.0.7 under linux-2.4.25 using LXR: base0fs, base1fs, base2fs, base3fs (aka Wrapfs).
Browse the main four fistgen file systems for fistgen-cvs (as of March 17, 2004) under linux-2.4.25 using LXR: base0fs, base1fs, base2fs, base3fs (aka Wrapfs).
You can download unreleased daily snapshots of the fistgen package.

Mailing Lists:

Click on the list name below to access the subscription Web page. That page will also tell you how to post to the list, access the lists' archives, change your subscription status, etc.

List Name Purpose Who Can Subscribe Who Can Post Comments

fistgen For users' and developers' questions, announcements, bug reports, and general. Anyone Subscribers only This list used to be called fist at cs.columbia.edu. Use new name please.

fistgen-announce For announcements only Anyone Developers only

Miscellaneous:

If you are interested in a stackable file systems in general, I have a collection of old stacking papers.

If you find a problem, please describe it in detail and submit a bug report via Bugzilla.

Download software.

Journal Articles:

#	Title (click for html version)	Formats	Published In	Date	Comments
1	The Case for Specialized File Systems, or, Fighting File System Obesity	PDF BibTeX	;login: The USENIX Magazine	Feb 2010
2	On Incremental File System Development	PS PDF BibTeX	ACM Transactions on Storage (TOS)	May 2006
3	Versatility and Unix Semantics in Namespace Unification	PS PDF BibTeX	ACM Transactions on Storage (TOS)	Feb 2006
4	Unionfs: Bringing File Systems Together	BibTeX	Linux Journal	Dec 2004
5	Writing Stackable Filesystems	BibTeX	Linux Journal	May 2003	Software described in article

Conference and Workshop Papers:

#	Title (click for html version)	Formats	Published In	Date	Comments
1	RAIF: Redundant Array of Independent Filesystems	PS PDF BibTeX	IEEE Conference on Mass Storage Systems and Technologies (MSST 2007)	Sep 2007
2	Kernel Support for Stackable File Systems	PS PDF BibTeX	2007 Ottawa Linux Symposium	Jun 2007
3	Making Linux Stacking-Friendly	BibTeX	First Linux Storage and Filesystem Workshop, held in conjuction with the fifth USENIX Conference on File and Storage Technologies (FAST 2007)	Feb 2007
4	UnionFS: User- and Community-oriented Development of a Unification Filesystem	PS PDF BibTeX	2006 Ottawa Linux Symposium	Jul 2006
5	Accurate and Efficient Replaying of File System Traces	PS PDF BibTeX	Fourth USENIX Conference on File and Storage Technologies (FAST 2005)	Dec 2005
6	Adding Secure Deletion to Your Favorite File System	PS PDF BibTeX	Third IEEE Security In Storage Workshop (SISW 2005)	Dec 2005
7	Stackable File Systems and Methods Thereof	PDF BibTeX	US Patent Application 10/862,212	Dec 2005	US Patent Application 10/862,212 publication
8	Increasing Distributed Storage Survivability with a Stackable RAID-like File System	PS PDF BibTeX	First IEEE/ACM Workshop on Cluster Security, in conjunction with the Fifth IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 2005)	May 2005	Won Best Paper Award
9	I3FS: An In-Kernel Integrity Checker and Intrusion Detection File System	PS PDF BibTeX	Usenix Large Installation Systems Administration Conference (LISA 18)	Nov 2004
10	Avfs: An On-Access Anti-Virus File System	PS PDF BibTeX	13th USENIX Security Symposium (Security 2004)	Aug 2004
11	A Versatile and User-Oriented Versioning File System	PS PDF BibTeX	Third USENIX Conference on File and Storage Technologies (FAST 2004)	Apr 2004
12	Tracefs: A File System to Trace Them All	PS PDF BibTeX	Third USENIX Conference on File and Storage Technologies (FAST 2004)	Apr 2004
13	Reducing Storage Management Costs via Informed User-Based Policies	PS PDF BibTeX	NASA/IEEE Conference on Mass Storage Systems and Technologies (MSST 2004)	Apr 2004
14	Cryptographic File Systems Performance: What You Don't Know Can Hurt You	PS PDF BibTeX	2003 IEEE Security In Storage Workshop (SISW 2003)	Oct 2003
15	NCryptfs: A Secure and Convenient Cryptographic File System	PS PDF BibTeX Slides	Usenix Technical Conference, General Track	Jun 2003
16	Fast Indexing: Support for Size-Changing Algorithms in Stackable File System	PS PDF BibTeX	Usenix Technical Conference	Jun 2001
17	FiST: A Language for Stackable File Systems	PS PDF BibTeX	Usenix Technical Conference	Jun 2000
18	Extending File Systems Using Stackable Templates	PS PDF BibTeX	Usenix Technical Conference	Jun 1999
19	A Stackable File System Interface For Linux	PS PDF BibTeX	LinuxExpo 99	May 1999

Technical Reports:

#	Title (click for html version)	Formats	Published In	Date	Comments
1	Storage Virtualization with a Stackable File System	PDF BibTeX	Stony Brook U. CS TechReport FSL-05-03	Dec 2005	M.S. Thesis
2	Versatility and Unix Semantics in a Fan-Out Unification File System	PS PDF BibTeX	Stony Brook U. CS TechReport FSL-04-01b	Oct 2004
3	Versatile File System Tracing with Tracefs	PS PDF BibTeX	Stony Brook U. CS TechReport FSL-04-05	Aug 2004	M.S. Thesis
4	Enhancing File System Integrity Through Checksums	PS PDF BibTeX	Stony Brook U. CS TechReport FSL-04-04	May 2004
5	Operating System Support for Extensible Secure File Systems	PS PDF BibTeX	Stony Brook U. CS TechReport FSL-04-02	May 2004	Ph.D. Research Proficiency Exam (RPE)
6	Cryptographic File Systems Performance: What You Don't Know Can Hurt You	PS PDF BibTeX	Stony Brook U. CS TechReport FSL-03-02	Aug 2003	Slightly expanded version of SISW'03 paper with same title.
7	Reducing Storage Management Costs via Informed User-Based Policies	PS PDF BibTeX	Stony Brook U. CS TechReport FSL-03-01	Mar 2003
8	The Design and Implementation of Elastic Quotas: A System for Flexible File System Management	PS PDF BibTeX	Columbia U. CS TechReport CUCS-014-02	Jun 2002
9	FiST: A System for Stackable File System Code Generation	PS PDF BibTeX	Columbia University	May 2001	Ph.D. Dissertation
10	Performance of Size-Changing Algorithms in Stackable File Systems	PDF BibTeX	Columbia U. CS TechReport CUCS-023-00	Dec 2000
11	Stackable File Systems As a Security Tool	PS PDF BibTeX	Columbia U. CS TechReport CUCS-036-99	Dec 1999
12	Usenetfs: A Stackable File System for Large Article Directories	PS PDF BibTeX	Columbia U. CS TechReport CUCS-022-98	Jul 1998
13	Cryptfs: A Stackable Vnode Level Encryption File System	PS PDF BibTeX	Columbia U. CS TechReport CUCS-021-98	Jun 1998	This tech-report had been cited more than 15 times before the new version of the work was published -- NCryptfs.
14	FiST: A File System Component Compiler	PS PDF BibTeX	Columbia University	Apr 1997	Ph.D. Thesis Proposal

Works In Progress:

#	Title (click for html version)	Formats	Published In	Date	Comments
1	Fast Indexing: Support for Size-Changing Algorithms in Stackable File System	PS PDF BibTeX		Oct 2001	Expanded version of Usenix 2001 paper

Past Students:

#	Name (click for home page)	Program	Period	Current Location
1	Charles P. Wright	PhD	May 2003 - May 2006	Partner, Senior Software Architect, Illumon (New York, NY)
2	Shrikar Archak	MS	Sep 2009 - Dec 2010	Senior Software Engineer, Core Deduplication Filesystem group, Datadomain/EMC
3	Puja Gupta	MS	Jan 2003 - Dec 2003	Software Engineering Manager, Darwin Runtime, Core OS, Apple Inc. (Cupertino, CA)
4	Harikesavan Pathangi Krishnan	MS	Jan 2003 - Dec 2003	Software Engineer, Engineering department for WAFS product line, Packeteer, Inc. (South Plainfield, NJ)
5	Rohit Kumar	MS	Jan 2018 - Dec 2018	Software Development Engineer II, Amazon Services, Amazon Services (Seattle, WA)
6	Kiran-Kumar Muniswamy-Reddy	MS	Jan 2002 - May 2004	Consulting Member of Technical Staff, Oracle Corp (Seattle, WA)
7	Sunil Satnur	MS	Sep 2004 - Dec 2005	Staff Engineer, Storage and Avaliability Group, VMware Inc. (Palo Alto, CA)
8	Kiron Vijayasankar	MS	Dec 2006 - Dec 2007	Member of Technical Staff, Engineering Development, Riverbed Technology (Sunnyvale, CA)
9	Charles P. Wright	BS (Honors)	Dec 2001 - May 2003	Application Software Developer, Walleye Software (New York, NY)
10	Josef "Jeff" Sipek	BS	Jul 2005 - May 2008	Senior Member of Technical Staff, Nutanix (Boston, MA)

Sponsors:

#	Sponsor	Amount	Period	Type	Title (click for award abstract)
1	HP/Intel	$131,529	2002-2003	Sole PI	Linux Application Performance and File System Security
2	HP/Intel	$22,490	2001-2002	Sole PI	Linux Network Scalability and File System Reliability

List Name	Purpose	Who Can Subscribe	Who Can Post	Comments
fistgen	For users' and developers' questions, announcements, bug reports, and general.	Anyone	Subscribers only	This list used to be called fist at cs.columbia.edu. Use new name please.
fistgen-announce	For announcements only	Anyone	Developers only