FSSDB
About FileSystemStorage FileSystemStorage aims to be a Python DBAPI 2.0 compliant driver for treating the file system as a database. This driver will provide a mechanism for creating entities via SQL, querying via SQL, and connection to local or remote hosts (remote is via ssh), among other features.
This does not aim to be an object-relational mapping (where the objects are components of a filesystem). It does, however, aim to provide the structure necessary to easily treat a filesystem as a database by enforcing a directory structure, metadata files, and naming convention.
Basic Principles- FSSDB does not address arbitrary data structions in the same way that a standard RDBMS doesn't.
- Because full paths to BLOBs determine a unique id, only one blob may be defined per table in the field defs.
- The order of the rows (i.e., the hierarchies of the row directories) is unimportant as long as it's consistant; row hierarchy does not imply meaning or association.
Requirements
The FutureThere are all kinds of things we have in mind for the future of FileSystemStorage. Here are just a few:- Python DBAPI 2.0 Compliance
- The option of having numerical/autogenerated file and directory structures vs. user-friendly names in a diretory structure
- Creating databases and tables vi SQL and command-line tools
- Implement commit/rollback functionality
- Remote capabilities using ssh/pexpect
- Indexing
- Cached queries
- Security and authentication modules
Usage Examples
Database connection and database creation.
>>> from adytum.storage.fssdb import fssdb
>>> conn_dict = {'host':'localhost', 'name':'test_db', 'username':None, 'password':None, 'path':'/tmp'}
>>> db = fssdb(conn_dict)
>>> curs = db.cursor()
>>> sql = "CREATE DATABASE '%s' %s" % (db.path, db.name)
>>> curs.execute(sql)
And now, look at the results on the file system:
mdcitx01:oubiwann 21:17:18 $ ls -al /tmp/TEST_DB/
total 8
drwxr-xr-x 3 oubiwann wheel 102 26 Sep 21:17 .
drwxrwxrwt 12 root wheel 408 26 Sep 21:17 ..
-rw-r--r-- 1 oubiwann wheel 58 26 Sep 21:17 .properties
With the contents of the .properties file being:
mdcitx01:oubiwann 21:17:21 $ cat /tmp/TEST_DB/.properties
[metadata]
type = database
name = TEST_DB
description =
Table creation:
>>> sql = "CREATE TABLE address_book (first_name VARCHAR(25), last_name VARCHAR(25), phone_number VARCHAR(15))"
>>> curs.execute(sql)
And now, look at the results on the file system:
mdcitx01:oubiwann 22:59:42 $ ls -al /tmp/TEST_DB/
total 16
drwxr-xr-x 5 oubiwann wheel 170 27 Sep 22:57 .
drwxrwxrwt 12 root wheel 408 27 Sep 22:57 ..
-rw-r--r-- 1 oubiwann wheel 58 27 Sep 22:57 .properties
drwxr-xr-x 3 oubiwann wheel 102 27 Sep 22:57 ADDRESS_BOOK
-rw-r--r-- 1 oubiwann wheel 2147 27 Sep 22:57 fssdb.log
as well as inside the .properties file:
mdcitx01:oubiwann 00:02:20 $ cat /tmp/TEST_DB/ADDRESS_BOOK/.properties
[metadata]
field defs = [('first_name', 'VARCHAR', '25'), ('last_name', 'VARCHAR', '25'), ('phone_number', 'VARCHAR', '15')]
type = table
name = ADDRESS_BOOK
description =
Inserting data into the table:
>>> sql = "INSERT INTO address_book(first_name, last_name, phone_number) VALUES('Duncan', 'McCloud', '011-555-1212')"
>>> curs.execute(sql)
Querying data:
>>> sql = 'SELECT last_name FROM address_book'
>>> curs.execute(sql)
>>> curs.fetchone()
'/tmp/TEST_DB/ADDRESS_BOOK/Duncan/McCloud'
>>> curs.fetchone()
'/tmp/TEST_DB/ADDRESS_BOOK/Duncan/McGreggor'
>>> curs.fetchone()
>>>
>>> sql = 'SELECT * FROM address_book'
>>> curs.execute(sql)
>>> curs.fetchone()
'/tmp/TEST_DB/ADDRESS_BOOK/Duncan/McCloud/011-555-1212'
>>> curs.fetchone()
'/tmp/TEST_DB/ADDRESS_BOOK/Duncan/McGreggor/555-1212'
>>> curs.fetchone()
>>>
News
We are now SELECTing...[ 2004.10.10] After a little hiatus, we've hit the code again, and have some good results -- we have SELECTs working properly. We also converted the search method to a generator, and are quite happy with the results so far. We're going to take another break for about a week or two, and then we'll be back to implement a special case of BLOB support, at which point it should be ready for an alpha release.
Working Code![ 2004.09.26] After a month in developer sandboxes, prototype code has matured and coallesced into a coherent library. As a result, we felt it appropriate to celebrate by creating a new Subversion repository to house the code. We are making rapid progress, with database creation done and table creation on its way. We're simultaneously working on the select implementation, and we should have more example usage ready by the end of the week.
New Project Web Site[ 2004.09.01] New site site is up. Look and feel borrowed from pyHTMLWidgets.
FileSystemStorage Launched on SF.net[ 2004.09.01] FileSystemStorage has an official home on SourceForge.net. Although the first version of the code is written, the project site is yet to be completed and there is nothing in CVS at the moment.
|
