Writing Modules

Writing Modules — basic gnome-vfs module concepts

Introduction

This section will introduce the basic concepts that are needed for writing GNOME Virtual File System modules.

GNOME VFS URIs (Uniform Resource Identifiers)

The GNOME Virtual file system uses URIs similiar to the standard WWW URIs. The basic difference between a VFS URI and WWW URI is that, while with WWW URIs you can only use a single protocol for accessing a certain file, with GNOME VFS URIs you can combine different access methods in sequence.

For example, suppose you want to access file hello.c in a tar.gz file which is in turn accessible through FTP from a remote machine. In order to access this file, you would need to:

Connect to the FTP site.
Fetch the tar.gz file.
Decompress the tar.gz file using GZIP.
Extract hello.c from the resulting uncompressed tar file.

The GNOME Virtual File System lets you express this by combining the three access methods (i.e. tar, GZIP and FTP) into a single URI. Access methods are combined in the URI by using the `#' character, followed by the name for the access method and the subpath for that specific access method. The subpath can be omitted for those storage methods that do not need a path to retrieve the file. (For example, a GZIP file always contains a single uncompressed file, so no path is needed to locate the uncompressed file within the GZIP file. But on the other hand, the TAR method requires a path to locate a specific file or directory.)

For example, in the case we outlined above, the URI would look something like:


        ftp://username:password@host.net/path/to/file.tar.gz#gzip:#tar:/path/to/hello.c

Each method/subpath couple is called a URI element. When URI elements are combined like this, each URI element uses the previous one to access a base resource into which it will look up a file, using the subpath information. For this reason, we will say that each element is the parent element for the following one.

The first URI element, the one which has no parent, is called the toplevel element. It does not use the `#' character; instead, it uses the standard syntax of WWW URIs:


        method://user:password@host/path/to/file

This way, normal WWW URIs can be used with the GNOME Virtual File System.

Toplevel elements are also special because they let users specify user names, passwords and host names, while non-toplevel elements don't.

The GnomeVFSURI type

Within the GNOME Virtual File System library, URI elements are represented by a special type, GnomeVFSURI, which is meant to represent user-provided URIs in a machine-optimized way.

Every GnomeVFSURI contains the following information:

A reference counter
A pointer to the parent GnomeVFSURI URI element.
The subpath.
The name of the access method.
A pointer to a GnomeVFSMethod object, describing the access method (see below).

GNOME Virtual File System access method implementation

In the GNOME Virtual File System, the implementations for all the access methods are loaded at runtime, as shared library modules. The modules are loaded during parsing of the string URI: if the parser encounters an access method for which no implementation is currently loaded, it retrieves the corresponding library file, dynamically links it into the executable, and initializes it.

After initialization, the module returns a special GnomeVFSMethod object that contains pointers to the various implementation functions for that specific method. By storing a pointer to this object into the GnomeVFSURI type, the VFS library is then able to use these functions for file access.

How file access is performed

When the VFS library needs to perform some file operation, it performs the following steps:

If the URI is given in textual form (i.e. as a string), it parses it and activates the necessary access method modules.
It retrieves a pointer to the lowmost level URI element.
It retrieves a pointer to the GnomeVFSMethod object that corresponds to the access method for that URI element.
It retrieves a pointer to the implementation function for that operation from the GnomeVFSMethodobject.
It invokes that implementation function passing the pointer to the lowmost level URI element.

Combining the access methods is always done within the method implementation. If the method implementation needs to do some file operation on the the parent URI element, it can do so by simply invoking the corresponding VFS function in, by using the parent pointer in the GnomeVFSURI object.

For example, suppose you have to read a simple URI like the following:


        file:/home/ettore/file.gz#gzip:

In this case, the GZIP method will be invoked with a pointer to the GnomeVFSURI describing the `gzip' part; then the GZIP method will be able to read file.gz by just invoking the corresponding GNOME VFS library function on its parent, and decompress it on the fly.

Implementing an access method in practice

Implementing a new access method is really not difficult at all. This section explains how this is done.

Using shared libraries

Every module must be compiled as a shared library (i.e. a .so file).

The current way for accessing the right module for the right method is very simple, and is based on file names. In practice, a module implementing access method named foo must be named libfoo.so. For example, the module implementing the ftp: access method is called libftp.so, the module implementing #gzip: access is called libgzip.so and so on.

This might change in the future.

The initialization/shutdown functions

Every shared library module must provide two functions:


GnomeVFSMethod *vfs_module_init (void);
void vfs_module_shutdown (GnomeVFSMethod *method);

These are the only functions that the VFS library will access directly. All the other symbols (i.e. functions and variables) in the module should be made static.

vfs_module_init() is called as soon as the module is loaded in memory. It will have to return a pointer to a GnomeVFSMethod object that will contain the pointers to the method's implementation functions. We will describe this later.

vfs_module_shutdown, instead, is called before the module is unloaded or the program that uses it dies. This functions should:

Deallocate all the memory allocated by the module.
Close all the file descriptors associated with the module.
Kill any external process spawned by the module.
In general, make sure that any operation that was going on before this function was called will be interrupted correctly, as soon as possible and without any leaks.

The GnomeVFSMethod object

This object is contains pointers to the module implementation functions.


GnomeVFSResult (* open)              (GnomeVFSMethodHandle **method_handle_return,
                                      GnomeVFSURI *uri,
                                      GnomeVFSOpenMode mode
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* create)            (GnomeVFSMethodHandle **method_handle_return,
                                      GnomeVFSURI *uri,
                                      GnomeVFSOpenMode mode,
                                      gboolean exclusive,
                                      guint perm
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* close)             (GnomeVFSMethodHandle *method_handle
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* read)              (GnomeVFSMethodHandle *method_handle,
                                      gpointer buffer,
                                      GnomeVFSFileSize num_bytes,
                                      GnomeVFSFileSize *bytes_read_return
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* write)             (GnomeVFSMethodHandle *method_handle,
                                      gconstpointer buffer,
                                      GnomeVFSFileSize num_bytes,
                                      GnomeVFSFileSize *bytes_written_return
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* seek)              (GnomeVFSMethodHandle *method_handle,
                                      GnomeVFSSeekPosition  whence,
                                      GnomeVFSFileOffset    offset
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* tell)              (GnomeVFSMethodHandle *method_handle,
                                      GnomeVFSFileOffset *offset_return);

GnomeVFSResult (* truncate)          (GnomeVFSMethodHandle *method_handle,
                                      GnomeVFSFileSize where
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* open_directory)    (GnomeVFSMethodHandle **method_handle,
                                      GnomeVFSURI *uri,
                                      GnomeVFSFileInfoOptions options,
                                      const GList *meta_keys,
                                      const GnomeVFSDirectoryFilter *filter
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* close_directory)   (GnomeVFSMethodHandle *method_handle
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* read_directory)    (GnomeVFSMethodHandle *method_handle,
                                      GnomeVFSFileInfo *file_info
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* get_file_info)     (GnomeVFSURI *uri,
                                      GnomeVFSFileInfo *file_info,
                                      GnomeVFSFileInfoOptions options,
                                      const GList *meta_keys
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* get_file_info_from_handle)
                                     (GnomeVFSMethodHandle *method_handle,
                                      GnomeVFSFileInfo *file_info,
                                      GnomeVFSFileInfoOptions options,
                                      const GList *meta_keys
                                      GnomeVFSCancellation *cancellation);

gboolean       (* is_local)          (const GnomeVFSURI *uri
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* rename)            (GnomeVFSURI *uri, const gchar *new_name
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* make_directory)    (GnomeVFSURI *uri, guint perm
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* remove_directory)  (GnomeVFSURI *uri
                                      GnomeVFSCancellation *cancellation);

GnomeVFSResult (* unlink)            (GnomeVFSURI *uri
                                      GnomeVFSCancellation *cancellation);

Handling cancellation

As VFS operations might be very long, especially in the case of transient errors (such as a network server that has gone down), the GNOME Virtual File System Library provides a standard way for handling cancellation of VFS operations.

The GnomeVFSCancellation object

The object that encapsulates this functionality is GnomeVFSCancellation. Most implementation functions get a pointer to such an object, and are expected to use this object to recognize when an operation should be interrupted.

The most simple way to check for a cancellation request is to poll the object with gnome_vfs_cancellation_check():

  
gboolean gnome_vfs_cancellation_check (GnomeVFSCancellation *cancellation);

This function will return a nonzero value if the current operation should be cancelled.

Notice that cancellation is an asynchronous operation that might happen outside your function, in parallel with the code that you are writing. For example, in the case of threads, the request will be set in the master thread; in the case of slave CORBA-driven processes, the request will be activated by a Unix signal. So you can expect a cancellation request to happen (and consequently be signalled in GnomeVFSCancellation) at any time.

For this reason, you should be calling this function periodically, whenever you are going to perform several iterations of the same task, or execute a single expensive task. When the function returns a nonzero value, the correct way to react is:

Clean things up so that the result of the operations that have been performed are all cancelled.
Return the GNOME_VFS_ERROR_CANCELLED error code.

But there are some other situations in which you want to be able to interrupt a I/O operation when a cancellation request is performed. In such cases, polling is not a viable option.

For this reason, GnomeVFSCancellation provides an alternative way of sending notifications, using a file descriptor. To use this feature, you should use the following function:


gint gnome_vfs_cancellation_get_fd (GnomeVFSCancellation *cancellation);

When this function is called, it will return an open file descriptor, which is the read-side of a pipe. The pipe will be given a character from the write side as soon as a cancellation request is sent. For this reason, you can check for a cancellation by using the select() system call: as soon as select reports that some data is available on the file descriptor, you know that a cancellation is being requested.

For example, if you are reading from a file descriptor and you want to check for a pending cancellation at the same time, you can set up selectfor checking if data is available on both the cancellation file descriptor and the file descriptor you are reading from.

Dealing with EINTR

In order to maximize the chance of cancelling an operation immediately, the GNOME Virtual File System can sends a signal to the asynchronous thread or process. This does not happen on all the systems and setups, though.

The result of this is that, if a process is in the middle of a Unix system call while receiving this signal, the system call might be interrupted and return a EINTR error.

For this reason, when you receive EINTR you should check if a cancellation request is pending, using gnome_vfs_cancellation_check() on the GnomeVFSCancellation object that the implementation function received:

If a cancellation is indeed pending (gnome_vfs_cancellation_check() returns a nonzero value), you should cancel the operation, cleaning up all the effects, and return GNOME_VFS_ERROR_INTERRUPTED or GNOME_VFS_ERROR_CANCELLED
Otherwise, retry the system call as you would normally do.

Basic guidelines for writing a module

Writing GNOME VFS modules is easy, but there are a few things that you must keep in mind when hacking them:

All of the code must be completely thread safe. The reason for this is that the asynchronous GNOME VFS engine will use threads when available; if you don't make sure that the code is thread-safe, every kind of weird and unexpected errors will happen. As debugging these problems can be very hard, it's important to write the code with threads in mind right from the start.
Use the special gnome_vfs_*_cancellable() VFS functions instead of the standard non-cancellable ones, passing them the same GnomeVFSCancellation object you are given, so that the operation can always be interrrupted at any time.
The code should respect the basic GNOME guidelines for source code indentation and style.

How to make the code thread safe

Although it might sound scary at first, making the code for the modules thread safe is not complicated at all.

First of all, make sure the amount of global variables is kept to the bare minimum. If possible, you should avoid them at all cost.

For those cases where globals are inevitable (such as caches, connection pools or things like that), you have to make sure every variable is properly associated with a mutex, and that the mutex is locked before every access to this variable and released afterwards. You can also use G_LOCK_DEFINE_STATIC, G_LOCK and G_UNLOCK for this.

Generally speaking, if you are going to dynamically allocate structures that are shared by more than one operation/file, you should provide all of them with their nice mutex locks.

Finally, make sure mutexes are used only if they are available. One way to do so is to use macros like the following:


#ifdef G_THREADS_ENABLED
#define MUTEX_NEW()     g_mutex_new ()
#define MUTEX_FREE(a)   g_mutex_free (a)
#define MUTEX_LOCK(a)   if ((a) != NULL) g_mutex_lock (a)
#define MUTEX_UNLOCK(a) if ((a) != NULL) g_mutex_unlock (a)
#else
#define MUTEX_NEW()     NULL
#define MUTEX_FREE(a)
#define MUTEX_LOCK(a)
#define MUTEX_UNLOCK(a)
#endif

G_LOCK_DEFINE_STATIC, G_LOCK and G_UNLOCK in GLib are always safe to use, as they are already defined to be nothing when thread support is not available.

(Probably it would be a good idea to have something in the private GNOME VFS API that does this stuff for all the modules.)