The Plugins Methodology

Download

Download the current version of my plugins interface definition, software and sample implementations

Motivation for yet another interface

At present (as of January 1999), visual Hebrew is supported by several software packages running under GNU/Linux. Logical Hebrew is not supported. In order to allow Hebrew speakers to get weaned off Microsoft based software, logical Hebrew support has to be implemented in several text handling software running under GNU/Linux, such as: WYSIWYG word processors, graphical WWW browsers, text editors, X-Window widgets (such as Motif/Lesstif or Tk text widgets), ICQ client software, TeX, database.

In order to achieve the aforementioned goal, the following obstacles must be overcome:

  1. Software is developed by people from all over the world. Most of them neither know nor care about Hebrew or the BiDi rules.
  2. The BiDi algorithm (for converting logical Hebrew text into visual Hebrew for the purpose of rendering it) is still a moving target.
It is my belief that the best solution is to specify an interface, which will let people who develop and improve the BiDi algorithm (and similar algorithms for more exotic languages) work independently from people who develop text processing applications.

Such an interface has to meet the following requirements:

  1. Provide enough flexibility that an application writer would never have to know the details of an exotic language in order to allow his application to work seamlessly in that language, by means of the right plugin.
  2. Work with C, not C++ - to make it easier to integrate it into legacy software.
  3. Be helpful not only to Hebrew and Arabic speakers, but also to all the languages of the world - so that application writers will choose it as their contribution to I18N'ing their applications.
  4. Be very efficient - so that wordprocessors and WWW browsers won't slow down their text refreshes.
  5. Be configurable according to user's inputs (such as the ability to choose one of few BiDi algorithms).
  6. Allow different widgets in the same application to use different plugins (for example, one might want to use an X-Window application with two text widgets - one using Hebrew+BiDi, the other using Thai with its own rendering algorithm).

More possible benefits from adoption of the plugin methodology:

  • It would no longer be necessary to coordinate worldwide the development of a single worldwide Unicode standard. It will be possible to allocate to each language a chunk of character codes, and tell speakers of that language to go ahead and develop, refine and debug their own text handling standards. Once they define their text handling standards and release to the public a plugin which implements their standards, the entire world can be compatible with what they specified, without making any further effort.
  • People will be able to develop and evolve text handling algorithms in their favorite language and immediately test their algorithms in several software packages, without having to manually modify or recompile all those packages. Thus, worldwide collaboration will shorten the way to high-quality BiDi algorithm (and corresponding algorithms in other exotic languages). Did I mention the buzzwords "Bazaar development model", "Benefits of open source software"?

Why not to use CORBA or Portable Layout Services

I decided to use the dynamic linking loader feature (see man dlopen or read /usr/include/dlfcn.h for details). This interface standard originated from Solaris and was implemented also in Linux. An alternative to this feature is the CORBA, which I rejected due to the following reasons:
  1. We need only to dynamically link procedures to an application. The full power of CORBA is not needed. In particular, no data objects need to be passed to other processes.
  2. CORBA is not terribly fast.
  3. CORBA requires the user to install a CORBA server in his system.

Another interface, which is supposed to meet the same I18N needs as the proposed plugin interface, is the Portable Layout Services (PLS).

The PLS suffers from the following disadvantages:

  1. It appears that application developers must know something about the target language (Hebrew, Hangul (the Korean writing system), or whatever) in order to make their application compatible with it.
  2. It is complicated. Application writers (including widget developers) want to concentrate upon their application rather than deal with all those complicated text handling issues.

The proposed plugin interface, on the other hand, strives to encapsulate all language-specific knowledge. In order to make an application support a new language (or even use a different algorithm for an existing language), all one needs is to replace the plugin, or to supply different configuration parameters to the plugin.

By the way, the PLS and the proposed plugin interface are not really mutually exclusive! They expose different levels of abstraction to application developers.

It is possible to develop a plugin, which employs the proposed interface, and which implements the PLS. Such a plugin would relieve application developers from having to deal with all complexities of PLS.

The four levels of the plugin interface

The proposed plugin interface is, as far as I could determine, a novel concept. Therefore, the first few rounds of specifying it may not get everything right. Another hindrance is that I am not yet sufficiently familiar with the internal workings of text handling applications in order to be able to design an interface, which efficiently meets their needs. The only project, of which I am aware, and which specified their interface needs is the Mozilla project.

In order to allow for an iterative process of defining the interface details, I defined four levels of plugin interfaces, as follows:

  • Level 0 - handles only fixed-size fonts. Suitable for CURSES-based applications and applications, which run on xterm.
    The plugin receives text, transforms it and hands it back to the application for drawing on the screen.
    Services which level 0 plugins can provide: line breaking, hyphenation, conversion from logical Hebrew into visual Hebrew (for either horizontal or vertical text), right-justifying lines by padding them by spaces from the left.
  • Level 1 - like level 0, but handles also proportional fonts. Suitable for text widgets in all X-Window based widget families.
    The plugin receives text with font/size/style information, transforms it and hands the transformed text back to the caller. Each glyph with the transformed text has associated with it font information, and the distance (in pixels) from the previous glyph.
    Services which level 1 plugins can provide: transformation of ligatures, kerning.
  • Level 2 - handles the most general text drawing requirements (for Thai, Mongolian, Hangul, etc.). Level 0 and level 1 plugins do not actually draw the transformed text into a drawable. They leave this to the client application. In contrast to them, level 2 plugins are actually responsible for drawing on the drawable (window or pixmap).
    A level 2 plugin receives text from the application and evaluates it. Then negotiates the application for the location and size of a rectangle into which the text is to be drawn. Then it actually draws whatever it pleases into the rectangle and informs the application which characters of the original text remain to be drawn.
    It is probable that if the PLS plugin were ever to be implemented, it would have been a level 2 plugin.
  • Level 3 - cooperating plugins. If each plugin handles only a subset of the Unicode, and if an application wants to render multilingual text, which no single plugin can handle it all, then it would be desirable to define some means whereby several plugins can cooperate with each other in correctly rendering the multilingual text.

So far, I worked out only the details of the level 0 plugin interface. The provided software package specifies and implements only this level.

I intend to define and implement also level 1, once I have access to more knowledge and experience about the actual performance of level 0 along with the real needs of applications.

However, I plan to leave to other people, who are more familiar than me with Eastern languages, to define levels 2 and 3 in detail.

Documentation of level 0 of the plugin interface

The plugin interface has two faces. One face interests the plugin writer. The other face interests the application developer.

The plugin writer is supposed to implement the following functions:

  • _init() - optional - performs any configuration-independent package initialization needed by the plugin.
  • _fini() - optional - performs any uninitialization operations needed by the plugin.
  • return_function_pointers() - returns to the application a data structure containing pointers to all plugin functions needed by the application.
  • parse_config() - parses an user-provided configuration string and stores it in an opaque data structure, for internal use by the plugin.
    Such a configuration string is expected to be passed to Xt-compatible widgets, which need plugins, by means of a resource setting. Typically it would contain the name of a file containing all the configuration information needed by the plugin.
  • validate_config() - validates the plugin configuration.
  • delete_config() - destroys the opaque data structure created by parse_config().
  • notify_modification() - the client calls this plugin procedure any time the original text is modified. Some of the provided information may be ignored by the plugin if its algorithms don't need it for optimal performance.
  • retrieve_rendered_line() - if necessary, this procedure does the actual work of transforming text from the client-maintained logical form into the plugin-produced visual form.
    The algorithm should support lazy evaluation (to save time when the original text is very long), and efficient propagation of modifications at middle of the text.

The application developer is supposed to allow the user to select a plugin and a configuration, independently for each text window (or widget) in the application. Such a selection is to be carried out by means of an application configuration file, or by means of a Xt-compatible resource file.

The application developer is supposed to provide the plugin with the following callback procedures:

  • get_orig_text_fragment() - used by the plugin to retrieve pointers to text fragments maintained by the client application.
  • get_rendered_line_size() - used by the plugin to determine the desired size of a line to be rendered. This provides the flexibility to allow clients to use non-rectangular text windows.

The application developer is supposed to use the following functions:

  • bind_to_plugin() - to select a plugin and load it into memory.
  • set_clone_rendering_context - to configure the plugin and create a rendering context.
    Rendering contexts fill a role similar to GC in Xlib. They pack together miscellaneous parameters, which stay constant across calls to the same plugin instance and for the same rendered text data structure.
  • construct_rendered_text() - to construct an instance of the rendered text data structure, to be manipulated by other plugin procedures.
    If an application is an editor, and the user modified a single line on the screen, the application is supposed to immediately refresh only the modified line. The rest of the window (or screen) is to be modified only when the application has entered an idle loop.
  • c_notify_modification() - each time the original text is modified.
  • c_retrieve_rendered_line() - each time it is desired to retrieve a rendered text line.
  • destroy_rendered_text(), destroy_rendering_context() - to free the data structures created by other plugin procedures.
  • unbind_from_plugin() - to release the plugin and free it from memory (if not used by other applications).
  • l2v_cursor_translation(), v2l_cursor_translation() - as needed to assist the client in cursor management. They translate from logical (offset into the original text) into visual (row and cell positions) mouse coordinates, and vice versa.

Questions and Answers

How do you handle selection?
The plugin interface specs define a procedure called v2l_cursor_translation(), which receives "visual coordinates" of the mouse and/or cursor. It then computes the offset into the original string corresponding to the visual position on the string and returns it to the application. The application can then use this position for selecting a substring (by marking its beginning and ending).
If I take a text editor like pico, stuff a level 0 plugin in it and start writing Hebrew, what happens? Does pico consult plugin on every keypress?
Yes, every keypress, which modifies the text of your file.
How does the application know on which line the cursor is?
The application is supposed to keep track of the visual cursor position. The plugin provides for translation of the visual cursor position into the logical cursor position in the original text.
I am still confused. Can you please summarize for me the above points?
  1. The editing commands of the application operate only on the logical text.
  2. Cursor positioning commands of the application operate only on the visual text.
  3. After each cursor movement (by command or by mouse), the application does not compute itself the new cursor position in the logical text, but instead asks the plugin to compute this for it.
  4. After each text change, the application hands the modified text to the plugin and tells it where the text was modified. The plugin transforms as much text as needed to refresh the display, and not more than that. The plugin then (in principle) informs the application which lines on the display need to be refreshed due to the text change.

How to use and modify the provided software package

  1. Download the plugins0.0-001.tar.gz package.
  2. tar zxvf plugins0.0-001.tar.gz
  3. cd include
  4. Set the environment variable TEXT_RENDERING_PLUGIN_PATH to the current directory.
    In bash, the command is:
    TEXT_RENDERING_PLUGIN_PATH=`pwd`;export TEXT_RENDERING_PLUGIN_PATH
  5. The program plugfilter0 is a driver, which passes text from stdin to stdout via the selected plugin. The command to invoke it is:
    ./plugfilter0 plugin-name [plugin-configuration]
  6. The only plugin provided at present is null.so, which does no transformation to the text. It also receives no configuration information.
    Thus, the following command has the same functionality as cat: ./plugfilter0 null < input_file > output_file
  7. I built the binaries in a RedHat 5.1 based Linux system, with kernel version 2.0.36, gcc version 2.7.2.3, libc.so.6, libdl.so.2, and some other things.
    To rebuild the binaries, merely type make in the include subdirectory. The file Makefile takes care of all the rest (in RedHat 5.1 based Linux system).
Note:
The hardwired default for TEXT_RENDERING_PLUGIN_PATH is /usr/lib/plugins/textrendering. This is subject to change in the future, according to whatever decision the FSSTND maintainers will make.