Metadata (META) Representation

To keep track of attributes that are associated with Flick's intermediate data structures, Flick stores metadata in the intermediate files that it creates (i.e., aoi and pres_c files). Metadata allows Flick to track the sources and sinks (destinations) of objects, to describe relationships among those sources and sinks, to attach additional semantics or classifications to data, and so on. All of this information is stored in a format called meta.

Metadata is used to classify objects in the intermediate files so that Flick can partition its output in various ways. For example, Flick may put certain types of data in one file and other types of data in another. Alternately, Flick might completely suppress the output of certain kinds of data. The meta format is currently constrained to representing relatively simple attributes -- e.g., the input files associated with various objects -- but the meta language may be extended in the future to support more complex kinds of annotations.

3.1 META Overview

The meta data structures are defined in mom/meta.x, and the library for manipulating meta data is contained in the directory libmeta.

meta provides two basic abstractions: io_files, representing input files,1 and data_channels, representing separate "streams" of data within a single file. These structures are used to classify data according to their source and purpose. An io_file is referenced by data structures that "arise" from the input file represented by the io_file object. A data_channel is used to classify the results generated by processing the input definitions. For example, multiple channels could be used to separate the client-side and server-side C declarations that result from a single idl definition.

In addition to the basic io_file and data_channel structures, meta defines structures for describing sets of files and channels. Sets are described by masks: patterns that may be used to find files and channels of interest.

3.2 META Data Structures

The primary metadata objects are contained within arrays which are collected under a root object of type meta:
 struct meta {
         io_file         files<>;
         data_channel    channels<>;
 };

A reference to an io_file or data_channel is implemented as an integer index into the appropriate array within a meta structure.
 typedef int io_file_index;
 typedef int data_channel_index;

Other meta data structures, such as descriptions of file and channel sets, are not stored in the root meta structure but are instead stored where they are used.

3.2.1 Files

Files are tracked with the io_file structure:
 struct io_file {
         string          id<>;           /* Filename */
         unsigned int    flags;          /* Holds the above flags */
         int             references;     /*
                                          * How many other files have included
                                          * this one
                                          */
         io_file_index   includes<>;     /* References to included files */
 };

The flags field contains a set of the following values:

IO_FILE_INPUT
Indicates that the io_file represents an input file.
IO_FILE_OUTPUT
Indicates that the io_file represents an output file. This flag is not currently used by Flick.
IO_FILE_BUILTIN
Indicates that the io_file does not correspond to any actual file. Rather, data from this "file" is already built into Flick.
IO_FILE_ROOT
Indicates that this io_file represents the root idl file from which all others are #included.
IO_FILE_SYSTEM
Indicates that this io_file represents a system file (i.e., a system header file, generally #included with <>'s).

The main set of io_file objects is generated by a Flick front end when it processes a root idl file. In addition, the front end will create any "builtin" files that it needs to indicate definitions that are known a priori to the front end. For example, the corba front end (described in Section 10.2) creates a builtin io_file to represent the source of the predefined CORBA::Object interface type. The set of io_files created by a front end encodes just about everything that Flick cares about from the files, from their names to their place(s) in the inclusion graph.

The io_file_mask structure describes a set of io_file objects:
 struct io_file_mask {
         unsigned int    mask_flags;     /* Flags specific to the mask */
         string          id<>;           /* An identifier to match */
         unsigned int    set_flags;      /*
                                          * Match when all of these flags are
                                          * set in the io_file
                                          */
         unsigned int    unset_flags;    /*
                                          * Match when all of these flags aren't
                                          * set in the io_file
                                          */
 };

This structure is supposed to be somewhat opaque; libmeta provides several functions for dealing with them. The construction of io_file_masks is handled by a tag list function (see Section 2.3.4) that sets all the fields and flags based on the tags passed in. For example, the following statement constructs an io_file_mask that matches only system files:
         meta_make_file_mask(FMA_SetFlags, IO_FILE_SYSTEM,
                             FMA_TAG_DONE);

The meta_make_file_mask function is fully described in Section 3.3, below.

3.2.2 Channels

Channels are described by the data_channel structure:
 struct data_channel {
         io_file_index           input;          /* input for the channel */
         string                  id<>;           /* semantic id */
         unsigned int            flags;          /* Holds the above flags */
         io_file_index           outputs<>;      /*
                                                  * List of output files for
                                                  * the channel (unused)
                                                  */
 };

Note that the output field is currently unused, because io_files are currently used only for input files. The possible flags within the flags field are these:

DATA_CHANNEL_SQUELCHED
Indicates that the channel is squelched: nothing on this channel should be output.
DATA_CHANNEL_DECL
Indicates that anything on this channel is a declaration.
DATA_CHANNEL_IMPL
Indicates that anything on this channel is a definition or implementation.

The purpose of a channel is to describe a data path for some object, thus allowing Flick to classify it and determine its origin and destination. This knowledge is primarily useful for squelching idl definitions that should not appear in the output. For example, a Flick back end may need to suppress definitions and/or declarations that arise from idl system header files: the definitions of system objects may be built into the targeted rpc/rmi runtime library. Similarly, a user might ask Flick to suppress definitions that come from #included idl files, but to keep the corresponding declarations. This is useful when the included idl files are intended to be compiled separately.

Similar to io_files, channels can be selected with the help of a mask structure:
 struct data_channel_mask {
         unsigned int    mask_flags;     /* Flags for the mask */
         io_file_mask    *input;         /* Input file mask */
         string          id<>;           /* ID to match on */
         unsigned int    set_flags;      /*
                                          * Match when all of these flags are
                                          * set in the io_file
                                          */
         unsigned int    unset_flags;    /*
                                          * Match when all of these flags aren't
                                          * set in the io_file
                                          */
 };

3.3 The META Library

The meta library contains a relatively small number of functions of creating and manipulating meta data structures. The source code for the library is in the libmeta directory, and the library header file is mom/libmeta.h. The various meta library functions are summarized below.

void init_meta(meta *m)
Initialize the given meta structure. The structure is set to contain no files or channels.
void check_meta(meta *m)
Check that the given meta structure is well-formed. This function uses assert to verify the contents of the structure.
void print_meta(meta *m, FILE *file, int indent)
Pretty-print the given meta structure to the given file, using the specified initial indentation value.

The following functions are useful for manipulating files:

io_file_index meta_add_file(meta *m, const char *id, int flags)
If the given meta contains an io_file with the specified id and flags, return the index of that file. Otherwise, create a new io_file within m, initialize it, and return the index of the newly created file.
io_file_index meta_find_file(meta *m, const char *id, int flags, int absolute_path)
Find a file in the given meta structure. If id is null, then match only the flags; similarly, if flags is zero, match only on id. The absolute_path argument determines whether id is an absolute file name or just the file part (i.e., non-directory part) of the name. Return the index of the located io_file or -1 if no such file exists.
void meta_include_file(meta *m, io_file_index file, io_file_index included_file)
Record the fact that the file at index file includes the file at index included_file.
void meta_check_file(meta *m, io_file_index file)
Check the integrity of the io_file at the given index in m. Errors are reported as assert failures (i.e., core dumps).
void meta_print_file(meta *m, FILE *file, int indent, io_file_index idx)
Pretty-print the io_file at the given index. The initial indentation level is specified by indent.
io_file_mask meta_make_file_mask(int tag, ...)
Create and return an io_file_mask structure, initialized according to the tag list given in the function arguments. (Tag list functions are described in Section 2.3.4.) A mask defines a match set of io_file patterns; a particular io_file can be compared to an io_file_mask to determine if the io_file meets the criteria specified by the mask, and is therefore a member of the mask's match set.

The tags and associated tag values for the meta_make_file_mask function are these:

FMA_TAG_DONE
(No required tag values.) Marks the end of the tag list.
FMA_MatchesID
(Required tag values: char *.) Specifies that the match set should include io_files whose complete file names match the given string. Matching is done with strcmp (i.e., exact string matching).
FMA_ExcludesID
(Required tag values: char *.) Specifies that the match set should exclude io_files whose complete file names match the given string.
FMA_MatchesDirID
(Required tag values: char *.) Specifies that the match set should include io_files containing file names with directory portions that match the given string.
FMA_ExcludesDirID
(Required tag values: char *.) Specifies that the match set should exclude io_files containing file names with directory portions that match the given string.
FMA_MatchesFileID
(Required tag values: char *.) Specifies that the match set should include io_files containing file names with file (non-directory) portions that match the given string.
FMA_ExcludesFileID
(Required tag values: char *.) Specifies that the match set should exclude io_files containing file names with file (non-directory) portions that match the given string.
FMA_SetFlags
(Required tag values: unsigned int.) Specifies that the match set should include io_files in which the specified flags are set. Additional flags may be set in the io_file, but only the specified flags are required to be set.
FMA_UnsetFlags
(Required tag values: unsigned int.) Specifies that the match set should include io_files in which the specified flags are unset.

An io_file_mask can specify at most one file name matching function: in other words, do not specify more than one of the above "ID" tags when creating a mask. In contrast, an io_file_mask can specify matching against both set and unset flags.

int meta_match_file_mask(meta *m, io_file_mask *ifm, io_file_index file)
Compare the specified io_file (i.e., the file at index file in m) against the given io_file_mask. Return a non-zero value if the file matches the mask -- meaning that the file is a member of the mask's match set -- and zero if it does not match.
void meta_squelch_file(meta *m, io_file_index file, data_channel_mask *dcm)
Squelch a set of data channels that come from the specified file and from all files that are included by that file. The squelched channels are those that match dcm. When a channel is "squelched," Flick will not produce output from data structures that refer to that channel.
void meta_squelch_files(meta *m, io_file_mask *ifm, data_channel_mask *dcm)
Apply meta_squelch_file to all of the files in m that match ifm. The squelched channels are those that match dcm.
void meta_print_file_mask(FILE *file, int indent, io_file_mask *ifm)
Pretty-print the given io_file_mask.

The following functions are useful for manipulating files:

data_channel_index meta_add_channel(meta *m, io_file_index input, const char *id)
Create a new data_channel within m. The new channel has the given id and refers to the indicated io_file. Return the index of the newly created channel.
data_channel_index meta_find_channel(meta *m, io_file_index input, const char *id, int  flags)
Find a channel in the given meta structure with the given attributes, and return the index of the located channel. If no such channel exists, return -1.
void meta_add_channel_output(meta *m, data_channel_index channel, io_file_index output)
Add an output file to the specified channel. Because io_ file s are currently used only for input files, this function is unused.
void meta_check_channel(meta *m, data_channel_index channel)
Check the integrity of the data_channel at the given index in m. Errors are reported as assert failures (i.e., core dumps).
void meta_print_channel(meta *m, FILE *file, int indent, data_channel_index channel)
Pretty-print the data_channel at the given index. The initial indentation level is specified by indent.
data_channel_mask meta_make_channel_mask(int tag, ...)
Create and return a data_channel structure, initialized according to the tag list given in the function arguments. A mask defines a match set of data_channel patterns; a particular data_channel can be compared to a data_channel_mask to determine if the data_channel meets the criteria specified by the mask, and is therefore a member of the mask's match set.

The tags and associated tag values for the meta_make_channel_mask function are these:

CMA_TAG_DONE
(No required tag values.) Marks the end of the tag list.
CMA_MatchesInput
(Required tag values: io_file_mask.) Specifies that the match set should include data_channels that refer to input files matching the specified io_file_mask.
CMA_ExcludesInput
(Required tag values: io_file_mask.) Specifies that the match set should exclude data_channels that refer to input files matching the specified io_file_mask.
CMA_MatchesID
(Required tag values: char *.) Specifies that the match set should include data_channels whose id field matches the given string. Matching is done with strcmp (i.e., exact string matching).
CMA_ExcludesID
(Required tag values: char *.) Specifies that the match set should exclude data_channels whose id field matches the given string.
CMA_SetFlags
(Required tag values: unsigned int.) Specifies that the match set should include data_channels in which the specified flags are set. Additional flags may be set in the data_channel, but only the specified flags are required to be set.
CMA_UnsetFlags
(Required tag values: unsigned int.) Specifies that the match set should include data_channels in which the specified flags are unset.

int meta_match_channel_mask(meta *m, data_channel_mask *dcm, data_channel_index channel)
Compare the specified data_channel (i.e., the channel at index channel in m) against the given data_channel_mask. Return a non-zero value if the channel matches the mask -- i.e., is in the mask's match set -- and zero if it does not.
void meta_squelch_channel(meta *m, data_channel_index channel)
Mark the indicated channel as squelched. When a channel is "squelched," Flick will not produce output from data structures that refer to that channel.
void meta_squelch_channels(meta *m, data_channel_mask mask)
Squelch all of the channels in m that match the given data_channel_mask.
void meta_print_channel_mask(FILE *file, int indent, data_channel_mask dcm)
Pretty-print the given data_channel_mask.

3.4 Summary and Comments

In retrospect, it is apparent that the concept of a data_channel could be generalized to encompass the abilities of an io_file structure. Currently, the only real difference between the two structures is that a file can have multiple inputs -- the included files -- but that a channel can have only one. By allowing channels to have multiple inputs, Flick might be able to express the inclusion file graph as well as other dependencies between channels.