# HOWTO: Use Farrago SQL to Access Metadata Managed by MDR

If you are using MDR as a persistent repository, chances are good that at some point you'll want to expose your repository data as SQL views. This makes it possible to use an off-the-shelf query/reporting tool to browse your repository, generate reports, or just about anything else.

The Farrago DBMS platform makes all of this possible via its SQL/MED extensibility feature. This document describes how to make use of the MDR plugin.

## Getting Started with Farrago

The first step is to get a Farrago build. See the developer jump-start page for details, or download prebuilt binaries. For very simple queries (e.g. filters, projections, and limited joins) you don't need the Fennel C++ components; pure Java can do the job. But if you want more comprehensive SQL support (e.g. ORDER BY), you'll need a build with Fennel.

After you have installed Farrago, make sure you're able to execute simple queries via iSQL-Viewer.

## Farrago Talks MDR

For an example MDR repository, we will use a test btree repository which is generated by the Farrago build. (If you're using prebuilt binaries, this isn't part of the distribution, but you can get it from the Farrago-only source download and change the unitsql path below accordingly.) This repository contains just the MOF metametamodel. The first step is to tell Farrago where to find this repository. This is analogous to mounting a device in Unix in order to make it accessible via the filesystem. Execute the following command from iSQL-Viewer:

    create server mof_repository
    foreign data wrapper sys_mdr
    options(
        "org.netbeans.mdr.persistence.Dir" 'unitsql/ddl/mdr',
        extent_name 'MOF',
        schema_name 'MODEL'
    );

Explanation:

- `mof_repository` is the name we will give the repository in Farrago. It will appear as a top-level catalog name.
- `sys_mdr` is the name of the foreign data wrapper used to access MDR. Farrago allows you to define and plug in your own, but this isn't necessary here because the system already provides one for MDR.
- `"org.netbeans.mdr.persistence.Dir"` tells MDR the directory in which to find the btree storage. This option name has to be quoted (with double-quotes as an identifier) in order for its case to be preserved correctly. Any properties understood by MDR can be passed in this way (allowing you to choose a storage representation other than btree, for example). The path is relative to the working directory; depending on where you ran iSQL-Viewer from, you may have to modify it accordingly. Using an absolute path is usually the best approach. Note that option values are always single-quoted (as string literals, not identifiers like option names).
- `extent_name` tells MDR what extent name to look for in the repository
- `schema_name` is bit of a hack. Normally, in a repository with a nested package structure, the `root_package_name` option would be specified instead, and all child packages of the root package would appear as schemas. However, the MOF repository has no nested packages at all, so in such cases the schema_name option should be specified to make the namespace conform to the standard catalog.schema.table convention. Choosing an upper-case name will allow you to avoid quoting it later.

Once the server definition above is created, you can directly query the repository extent if you know the available class names from the model. Let's get a list of all of the metaclasses defined by MOF:

    select * from mof_repository.model."Class";

NOTE: object names from MDR (like `"Class"`) are case sensitive and have to be quoted when they aren't upper-case to begin with. The reason `mof_repository` and `model` don't have to be quoted is that they were defined by Farrago DDL, not MDR. The results should look something like this:

Number of Records :: 28
name | annotation | container | isRoot | isLeaf | isAbstract | visibility | isSingleton | mofId | mofClassName
--- | --- | --- | --- | --- | --- | --- | --- | --- | ---
Tag |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:000000000000036B | Class
Constant |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000364 | Class
Constraint |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:000000000000035E | Class
Parameter |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000352 | Class
Import |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:000000000000034D | Class
Package |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000343 | Class
AssociationEnd |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:000000000000033E | Class
Association |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000331 | Class
Exception |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000328 | Class
Operation |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000322 | Class
BehavioralFeature |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:000000000000031B | Class
Reference |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000319 | Class
Attribute |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:000000000000030E | Class
StructuralFeature |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:000000000000030A | Class
Feature |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000306 | Class
AliasType |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000301 | Class
StructureField |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002FF | Class
StructureType |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002FC | Class
CollectionType |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002F9 | Class
EnumerationType |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002F6 | Class
PrimitiveType |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002F3 | Class
DataType |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002F1 | Class
Class |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | false | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002E2 | Class
Classifier |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002DB | Class
TypedElement |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002D9 | Class
GeneralizableElement |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002D4 | Class
Namespace |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002BD | Class
ModelElement |  | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394 | false | false | true | public_vis | false | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000002A1 | Class

Now we know the column names, so we can issue a more meaningful query, like finding the names of all abstract classes. Since the `isAbstract` column has type BOOLEAN, we can use it directly in the WHERE clause.

    select "name" from mof_repository.model."Class" where "isAbstract";

Expected results:

Number of Records :: 9
name
BehavioralFeature
StructuralFeature
Feature
DataType
Classifier
TypedElement
GeneralizableElement
Namespace
ModelElement

## Result Set Details

In the first result set above, notice the hexadecimal string ID's for the `container` and `mofId` columns. These are MOFID values, which are essentially object identifiers. The `mofId` attribute represents the ID for the object represented by the row in question (so in the above example, they are ID's for `Class` instances). The association between `Namespace` and `ModelElement` gives rise to the `container` column in the `Class` result set; it's really a foreign key referencing the `mofId` column of the `Namespace` result set. MDR takes care of the object-oriented aspects, so we can query for `Package` instances, which are a subset of all `Namespace` instances:

    select "name","mofId" from mof_repository.model."Package";

Expected results:

Number of Records :: 3
name | mofId
--- | ---
CorbaIdlTypes | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:00000000000003A3
Model | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000394
PrimitiveTypes | 7D749D32-73FA-11D8-9D44-AFBC9CB2AA77:0000000000000272

Notice that the `mofId` for the row with `name=Model` is the same as the `container` ID for all of the classes in the first result set. So this means all MOF classes are defined in the `Model` package. You can use a join to navigate the association; try the query below:

    select
        n."name" as container_name,
        e."name" as element_name
    from
        mof_repository.model."Package" n
    inner join
        mof_repository.model."ModelElement" e
    on n."mofId" = e."container";

Some other result set details:

- Besides the `mofId` column, there is an additional meta-column `mofClassName` associated with every table. When you query a leaf-level class in an object hierarchy, this value will always be the same (the name of the class you queried). However, when you query a non-leaf class (e.g. `select "name","mofClassName" from mof_repository.model."Namespace"`), the `mofClassName` column can be used to determine the actual leaf type of each object returned.
- Datatypes of result columns are inferred automatically using repository metadata. All columns show up as nullable (even when the underlying attribute is mandatory).
- MDR does not supply any precision information for string datatypes, so Farrago (arbitrarily) chooses VARCHAR(128) for the representation. This will probably change in the future; for now, a mechanism is available (described in the next section) for overriding the default with a different precision.

## Local Metadata

So far, we have been querying the repository directly using our knowledge of its underlying package structure. Normally, you will instead want to capture at least a subset of the repository's metadata in Farrago's own catalog so that it is available in standard format (JDBC metadata calls, INFORMATION_SCHEMA, etc.) for use in third-party query tools such as iSQL-Viewer (or in your own application). You will probably also want to define your own data dictionary views on top of the raw repository tables to transform the metadata into a more useful format. There are a number of ways to go about this:

1. Use the SQL/MED CREATE FOREIGN TABLE command to import metadata for individual tables into the Farrago catalog, specifying column names and types explicitly.
2. Use the SQL/MED CREATE FOREIGN TABLE command to import metadata for individual tables into the Farrago catalog, allowing the system to infer the column names and types.
3. Use the SQL/MED IMPORT FOREIGN SCHEMA command to import metadata for a collection of tables all at once. This is not yet implemented by Farrago, so it won't be discussed here.
4. Use the CREATE VIEW command to define views directly against the repository.
5. Use the CREATE VIEW command to define views against metadata imported via one of the first three methods.

Here's a CREATE FOREIGN TABLE example:

    create schema mof_schema;

    create foreign table mof_schema.exceptions(
        name varchar(20),
        annotation varchar(1),
        container char(54),
        "SCOPE" varchar(20),
        visibility varchar(20),
        exception_id char(54),
        unused_name varchar(20))
    server mof_repository
    options (class_name 'Exception');

Explanation:

- We first create a new schema `mof_schema` under the default LOCALDB catalog. This schema will contain our imported metadata. This imported metadata will remain available even if the repository becomes unavailable.
- In this case, the CREATE FOREIGN TABLE command overrides the default names and datatypes known to the repository. The names are defined as case insensitive, so no identifier quotes will be required when querying this table, except for the `scope` column whose name happens to be a SQL reserved word. And the datatypes are more reasonable than the defaults (e.g. we know that the MOFID's for this repository are fixed-length 54-character strings).
- The table name `exceptions` is never seen by the repository at all. Instead, we explicitly tell the repository the desired `class_name` via the options clause at the end.
- If the column names and types had been omitted, they would have been derived automatically from the repository metadata. This may be undesirable when you need control over case-sensitivity or datatypes.

Let's query the table via the local metadata:

    select name,visibility
    from mof_schema.exceptions;

Number of Records :: 2
NAME | VISIBILITY
--- | ---
NameNotResolved | public_vis
NameNotFound | public_vis

If this is the only information desired, we can create a view to capture it:

    create view mof_schema.exception_visibility as
    select name,visibility
    from mof_schema.exceptions;

The same view could instead be defined directly against the repository metadata:

    create view mof_schema.exception_visibility as
    select "name" as name,"visibility" as visibility
    from mof_repository.model."Exception";

Note that in this second view definition, the column types would be based on the repository metadata rather than on the overriding types specified when the table definition was imported.

## Try This At Home

If you made it through the above walkthrough, you're ready to give it a try with your own repository by following similar steps. There are a few things you should take into consideration first:

- Repository Storage: If you're not using the default btree storage, you'll need to specify different options to the `create server` command. For example, if you are using JDBC storage, you might specify something like:

      options(
          "org.netbeans.mdr.storagemodel.StorageFactoryClassName" = 'org.netbeans.mdr.persistence.jdbcimpl.JdbcStorageFactory',
          "MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.driverClassName" = 'org.hsqldb.jdbcDriver',
          "MDRStorageProperty.org.netbeans.mdr.persistence.jdbcimpl.url" = 'jdbc:hsqldb:/home/hsqldb/stored_repos'
      )

- Package Structure: If your repository model has no nested packages, you can use the `schema_name` option as in the MOF example. Otherwise, you'll need to specify the `root_package_name` option instead. All packages under this root package will appear as schemas under your repository catalog. If your package nesting structure has a depth greater than 2, you'll probably have to choose multiple roots and issue multiple `create server` statements accordingly.

- Concurrent Access: MDR's default btree storage does not allow more than one process at a time to open the btree files. This means Farrago and your application have to take turns accessing the data, which is probably impractical. MDR's JDBC storage may provide a better solution, although it needs improvements in performance/concurrency control. So for now, usage of Farrago for MDR querying is probably limited to proof-of-concept.

Let us know how it goes for you.

## TODO

Show how to use IMPORT FOREIGN SCHEMA once it is working, and also show metadata browsing via iSQL-Viewer once that is working.