...
|
Design the connector
Before delving straight into the implementation of a connector, you really need to start with a level of design work. Fundamentally this will involve two steps:
|
Map to the Egeria meta-model concepts
The best place to start with the design work is to understand the meta-model of Egeria itself. Consider how your metadata repository will map to the fundamental Egeria metadata concepts: Entities, Classifications, and Relationships.
...
For example, in the implementation of the sample IGC connector we suggest using categories with specific names in IGC to represent certain classifications. Additionally, one of the reasons for implementing a read-only connector is that we can still handle relationships without any properties: by simply having the properties of any Egeria relationships we translate from IGC all be empty.
Map to the Egeria open metadata types
Once you have some idea for how to handle the mapping to the meta-model concepts, check your thinking by working through a few examples. Pick a few of the open metadata types and work out on paper how they map to your metadata repository’s pre-existing model. Common areas to do this would be e.g. GlossaryTerm
, GlossaryCategory
for glossary (business vocabulary) content; RelationalColumn
, etc for relational database structures; and so on.
...
- Ignore any open metadata types that do not map to your pre-existing types.
- Add any Egeria open metadata types that do not exist in your repository.
- Add Egeria open metadata properties to your pre-existing types when Egeria has additional properties that do not yet exist in your type(s).
- Implement a read-only connection (possibly with some hard-coding of property values) for types that are partially map-able, but not easily extended to support the full set of properties defined on the open metadata type.
- and so on.
Setup the connector project
Implementing an adapter can be greatly accelerated by using the pre-built base classes of Egeria. Therefore building a connector using Java is likely the easiest way to start. This requires an appropriate build environment comprised of both Java (minimally v1.8) and Maven. Egeria has been designed to allow connectors to be developed in projects independently from the core itself. Some examples have already been implemented, which could provide a useful reference point as you proceed through this walkthrough:
|
...
Naturally change the version to whichever version of Egeria you'd like to build against. The dependencies listed ensure you'l have the necessary portion of Egeria to build your connector against.
Implement the repository connector
The repository connector exposes the ability to search, query, create, update and delete metadata in an existing metadata repository. As such, it will be the core of your adapter. You can start to build this within your new project by creating a new Maven module called something like |
Within this adapter
module implement the following:
Implement an OMRSRepositoryConnectorProvider
Start by writing an OMRSRepositoryConnectorProvider
specific to your connector, which extends OMRSRepositoryConnectorProviderBase
. The connector provider is a factory for its corresponding connector. Much of the logic needed is coded in the base class, and therefore your implementation really only involves defining the connector class and setting this in the constructor.
...
Note that you'll need to define a unique GUID for the connector type, and a meaningful name and description. Really all you then need to implement is the constructor, which can largely be a copy / paste for most adapter
s. Just remember to change the connectorClass
to your own, which you'll implement in the next step (below).
Implement an OMRSRepositoryConnector
Next, write an OMRSRepositoryConnector
specific to your connector, which extends OMRSRepositoryConnector
. This defines the logic to connect to and disconnect from your metadata repository. As such the main logic of this class will be implemented by:
...
You may want to wrap the metadata repository client’s methods with your own methods in this class as well. Generally think of this class as “speaking the language” of your proprietary metadata repository, while the next class “speaks” Egeria.
Implement an OMRSMetadataCollection
Finally, write an OMRSMetadataCollection
specific to your repository, which extends OMRSMetadataCollectionBase
. This can grow to be quite a large class, with many methods, but is essential for the participation of your metadata repository in a broader cohort. In particular, it is heavily leveraged by Egeria’s Enterprise Connector to federate actions against your metadata repository. As such, this is how your connector “speaks” Egeria (open metadata).
...
- a connection to be instantiated to your repository, and
- translation between your repository’s representation of metadata and the open metadata standard types.
Package your connector
To make your connector available to run within the OMAG Server Platform, you can package it into a distributable In this module’s POM file include your adapter module (by |
...
Of course, if you intend to embed or otherwise implement your own server, the packaging mechanism will likely be different. However, as mentioned in the previous step this should provide a quick and easy initial way of testing the functionality of the connector against the core of Egeria.
Startup the OMAG Server Platform with your connector
Assuming you’ve built your connector .jar file using the approach outlined above, you’ll now have a .jar file under the distribution/target/ directory of your project: for the Apache Atlas example, this would be distribution/target/egeria-connector-apache-atlas-package-1.1-SNAPSHOT.jar . |
...
Start with just the following:
Enable the OMAG Server as a repository proxy
Enable the OMAG Server as a repository proxy by specifying your canonical OMRSRepositoryConnectorProvider
class name for the connectorProvider={javaClassName}
parameter and POST
ing to:
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "class": "Connection", "connectorType": { "class": "ConnectorType", "connectorProviderClassName": "org.odpi.egeria.connectors.apache.atlas.repositoryconnector.ApacheAtlasOMRSRepositoryConnectorProvider" }, "endpoint": { "class": "Endpoint", "address": "{{atlas_host}}:{{atlas_port}}", "protocol": "http" }, "userId": "{{atlas_user}}", "clearPassword": "{{atlas_password}}" } |
Start the server instance
Start the OMAG Server instance by POST
ing to:
...
During server startup you should then see various messages related to the metadata type registration process as the open metadata types are checked against your repository. (These in turn call the methods you’ve implemented in your OMRSMetadataCollection
.) You might naturally need to iron out a few bugs in those methods before proceeding further…
Test your connector's basic operations
Each time you change your connector code, you’ll naturally want to re-build it (
From there you can continue to override other methods of the |
A logical order of implementation might be:
Read operations
getEntitySummary()
... which you can test through GET
to:
Code Block |
---|
http://egeriahost:8080/servers/test/open-metadata/repository-services/users/myself/instances/entity/{{guidOfEntity}}/summary |
getEntityDetail()
... which you can test through GET
to:
Code Block |
---|
http://egeriahost:8080/servers/test/open-metadata/repository-services/users/myself/instances/entity/{{guidOfEntity}} |
getRelationshipsForEntity()
... which you can test through POST
to:
...
These are likely to require the most significant logic for any mappings / translations you’re doing between the open metadata types and your own repository. For example, with Apache Atlas these are where we translate between the Apache Atlas native types like AtlasGlossaryTerm
and its representation in the Apache Atlas java client and the open metadata type GlossaryTerm
and its representation through the standard OMRS interfaces.
Search operations
The other main area to then implement is searching, for example:
findEntitiesByProperty()
... which you can test through POST
to:
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "class": "EntityPropertyFindRequest", "typeGUID": "0db3e6ec-f5ef-4d75-ae38-b7ee6fd6ec0a", "pageSize": 10, "matchCriteria": "ALL", "matchProperties": { "class": "InstanceProperties", "instanceProperties": { "displayName": { "class": "PrimitivePropertyValue", "instancePropertyCategory": "PRIMITIVE", "primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING", "primitiveValue": "\\QEmpl\\E.*" } } }, "limitResultsByClassification": [ "SpineObject" ] } |
findEntitiesByClassification()
... which you can test through POST
to:
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "class": "EntityPropertyFindRequest", "typeGUID": "0db3e6ec-f5ef-4d75-ae38-b7ee6fd6ec0a", "pageSize": 100, "matchClassificationCriteria": "ALL", "matchClassificationProperties": { "class": "InstanceProperties", "instanceProperties": { "scope": { "class": "PrimitivePropertyValue", "instancePropertyCategory": "PRIMITIVE", "primitiveDefCategory": "OM_PRIMITIVE_TYPE_STRING", "primitiveValue": "*local*" } } } } |
findEntitiesByPropertyValue()
... which you can test through POST
to:
...
Once you have those working, it should be relatively easy to go back and fill in areas like the other TypeDef-related methods, to ensure your connector can participate appropriately in a broader open metadata cohort.
Write operations
While the ordering above is necessary for all connectors, if you’ve decided to also implement write operations for your repository there are further methods to override. These include:
...
If you are only implementing a read-only connector, these methods can be left as-is and the base class will indicate they are not supported by your connector.
Add an event mapper
The event mapper connector enables events from an existing metadata repository to distribute changes to metadata to the rest of the metadata repositories who are members of the same OMRS cohort. It is not a mandatory component: as long as your connector can “speak” Egeria through an OMRSMetadataCollection it can participate in an open metadata cohort via the Enterprise Connector. However, if your metadata repository already has some kind of event or notification mechanism, the event mapper can be an efficient addition to participating in the broader open metadata cohort. |
Within the same adapter
Maven module, perhaps under a new sub-package like ...eventmapper
, implement the following:
Implement an OMRSRepositoryEventMapperProvider
Start by writing an OMRSRepositoryEventMapperProvider
specific to your connector, which extends OMRSRepositoryConnectorProviderBase
. The connector provider is a factory for its corresponding connector. Much of the logic needed is coded in the base class, and therefore your implementation really only involves defining the connector class and setting this in the constructor.
...
Note that you’ll need to define a unique GUID for the connector type, and a meaningful name and description. Really all you then need to implement is the constructor, which can largely be a copy / paste for most adapters. Just remember to change the connectorClass
to your own, which you’ll implement in the next step (below).
Implement an OMRSRepositoryEventMapper
Next, write an OMRSRepositoryEventMapper
specific to your connector, which extends OMRSRepositoryEventMapperBase
and implements VirtualConnectorExtension
and OpenMetadataTopicListener
. This defines the logic to pickup and process events or notifications from your repository and produce corresponding OMRS events. As such the main logic of this class will be implemented by:
...
To add the event mapper configuration to the OMAG Server Platform configuration you started with above, add:
Configure the cohort event bus
This should be done first, before any of the other configuration steps above, by POST
ing to:
...
Code Block | ||||
---|---|---|---|---|
| ||||
{ "producer": { "bootstrap.servers":"kafkahost:9092" }, "consumer": { "bootstrap.servers":"kafkahost:9092" } } |
Configure the event mapper
This can be done nearly last, after all of the other configuration steps above but still before the start of the server instance. Specify your canonical OMRSRepositoryEventMapperProvider
class name for the connectorProvider={javaClassName}
parameter and connection details to your repository’s event source in the eventSource
parameter by POST
ing to:
...
Code Block |
---|
http://egeriahost:8080/open-metadata/admin-services/users/myself/servers/test/local-repository/event-mapper-details?connectorProvider=org.odpi.egeria.connectors.apache.atlas.eventmapper.ApacheAtlasOMRSRepositoryEventMapperProvider&eventSource=atlashost:9027 |
Test your connector's conformance
Aside from the API-based testing you might do as part of the on-going implementation of your This will provide guidance on what features you may still need to implement in order to conform to the open metadata standards, as well as ensuring those you have already implemented behave appropriately to participate in an open metadata cohort. Once your connector conforms, you should also then have the necessary output to apply to use the ODPi Egeria Conformant mark. |
...