Writing Java APIs Using Apache Atlas Client

In the previous blog, Data Governance using Apache ATLAS we discussed the advantages and use cases of using Apache Atlas as a data governance tool. In continuation to it, we will be discussing building our own Java APIs, which can interact with Apache Atlas using Apache atlas client to create new entities and types in it.

How to Create New Entities and Types Using Atlas Client

Atlas Client Maven Dependency

The following dependencies can be used to update pom.xml:

<dependency>
 <groupId>org.apache.atlas</groupId>
 <artifactId>atlas-client-common</artifactId>
 <version>1.1.0</version>
</dependency>
<dependency>
 <groupId>org.apache.atlas</groupId>
 <artifactId>atlas-common</artifactId>
 <version>1.1.0</version>
</dependency>
<dependency>
 <groupId>org.apache.atlas</groupId>
 <artifactId>atlas-client-v1</artifactId>
 <version>1.1.0</version>
</dependency>

Setting up Atlas-application.properties

Apache Atlas Client uses atlas-application properties to establish a connection between our API and Apache Atlas server. The properties should be placed in resources/atlas-application.properties.

#########  Server Properties  #########
atlas.rest.address=http://localhost:21000

atlas.hook.demo.kafka.retries=1
atlas.kafka.zookeeper.connect=localhost:3181
atlas.kafka.bootstrap.servers=localhost:9092
atlas.kafka.zookeeper.session.timeout.ms=400
atlas.kafka.zookeeper.connection.timeout.ms=200
atlas.kafka.zookeeper.sync.time.ms=20
atlas.kafka.auto.commit.interval.ms=1000
atlas.kafka.hook.group.id=atlas

Creating a Connection to Atlas Server

To create a connection with Apache atlas Server, baseUrl and username, the password is to be passed in AtlasClient constructor,

public AtlasClient(String[] baseUrl, String[] basicAuthUserNamePassword) {
...
}

For example,

private final AtlasClient atlasClient;
AtlasEntityExample() {
   atlasClient = new AtlasClient(new String[]{"http://localhost:21000"}, new String[]{"admin", "admin"});
}

Listing Different Types Registered With Apache Atlas

private void listTypes() throws AtlasServiceException {
   System.out.println("Types registered with Atlas:");   List<String> types = atlasClient.listTypes();   for (String type : types) {       System.out.println("Type: " + type);   } }

The method on execution will provide an output like this,

Type: hive_principal_type
Type: AtlasGlossaryTermRelationshipStatus
Type: file_action
Type: AtlasGlossaryTermAssignmentStatus
Type: hive_order
Type: hive_serde
Type: fs_permissions
Type: DataSet
Type: Process
Type: hive_table
Type: avro_primitive
Type: storm_node
Type: Referenceable
...
...
...

Registering New Types in Apache Atlas

private void createTypes() throws Exception {
   TypesDef typesDef = createTypeDefinitions();

   String typesAsJSON = AtlasType.toV1Json(typesDef);   System.out.println("typesAsJSON = " + typesAsJSON);   atlasClient.createType(typesAsJSON); } TypesDef createTypeDefinitions() throws Exception {   ClassTypeDefinition dbClsDef = TypesUtil           .createClassTypeDef(DATABASE_TYPE, DATABASE_TYPE, null,                   TypesUtil.createUniqueRequiredAttrDef("name", AtlasBaseTypeDef.ATLAS_TYPE_STRING),                   attrDef("description", AtlasBaseTypeDef.ATLAS_TYPE_STRING), attrDef("locationUri", AtlasBaseTypeDef.ATLAS_TYPE_STRING),                   attrDef("owner", AtlasBaseTypeDef.ATLAS_TYPE_STRING), attrDef("createTime", AtlasBaseTypeDef.ATLAS_TYPE_LONG));         TraitTypeDefinition jdbcTraitDef = TypesUtil.createTraitTypeDef("JdbcAccess_v11", "JdbcAccess Trait", null);     return new TypesDef(Collections.<EnumTypeDefinition>emptyList(), Collections.<StructTypeDefinition>emptyList(),           Arrays.asList(jdbcTraitDef),           Arrays.asList(dbClsDef)); } AttributeDefinition attrDef(String name, String dT) {   return attrDef(name, dT, Multiplicity.OPTIONAL, false, null); } AttributeDefinition attrDef(String name, String dT, Multiplicity m, boolean isComposite,                           String reverseAttributeName) {   Preconditions.checkNotNull(name);   Preconditions.checkNotNull(dT);   return new AttributeDefinition(name, dT, m, isComposite, reverseAttributeName); }

If the new type has been successfully created in Atlas Server, a success code of 200 will be rerun,

------------------------------------------------------
Call         : GET api/atlas/types
Content-type : application/json; charset=UTF-8 
[Accept       : application/json 
HTTP Status  : 200
------------------------------------------------------

Creating New Entity for a Registered Type

private String createEntity()
       throws AtlasServiceException {

   Referenceable referenceable = new Referenceable("DB_v11");
   referenceable.set("name", "testt-db-v1");
   referenceable.set("description", "desc");
   referenceable.set("owner", "himani");
   referenceable.set("locationUri", "test-db-v2");

   String entityJSON = AtlasType.toV1Json(referenceable);   System.out.println("Submitting new entity= " + entityJSON);   List<String> entitiesCreated = atlasClient.createEntity(entityJSON);     return entitiesCreated.get(entitiesCreated.size() - 1); }

A unique GUID of the entity will be returned if it has been successfully created in the Server. For example,

------------------------------------------------------
Call         : POST api/atlas/entities
Content-type : application/json; charset=UTF-8 
Accept       : application/json 
Request      : [{"typeName":"DB_v11","values":{"owner":"himani","name":"testt-schema-v2","description":"desc","locationUri":"test-schema-v2"},"id":{"id":"-4336797886256","typeName":"DB_v11","version":0,"state":"ACTIVE","jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id"},"traits":{},"traitNames":[],"systemAttributes":null,"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference"}]
HTTP Status  : 201
Response     : {"entities":{"created":["20c8b710-eed5-4512-beda-d9fcf08f4bb7"]},"requestId":"..."} ------------------------------------------------------

Deleting Entities From Atlas Server Using GUIDs

The following code snippet can be used to delete existing entities from Atlas Server:

private List<String> deleteEntity(final String... guids) throws AtlasServiceException {
   return atlasClient.deleteEntities(guids).getDeletedEntities();
}

The complete example code for creating a CRUD Java API to interact with Apache Atlas Server can be found here.

References

• https://github.com/knoldus/atlas-java-crud
• https://blog.knoldus.com/apache-atlas/
• https://github.com/apache/atlas

This article was first published on the Knoldus blog.