DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • From Data Lakes to Intelligence Lakes: Augmenting Apache Iceberg With Generative AI Metadata on AWS
  • Real-Object Detection at the Edge: AWS IoT Greengrass and YOLOv5
  • Managing AWS Managed Microsoft Active Directory Objects With AWS Lambda Functions
  • Configuring Amazon S3 Using Mulesoft

Trending

  • How (and Why) to Build Dynamic Data Pipelines
  • Getting Started With Milvus: Vector Search for Enterprise Java Applications
  • When to Use AMQP, JMS, Kafka, or MQTT: Trade-Offs, Not a Winner
  • Best of Three Worlds: AI Agents Writing In Rust and Java Beat the C Library

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Data Engineering
  3. Databases
  4. How To: Set and Retrieve Metadata of an S3 Object

How To: Set and Retrieve Metadata of an S3 Object

By 
Vijaykishan Shyamsundar
·
Jan. 14, 20 · Tutorial
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
33.7K Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

In the article we will see how to upload the metadata of an object while uploading to S3 programatically.

I will be using Node.js to demonstrate the same.

Before jumping to that let’s see what is metadata and it’s use case

We have 2 types of metadata

  1. System Metadata    E.g. Date, Content-Length
  2. User defined metadata

While we upload the object, we can add custom metadata about the object along that as a key-value pair

 

Note: All user-defined metadata keys in lowercase

 

Let’s say we are having a platform where contents are added by multiple organization. In this scenario adding OrganizationID as one of the metadata will be handy.

Following code helps you to upload an object to s3 along it’s metadata.

JavaScript




x
43


 
1
var AWS = require('aws-sdk')
2
var credentials = require('./cred');
3
var s3 = new AWS.S3({accessKeyId: credentials.accesskey,
4
    secretAccessKey: credentials.secretkey,region:credentials.region});
5
var fs = require('fs');
6
 
          
7
fs.readFile('file2.txt', function (err, data) {
8
    if (err) { throw err; }
9
    var base64data = new Buffer(data, 'binary');
10
    s3.putObject({
11
        Bucket: 'uploadmetatest',
12
        Key: 'vijaykishan.txt',
13
        Metadata: {OrgID: 'TEKKIHUT', PROFILE: 'HD'},
14
        Body: base64data,
15
        ACL: 'public-read'
16
    }, function (resp) {
17
        console.log('Successfully uploaded package.');
18
    });
19
});


 

Let’s see how to retrieve the metadata of an uploaded object using node.js.

Following code helps you to retrieve the metadata of an object uploaded to s3.

JavaScript
xxxxxxxxxx
1
25
 
1
var params = {
2
    Bucket: "uploadmetatest",
3
    Key: "file2.txt"
4
};
5
s3.headObject(params, function (err, data) {    if (err) console.log(err, err.stack); // an error occurred
6
    else {
7
        console.log(data.Metadata['orgid']);
8
        console.log(data.Metadata['profile']);
9
   }       
10
});

 

Note

The PUT request header is limited to 8 KB in size. Within the PUT request header, the user-defined metadata is limited to 2 KB in size. The size of user-defined metadata is measured by taking the sum of the number of bytes in the UTF-8 encoding of each key and value.

 

Hope this helps

Metadata AWS Object (computer science)

Opinions expressed by DZone contributors are their own.

Related

  • From Data Lakes to Intelligence Lakes: Augmenting Apache Iceberg With Generative AI Metadata on AWS
  • Real-Object Detection at the Edge: AWS IoT Greengrass and YOLOv5
  • Managing AWS Managed Microsoft Active Directory Objects With AWS Lambda Functions
  • Configuring Amazon S3 Using Mulesoft

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook