This content originally appeared on Level Up Coding - Medium and was authored by Hayk Simonyan
Object Storage (BLOBs) Explained for System Design
What is Object Storage (BLOB)?
Object Storage is a special type of storage designed to handle massive amounts of unstructured data.
If you’re designing for the cloud or dealing with massive datasets, then you’ll need to use some sort of object storage there.
These are the services like:
- Amazon S3— Probably the most popular one, S3 is a Simple Storage Service that is used for storing large files and retrieving them quickly.
- Google Cloud Storage — Google Cloud offers its object storage service called Google Cloud Storage.
- Azure Blob Storage — Azure also provides a service called Azure Blob Storage.
As you can see, some services, like Azure, also call it BLOB storage, which stands for Binary Large Object.
But What is an Object?
So, what’s an object, exactly? It’s just a chunk of data. It could be an image, a video, or even an entire virtual machine.
And each object comes with a few essential bits of information:
- Data: The actual binary content that you’re storing.
- Unique Identifier: The object’s address within the storage system.
- Metadata: Information about the object (filename, type, size, creation date, etc.)
Object Storage vs. Databases vs. File Systems
But why don’t we use our database to store these large files?
Well, we still use our database to store the URI and some more metadata about these image or video files, but the actual file is stored in Object Storage.
If you throw a bunch of huge files into your database, that will raise performance issues, and our queries will become slower to execute.
And what about File Systems? Can we store these images and videos in some file storage like Google Drive or OneDrive, or what if we store them in our server's file system and read them from there?
File systems are great for local storage and organization, but they don’t scale as easily as object storage.
Why? — Because file systems have a hierarchical structure, meaning they are structured like trees with folders, whereas object storage is flat and unstructured — which gives them the advantage of being highly scalable and fast for file retrievals.
Folders in BLOB Storage?
So, does this mean that there are no folders in object storage?
Many platforms like AWS S3 offer a “folder-like” experience by using delimiters in object names (e.g., “photos/vacation/beach.jpg”) — meaning they split the path to object with forward slashes and represent them as folders.
But this is just an illusion to make it easier to use—technically, there are no folders in object storage.
Unique Identifiers
As we saw, each object gets a unique identifier (think of a bucket as a container for objects). It’s usually unique within the bucket (your object’s container) and the specific storage service, like in S3.
However, since bucket names must be unique across all AWS accounts, your object names are unique globally, not just inside your bucket.
Interacting with Object Storage
Most object stores offer a RESTful API for interacting with objects. You send HTTP requests (PUT, GET, DELETE) to manipulate data.
Uploading an object
For instance, you can upload objects to S3 storage using a POST request or by using the AWS platform directly.
Retrieving an object
Similarly, you can read your uploaded objects with a GET request or by using the AWS console.
Updating an object
However, editing is typically done by replacing the entire object with a new version. This is different from file systems, where you can modify parts of a file in place.
To edit a file in object storage, you typically remove it with a DELETE request and then create a new updated one with a POST request.
In addition, there are many libraries and SDKs that make this process easier in different programming languages.
Summary
To conclude, object storage is designed for speed — it quickly locates objects based on their identifiers.
In addition to that, many object storages use a distributed architecture to distribute traffic across multiple storage nodes.
Frequently accessed objects are stored in faster memory (cache) for even quicker retrieval.
Many object storage platforms offer security features such as signed URLs and signed cookies to control access to your objects. If you want to learn more about how these features work, check out my upcoming posts.
More Free Tutorials like this at Web Dev Mastery Community.
Object Storage (BLOBs) in System Design was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Hayk Simonyan
Hayk Simonyan | Sciencx (2024-07-02T22:55:05+00:00) Object Storage (BLOBs) in System Design. Retrieved from https://www.scien.cx/2024/07/02/object-storage-blobs-in-system-design/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.