This content originally appeared on DEV Community and was authored by Ian Macartney
JSON Lines is a file format that stores one JSON object per line in a file. This is more scalable than storing it as a JSON array, since most JSON parsers require loading the full stringified array into memory before it can parse it. With JSON Lines, your code can load one line at a time. If you don't hold onto each object (for instance, you do some operation and save it elsewhere), your code can theoretically read an infinitely long file without running out of memory. It also allows you to easily read a stream of data.
To make these files, each object needs to be on its own line. and the file extension is .json*l*
For Convex import, we allow you to specify either format: a .json or .jsonl. However, we limit the .json size to 8MB. So one question that comes up is how can you transform your .json file into .jsonl?
Using jq to convert .json to .jsonl
jq is a common tool for working with JSON from the commandline.
Here's the one-liner to convert a .json file into .jsonl using jq:
jq -c '.[]' ./mydata.json > mydata.jsonl
Example
Input: ./mydata.json
[
{
"foo": "bar"
},
{
"easy_as": 123
}
]
Output: ./mydata.jsonl
{"foo":"bar"}
{"easy_as":123}
This content originally appeared on DEV Community and was authored by Ian Macartney
Ian Macartney | Sciencx (2024-07-24T00:05:57+00:00) Convert your .json array to a .jsonl (JSON Lines). Retrieved from https://www.scien.cx/2024/07/24/convert-your-json-array-to-a-jsonl-json-lines/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.