This content originally appeared on DEV Community and was authored by Mark Kiehl
I released a series of seven free public articles on Medium.com “How to build a modern data platform on the free tier of Google Cloud Platform”. The lead article is available at: https://medium.com/@markwkiehl/building-a-data-platform-on-gcp-0427500f62e8
Part One “Building a Data Platform on GCP” defined the functional requirements, and detailed how to install the required software.
Part Two “GCP Infrastructure & Authentication” explained how to use Google application default credentials (ADC) to authenticate a user-managed service account.
Part Three “Google Cloud Pub/Sub Messaging” showed how to use a Python script to generate and subscribe to the Google Pub/Sub Messaging service.
Part Four “Containerization using Docker” covered how to build a local Docker image for a Python script, run it locally, and then push it to Google Artifact Registry (repository).
Part Five “Google Cloud Run Jobs & Scheduler” demonstrated how to configure Google Cloud Run Jobs and Cloud Scheduler Jobs using Google CLI to execute a Python script stored in Google Artifact Registry on a specified interval from any Google region.
Part Six “Google BigQuery Cloud Database” set up a Google BigQuery dataset and table using the Google CLI, and then a Python script was used to write and query data with SQL.
Part Seven “Google Cloud Analytics” explored how to extract data from a Google BigQuery table, load it into a Pandas DataFrame, and effortlessly perform analysis and visualizations — all from a Python script.
This content originally appeared on DEV Community and was authored by Mark Kiehl
Mark Kiehl | Sciencx (2024-10-05T13:46:05+00:00) How to build a modern data platform on the free tier of Google Cloud Platform. Retrieved from https://www.scien.cx/2024/10/05/how-to-build-a-modern-data-platform-on-the-free-tier-of-google-cloud-platform/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.