As a data analyst/business analyst, I have some very basic data engineering experience (e.g. I set up our analytics warehouse, maintain the database models, write Python scripts on my laptop to ETL some Google sheets/Google Forms datasets), but would love to figure out how to move into the “next level” and learn how to make proper “data products” that live on their own and that other business users can interact with easily.
For example, at my current company (Mac & Mia), I have a hacky way to calculate a user “style score” based on the photos a user likes on our website. I grab the user data from our production database, run it through a little Python script on my laptop and spit out the results into a table in our analytics warehouse. Then, I have a dbt model that runs on top of that data in our analytics warehouse to connect it to the rest of the user data which we can then see in Looker around 4 hours later (or more if I don’t have my laptop on to run the cron jobs). The delay has been acceptable for now but I would love to learn how to set up a little web server that can accept POST requests, do some calculations or transformations on the data and return it back to the requesting client for whenever the speed does become more mission critical.
I have what I believe to be a general grasp of the concept but start to get overwhelmed by the vast array of options (where do I host it, what’s the right API endpoint structure, do I need a message broker) I seem to have when I go into setting up a proper web app for it.
Anyone have recommendations for where to get started? Is this something that’s best learned on a place like Udemy?