Has anyone tried running Singer on AWS Lambda? A user in the Singer Slack is trying to find a serverless cloud solution to run Singer on.
I’m looking into this soon, currently using spot instances. Anything new?
I recognise this is a an old post. It’s a question I’m interested in too.
I’ve spent some time using AWS Glue Python Shell jobs for some ETL work. It’s possible that the shell jobs might work - they have a built in scheduler.
I know it’s a super delayed reponse, but here is my experience I can share in case someone lands on this thread by search: I had implemented both AWS Lambda and Fargate for a production use. I would not recommend Lambda unless your singer process completes in a few minutes because Lambda has 15min time limit. Now for Fargate, I just open-soured this new project called handoff. It is a general Unix pipeline deployment but the original usecase is singer tap/target execution. It would be great if you can take a look https://pypi.org/project/handoff so I can improve. Doc has singer examples, too: https://dev.handoff.cloud Thanks!