Apify

Apify runs on a highly scalable infrastructure that runs millions of web automation and web scraping jobs weekly to process way over a billion web pages every month. We run on a cluster of Linux servers on Amazon EC2 to process our workloads, multiple Kubernetes clusters to host our applications and services, and store data in MongoDB, DynamoDB, S3, Redis, and SQS. Our system is built with Node.js and many other technologies. We use Kubernetes, Helm, Github Actions, and CloudFormation to make our deployments smooth, and ship to production multiple times per day. We monitor using New Relic One, Sentry, LogDNA, and other services. We're passionate about delivering the best service to our customers using the best technology possible. Apify is made by developers for developers. We're building a product that we use ourselves every day and we are proud of. We love open-source and contribute to it–check out our Github. Help us make the Web more open and programmable!