NEXON DEV VINA
Project BC
A large-scale distributed crawling system with 800 worker instances.
- Field period
- Roles
- Developer · DBA · SRE
- Team
- 5 people, including 2 overseas teammates
1 yr 7 mos · Completed
What changed
The work, in outcomes.
Reduced deployment cost by 60% with fault-tolerant workloads on AWS ECS.
Autoscaled workers using custom AWS CloudWatch metrics.
Improved throughput and memory use with streaming and micro-batching.
Used Grafana, InfluxDB, and ad-hoc Redis and MySQL queries for on-call troubleshooting.
Maintained crawlers for YouTube, Bilibili, and Steam.
Built an ingestion pipeline with Snowflake and MySQL stored procedures.
Built a Redis queue with at-least-once delivery from scratch.
Written context
Related field notes
This project does not have a published field note yet. The project page will collect them here as they grow.
Nothing published yet
The field book is still open.
For now, the outcomes and technology above are the complete project record.