Data Platform Engineer

Deskripsi Pekerjaan

• Build and maintain reliable web/app crawlers, scrapers, and external API integrations across e-commerce, marketplace, retail, and other external sources.
• Analyze browser, mobile application, API, and network traffic to understand how data is delivered and design the appropriate acquisition approach and runtime, from direct HTTP/API extraction to browser automation.
• Build acquisition systems that remain reliable as external websites, applications, APIs, and data contracts evolve.
• Handle real-world source complexity such as sessions, tokens, pagination, dynamic content, rate limits, retries, concurrency, and partial failures.
• Build resilient and maintainable acquisition systems with appropriate testing, idempotency, deduplication, retries, safe reruns, data-quality validation, monitoring, and alerting.
• Operate acquisition workloads using appropriate workflow orchestration, compute, data storage/warehousing, and cloud infrastructure.
• Continuously improve reliability, scalability, maintainability, and total cost of ownership, including evaluating build-vs-buy approaches for external data.
• Explore new technologies and AI-assisted development or automation to improve how external-data pipelines are built, operated, and maintained.
• Collaborate with Pricing, Data Warehouse Engineering, Data Analysts, and other stakeholders to turn external-data needs into sustainable platform capabilities.

Job Requirements:

1. In order to launch successfully, you need to have:
2. Strong Python/software engineering fundamentals and experience building maintainable production systems.
3. Hands-on experience building or operating web scraping, crawling, external API, or similar data acquisition systems.
4. Strong understanding of HTTP, APIs, JSON, headers, cookies, sessions, authentication, pagination, and request/response behavior.
5. Hands-on experience with browser/network debugging and the ability to investigate how websites or applications communicate with their backend services.
6. Experience with both HTTP-based extraction and browser automation, using tools such as Requests/HTTPX, Selenium, Playwright, Scrapy, or equivalent.
7. Understanding of production reliability patterns such as retry, timeout, rate limiting, concurrency, idempotency, and failure recovery.
8. Solid understanding of SQL, data storage/warehousing, schema evolution, workflow orchestration, containerized workloads, and cloud infrastructure fundamentals.
9. Strong troubleshooting skills, especially when working with frequently changing external systems outside your direct control.