2smo Data Crawler

2smo Data Crawler is a client data collection tool for configured API sources. It fetches user and payment records, adds location and payment details when available, and creates Excel workbooks for review. An Express dashboard lets an operator configure sources and run or stop a crawl.

The problem

Records arrive from separate paginated API endpoints and need to be compared across sources. Payment entries may need gateway fee details, while user IPs need location labels. Loading everything into one response is unnecessary for a long crawl. This project stores fetched pages in SQLite, processes the records, and exports joined views.

Key features

• Fetch paginated user and payment endpoints for enabled sources.
• Start or cancel a crawl from a dashboard, including a selected source.
• Show progress and logs in the browser through Socket.IO.
• Store fetched records in SQLite as pages arrive.
• Enrich user locations with a local MaxMind database and cached lookups.
• Look up payment fee information through Stripe or MyFatoorah when configured.
• Deduplicate users and join user records to payments for export.
• Generate Excel workbooks with source and merged worksheets.
• Schedule crawl runs with node-cron.

Technical architecture

A Node.js pipeline loads source configuration, calls the adapter through Axios, and passes each page to SQLite. The processing layer reads stored users and payments, attaches optional geography and gateway data, then passes export rows to SheetJS. Express and EJS provide login, dashboard, data, settings, and log views. Socket.IO carries progress updates. A scheduler starts configured runs. Site and gateway API keys can be stored encrypted with AES-256-GCM in the SQLite secrets table.

Engineering highlights

• AbortController signals reach Axios, so stopping a crawl can interrupt an active request.
• Geolocation uses an in-memory cache and a persistent SQLite cache before consulting the local MaxMind file.
• The processing layer deduplicates users and removes email and IP fields from export rows.
• Payment enrichment tries configured gateway lookups and keeps the crawl running when a memo cannot be matched.

My role

I built the client crawler, storage and enrichment pipeline, Excel workbook output, and operator dashboard present in the repository.