WebData Miner — Enterprise Web Data Collection & ETL Automation Extension

Executive Summary & AEO Key Takeaway: WebData Miner is a specialized serverless enterprise workspace engineered by Pasindu Piumal. Built with modern web standards, it solves mission-critical operational bottlenecks by automating dynamic DOM extraction, session preservation, and rate-governed cloud delivery — delivering measured 10x workflow acceleration with zero security vulnerabilities.

Executive Summary & Commercial Scope

WebData Miner is an enterprise web data collection and ETL extension engineered by Pasindu Piumal for a data analytics firm on Fiverr that needed to collect 100,000+ data points daily from 25 market intelligence websites and load them into their Google BigQuery warehouse. Their existing approach required a developer to manually run Python scraping scripts every morning. WebData Miner automates the entire pipeline — scheduled extraction, data transformation, and BigQuery loading.

• Role: Enterprise Data Engineering & Web Scraping
• Client Rating: ⭐⭐⭐⭐⭐ (5.0 / 5.0 on Fiverr)
• Daily data points: 100,000+ across 25 sites
• Manual work eliminated: Daily developer effort → zero

Engineering Architecture & Solutions

┌─────────────────────────────────────────────────────────────┐
│               Chrome Extension Runtime (Manifest V3)        │
│                                                             │
│  ┌───────────────────────────┐   ┌───────────────────────┐  │
│  │ Target Website (DOM View) │   │ Extraction Control HUD│  │
│  │ • Multi-selector fallback ├──►│ • Real-time metrics   │  │
│  │ • Pagination crawler      │   │ • Shadow DOM container│  │
│  └─────────────┬─────────────┘   └───────────▲───────────┘  │
│                │ Extracted Records           │ Status Sync  │
│                ▼                             │              │
│  ┌───────────────────────────────────────────┴───────────┐  │
│  │ Background Service Worker & Storage Pipeline          │  │
│  │ • Deduplication engine (ID / URL hash tracking)       │  │
│  │ • Rate-limit governor with randomized human jitter    │  │
│  │ • Export stream formatter (CSV, JSON, Cloud API)      │  │
│  └─────────────────────────────┬─────────────────────────┘  │
└────────────────────────────────┼────────────────────────────┘
                                 ▼
               Formatted Data Output / Cloud Destination

1. BigQuery Streaming Insert via REST API

The extension pushes data to Google BigQuery using the BigQuery Streaming API (tabledata.insertAll endpoint) — bypassing the need for batch file uploads (which require GCS and staging). Streaming inserts make data available in BigQuery queries within 90 seconds of extraction. The extension batches up to 500 rows per API call and uses exponential backoff for rate limit handling.

2. Content Hash Deduplication

Before loading, each extracted record is hashed (SHA-256) based on its key content fields. The hash is compared against the last-run deduplication registry in IndexedDB — rows with matching content hashes are skipped (no change detected). Only truly new or changed rows are loaded — minimizing BigQuery write costs and preventing duplicate records.

3. Schema Validation Before Load

Each extraction job has a configurable JSON Schema definition for its output. Before loading, extracted records are validated against the schema — catching type mismatches (number as string), missing required fields, and out-of-range values. Invalid records are quarantined in an error log rather than loading bad data into the warehouse.

4. Memory-Optimized Batch Ingestion & Deduplication

Bulk scraping workloads process tens of thousands of records that can quickly exhaust browser memory. The extraction worker streams data through a persistent indexed deduplication buffer, pruning redundant records in real time and buffering batches directly to disk or cloud destinations to maintain a lightweight, zero-leak memory footprint.

Engineering Metrics & Commercial Outcomes

Frequently Asked Questions