Case studyRAG · LLM · FastAPI · Pinecone · Pythonfiltr-omega.vercel.app ↗8 min read
Filtr
RAG-Based Insight Engine for Product Managers
Problem
Product managers at startups spend 3 to 6 hours every week doing the same thing manually: reading through Slack threads, scanning Jira boards, and skimming call transcripts to figure out what users are actually complaining about. The output is a mental model that lives in their head, is non-reproducible, and disappears when they leave the company.
The tools that exist don't help. Jira search is keyword-only. Slack search has no cross-channel synthesis. Gong and Chorus are transcript-only and expensive. Nothing cross-references all three sources semantically.
The specific pain: a PM opens their tools on Monday morning and has no fast, reliable way to answer "what is my top user problem this quarter across all channels?" They have to read everything. Every time.
I was the user. I built Filtr to solve it.
Decision
The core hypothesis: if you embed multi-source PM data into a shared vector space and run semantic search over it, a PM can ask natural language questions and get sourced, synthesized answers in under 3 minutes, without reading anything manually.
What I built in V1: A hosted upload-and-query RAG tool. Upload Slack exports (JSON), Jira exports (CSV), and call transcripts (TXT/PDF). Ask natural language questions. Get answers with source attribution showing which file and which chunk each insight came from.
What I built in V2: The Insight Engine. Instead of waiting for the PM to know what to ask, Filtr automatically clusters the uploaded data into the top 5 issue themes after every upload, ranked by mention count, with source breakdown (Slack / Jira / Transcript). The PM sees "Mobile Checkout Failures: 7 mentions (5 Jira, 2 Slack)" before typing a single query.
What I rejected:
Per-chunk LLM tagging at ingestion was the obvious alternative for clustering. Tag every chunk with an issue category using an LLM call, then aggregate counts. I rejected it because at Gemini free tier (15 RPM), tagging 300 chunks from a real upload would take 20+ minutes. It also scales linearly with data size and provides no meaningful advantage over embedding based clustering for distinct issue themes. The tradeoff accepted: k-means on embeddings degrades on noisy informal language (Slack casual messages) where semantically similar content uses very different phrasing. Acceptable for MVP. Source aware clustering (separate k-means per source type, then merge) is scoped for V2.1.
System Architecture
The architecture has three layers:
Ingestion layer: FastAPI backend receives uploaded files. Three parsers handle Slack (message level chunking), Jira (ticket level chunking), and transcripts (300-token chunks with 50-token overlap). All chunks are embedded in a single batch call to gemini-embedding-001 (768 dims) via direct REST, not the SDK. Vectors are stored in Pinecone with metadata: source_type, session_id, original_text.
Query layer: On query, the user's question is embedded using the same model with taskType: retrieval_query. Top 20 candidates retrieved from Pinecone by cosine similarity. Cohere Rerank (rerank : english v3.0) rescores the candidates for relevance to the specific query. Top 5 reranked chunks passed to GroQ (Llama 3.3 70B) for answer generation. Answer is grounded: the prompt instructs the model to answer only from retrieved chunks and say so explicitly if they don't contain the answer.
Clustering layer: After ingestion, a separate /insights endpoint runs k means (k=5, scikit learn) on all session vectors fetched from Pinecone. Three representative chunks per cluster (closest to centroid) are passed in a single LLM call to GroQ for cluster naming. Cluster names, mention counts, and source breakdowns are returned as structured JSON.
Rate limit management: All three API providers (Gemini, GroQ, Cohere) are managed through a KeyPoolManager singleton. Round robin key rotation across 4 Gemini keys and 3 GroQ keys from separate accounts. On 429, the key is put on 65 second cooldown and the next available key is used immediately. This was the critical infrastructure that made the free tier architecture viable at any real load.
Technical Choices
Gemini gemini embedding 001 for embeddings, GroQ Llama 3.3 70B for generation. Originally everything was Gemini. The rate limit problem forced a split: Gemini's free tier at 15 RPM meant embedding + cluster naming + query answering all firing within seconds would exhaust quota and push /insights past Render's 30 second response timeout (resulting in 502s). Separating embedding (Gemini) from generation (GroQ) meant two independent rate limit pools. GroQ's free tier is generous on tokens and has effectively no RPM limit at this scale.
Direct REST over SDK. The google generativeai SDK routes through gRPC and has version compatibility issues that caused silent failures. Both text embedding 004 and gemini 1.5 flash were retired in early February 2026 mid build. Direct REST calls via requests are explicit, debuggable, and not tied to SDK version. Every API call in the codebase uses requests.post() directly.
Batch embeddings. The first version of embed_texts() called the Gemini API once per chunk in a sequential loop. For a 30 chunk upload (small), that was 30 sequential API calls taking ~3 minutes. Switched to batchEmbedContents: one API call for all chunks regardless of count. Upload time dropped from ~3 minutes to ~10 seconds.
Cohere reranking. Cosine similarity alone was producing "no context found" responses on valid queries because the top 8 candidates by cosine score weren't always the most contextually relevant. Adding a reranking pass with Cohere's rerank english v3.0 model re scores top 20 cosine candidates for query relevance. Retrieval quality improved meaningfully on edge cases.
Session based ephemeral architecture. No user accounts. No persistent storage. Every upload generates a session_id. Pinecone vectors tagged with that session_id. 4 query cap per session enforced in memory. This kept the architecture simple, avoided auth complexity entirely, and is appropriate for a tool where users upload their own proprietary PM data; they likely don't want it persisted.
Metrics
V1 targets (measured on beta cohort with mock data):
| Metric | Target | Result |
|---|---|---|
| Retrieval Relevance Score | >80% chunks rated relevant | Validated on 4 test queries with mock data |
| Time to First Insight | <3 minutes upload to first result | ~45 seconds on warm backend |
| Session Completion Rate | >60% complete 3+ queries | Confirmed on internal testing |
V2 targets:
| Metric | Target |
|---|---|
| Cluster Name Accuracy | >75% rated useful via thumbs up |
| Time to First Cluster | <90 seconds post upload |
| Query volume per session | +20% vs V1 baseline |
The riskiest assumption in V2 was cluster trust: would PMs act on auto generated cluster names without manual synthesis? This was the primary thing being tested in beta.
What Broke
The Gemini 15 RPM wall. This was the biggest architectural failure of the build. I designed the system with a single Gemini API key handling embeddings, cluster naming, and query answering. In production, a single upload triggering all three sequentially would exhaust the 15 RPM quota. The cluster naming LLM call would hit a 429, wait 30 seconds, retry, and push the total /insights response time past Render's 30 second timeout. The result: a 502 error on the most important new feature of V2, every time.
The fix required a full task split architecture redesign: Gemini handles only embeddings, GroQ handles all generation. Plus the KeyPoolManager for round robin rotation across 4 Gemini keys and 3 GroQ keys from separate accounts. What should have been a 2 hour feature became a 2 day infrastructure problem.
Root cause I missed at design time: I had accounted for the 15 RPM limit in isolation for each task but not for the compounded rate across tasks that fire sequentially within the same request cycle. The batch embeddings fix helped (30 calls → 1 call) but didn't fully solve it because cluster naming and query answering each still consume RPM after the embedding call.
Lesson: When designing against rate limited free APIs, model the worst case request cascade, not individual call counts.
The React StrictMode double invoke. In development, React 18 StrictMode double invokes effects. Two /insights fetch calls fired on every upload. The first returned empty clusters (Pinecone hadn't finished indexing), the .finally() set insightsLoading(false), and the second returned correct data but the state update was getting overwritten in the race. Cluster cards showed nothing despite correct API responses. Fixed with a 2 second setTimeout before the fetch to ensure Pinecone indexing completes, and local state tracking instead of relying on backend counters for the query cap.
API keys auto revoked twice. The Gemini API key appeared in the browser's Network tab during local development. Google auto revokes keys detected in browser accessible network traffic. The fix was restricting the key to generativelanguage.googleapis.com in Google AI Studio and being more careful about dev tools. Lost half a day to this twice.
PowerShell encoding corruption. Every time I used PowerShell to write Python files with Unicode characters (em dashes, box-drawing chars in comments), the file got corrupted with â€" and â"€ byte sequences. Spent significant time chasing bugs that turned out to be encoding artifacts. Rule established: never use PowerShell to write Python files. Use Python to write Python.
Retrospective
What I'd do differently:
Design the rate limit architecture first, before writing a single line of feature code. The 15 RPM wall was knowable at day one I knew the limit existed, I just underestimated the compounded impact of sequential API calls within a single request. The entire KeyPoolManager infrastructure, the Gemini/GroQ task split, and two days of debugging could have been avoided with 30 minutes of upfront capacity planning.
The session based ephemeral architecture was the right call for V1. But I'd add a lightweight analytics layer from day one, not for users, for me. Knowing which queries were firing, which clusters were getting thumbs-up, and where sessions were dropping off would have made V2 scoping decisions much faster. I was flying blind on what users actually did in the product.
Batch embeddings should have been the default from the start. Sequential-per-chunk was the naive first implementation. The performance difference (3 minutes vs 10 seconds) is significant enough that it should have been in the original design, not a hotfix after seeing users wait.
The product decision I'm most uncertain about: the 4-query cap. It prevents abuse on the free API pool, which was necessary, but it also caps the most valuable thing Filtr can do -- the more you query, the more signal you extract. If I were building V3, I'd replace the hard cap with a soft nudge and move toward authenticated accounts with per-user quota management.