Note: submission deadline in 1h, good luck to everyone.
\nI'd like to share with you some insights after ~200h, and $200 of API spending: Databricks/OpenAI/Parasail closing in about 1B tokens spent π
\nDocument Purpose: Foundation for Kaggle submission justification
\nDate: 2025-10-21
Document Ranking:
\nChunk Ranking:
\nAll tasks scored independently using three metrics at top-5 positions:
\nThe Leaderboard only shows MAP@5, which is not the full picture (also focus on nDCG@5 mentioned in Discussion topic by organizers)
\nQuery Message Characteristics:
\nScore Distribution Non-Uniformity:
\n| Score | \nFrequency | \nPercentage | \n
|---|---|---|
| 0 | \n6,064 | \n24.3% | \n
| 1 | \n5,367 | \n21.5% | \n
| 2 | \n4,951 | \n19.9% | \n
| 3 | \n4,605 | \n18.5% | \n
| 4 | \n3,943 | \n15.8% | \n
Scale & Context Window:
\nOn data labelling:
\nDistribution of Chunk Relevance dev-set:
\n| Metric | \nValue | \n
|---|---|
| Relevant chunks (score > 0) | \n5.1% of all chunks | \n
| Queries with only 1-2 relevant chunks | \n30.67% (needle-in-haystack) | \n
| Queries with >10 highly relevant chunks | \n4.25% | \n
Distribution of ground truth dev-set:
\n| Score | \nFrequency | \nPercentage | \n
|---|---|---|
| 0 (irrelevant) | \n3,029,827 | \n94.9% | \n
| 1 (relevant) | \n118,778 | \n3.7% | \n
| 2 (highly relevant) | \n44,847 | \n1.4% | \n
Token Distribution (eval-set):
\n| Context Window | \nQueries | \nPercentage | \n
|---|---|---|
| >32K | \n135 | \n67.5% | \n
| >60K | \n106 | \n53.0% | \n
| >128K | \n18 | \n9.0% | \n
Definition: Sparse signal detection - finding 1-2 relevant chunks among 100-300 candidates.
\nQuantitative Impact:
\nConcrete Example (Query q2e28fdef127e):
\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\nTotal chunks: \nRelevant: chunk (position / ~ % relative context)\nContent: [] State- loan distribution\n Washington , | Ohio , | York | Colorado ,\nFailure Mechanism:
\nImpact: Even advanced LLMs struggle when relevant chunks lack direct keyword matches and require financial domain inference. Position does matter, needles in early/last relative positions have better chance - \"lost in the middle\".
\nProblem: 67.5% of chunk queries exceed 32K tokens (refer to Context Rot recent findings), 10% exceed 120k tokens
\nEval Set Statistics:
\n| Percentile | \nMessage Tokens | \nChunks | \n
|---|---|---|
| P50 | \n63,756 | \n160 | \n
| P90 | \n120,451 | \n~300 | \n
| Max | \n224,000 | \n661 | \n
Processing Constraint:
\nTrade-off: More splits enable good processing early BUT create compound drop problem (Challenge 3).
\nDefinition: Local filtering loses globally relevant chunks in multi-stage ranking.
\nDataflow Example (5-part split):
\nQuery: chunks, relevant\n\nPart : chunks, relevant β Rank β top β If relevant ranks th β DROPPED\nPart : chunks, relevant β Rank β top β If relevant ranks th β DROPPED\n...\nPart : chunks, relevant β process\n\n: part creates opportunity\n Dropped chunks NEVER reach ranking stage\nExperimental Evidence (split size analysis, n=100):
\n| Split Size | \nLocal Recall | \nZero-Score Rate | \n
|---|---|---|
| 3K tokens/part (many splits) | \n75.35% | \n3.3% | \n
| 15K tokens/part (optimal) | \n75.35% | \n~15-18% | \n
| 30K tokens/part (fewer splits) | \n64.16% | \n28.6% | \n
Critical Finding: Splitting enable 'divide and conquer' (insight from the organizers), but too many splits -> global pool dilution (next).
\nDefinition: Chunks surviving local filtering still face overwhelming competition in global pool.
\nConcrete Example (Query q4054d9fe6d42):
\n: chunks, relevant, using top-k= (because each split now has few chunks)\n split β parts \n\n Stage:\n : relevant chunks\n : chunks\n : relevant chunks β Sent to global pool β\n\n Pool:\n candidates: parts Γ = candidates\n density: / = .% relevant\n : / = .% irrelevant\n\n Re-Ranking (top- selection):\n : relevant chunks\n : relevant chunks\n\n Drop Rate: β = % of surviving chunks lost at global stage\n Score: .\nMechanism: Surviving chunks compete in global pool. Stronger competition with 'distractors' (seemingly relevant, or weakly relevant).
\nComparison (same query):
\n split β parts\n\n Stage: ~ chunks survive ( dropped locally)\n Pool: Γ = candidates\n density: / = % relevant (vs .% for K)\n Selection: ~ relevant chunks\n Drop Rate: β = % (better than K's %)\n Score: .-. (x improvement)\nInsight: Signal-to-noise ratio in later-stage pool determines final performance. There in an inverted-u curve when it comes to splitting (assume fixed top-k, I also tried dynamic k but more issues there).
\nDefinition: queries require inference fail more when context is fragmented across splits.
\nExample Dataflow (Query q2e28fdef127e):
\n \n\nPart (chunks ): Contains chunk (state-level loan table)\n β ranks in isolation\n β No linking to \n β Chunk . (weak relevance)\n β Dropped from local top\n\nPart (chunks ): Contains chunks about (conceptual β part, never sees chunk data\n\nPart (chunks ): Contains chunks about (meeting transcript)\n β part, never sees chunk data\n never synthesizes connection () Geographic loan data [Part ]\n () Expansion strategy [Part ]\n () Investor view framing [Part ]\n β Final . (complete failure)\nImpact: Context fragmentation breaks holistic understanding needed for semantic inference in financial domain.
\nHypothesis: if chunks are small and uniform (say 50-200 tokens)
\nSingle-stage ranking ( chunks):\n β LLM sees chunk ( loan )\n β Also sees expansion strategy discussion\n β Also sees investor question framing\n β Can infer: \n β Makes final top-\nBut this is not the reality. Extreme variance found in chunks' token counts, leptokurtic (1-71K), flood context window.
\nDefinition: LLMs struggle to find needles when: (1) prompt is longer, and (2) relevant chunk is not in first-last 20% of context window.
\nWhy Extended Context Fails:
\n\n\n \n \n \n \n\n\n \n \n \n \n \nAttention Allocation Comparison:
\n| Strategy | \nChunks/Context | \nAttention/Chunk | \nGlobal Pool | \nGlobal Attention | \n
|---|---|---|---|---|
| 30-chunk parts | \n30 | \n3.3% | \n50 candidates | \n2.0% | \n
| 95-chunk parts | \n95 | \n1.1% | \n475 candidates | \n0.2% | \n
| Extended (full) | \n309 | \n0.3% | \nN/A | \n0.3% | \n
Empirical Validation: Long-context models struggle, perform worse than splitting. Splitting is critical, should NOT fit everything in one query.
\nIntuition: Fewer splits β Fewer local competitions β Less compound drop risk.
\nReality: Larger splits create saturated ranking tasks that degrade performance.
\nExperimental Evidence (2-split performance comparison):
\n| Threshold | \nChunks/Part | \nTokens/Part | \n2-Split Local Recall | \n
|---|---|---|---|
| 15K | \n~50-70 | \n15K | \n80.55% (BEST) | \n
| 30K | \n~80-100 | \n30K | \n66.07% (-14.5pts) | \n
| 45K | \n~100-150 | \n45K | \n54.41% (WORST, -26.1pts) | \n
Mechanism:
\nOptimal Point: ~15K threshold (50-70 chunks/part) balances all three factors.
\nNaive Solution: Increase local top-K from 5 to 10 to capture more relevant chunks.
\nWhy It Fails:
\nExample: =5 =10, given 10 equal splits\nLocal Ranking (Part-2, 18 chunks, =10):\n Position 6: Relevant chunk, score 0.73 β CAPTURED with =10 β (would drop with =5)\n\n\nGlobal Ranking : 50 candidates -> 100 candidates:\n Relevant chunk STILL has score 0.73 (unchanged)\n But now competes with 90 instead of 40 chunks.\n β Relevant chunk ranks ~45th globally\n β DROPPED final top-5 β\nMathematical Formula:
\n\n\n :\n \n \n : \nCritical Insight: We cannot improve global outcomes without changing the scores in the 2nd stage -> re-ranking using math formula rarely works. This agrees with the baseline setup from the organizers.
\nSolution: Global re-scoring assigns NEW scores in full context:
\nEmpirical Validation: Global re-scoring provides +5.8pts improvement over single-stage architecture.
\nProposed Approach: Use BM25/embeddings to filter chunks before LLM ranking.
\nWhy It Fails:
\nPipeline: BM25 β :\n: \nChunk : [TABLE] \n\nBM25 :\n Question : [, , , ]\n Chunk : [, , , , ]\n Lexical : MINIMAL ( weak match on geographic terms)\n BM25 : ~- out of chunks\n\nFilter to top-: chunks\n β Chunk in top-\n β DROPPED at Stage \n β Never reaches LLM\n β Final : \nFundamental Error: Financial questions require conceptual understanding:
\nBM25/embeddings cannot infer these semantic connections β Drops relevant chunks β Permanent information loss.
\nConclusion: Lexical filtering BEFORE deep reasoning is more harmful than multi-stage LLM architecture. Only LLM can perform required semantic inference.
\nAssumption: Longer context windows (200K) solve splitting problem by processing all chunks together.
\nReality: Attention degradation worse than multi-stage processing.
\nEvidence (predicted performance for 309-chunk query):
\n\n \n \n \n \n\n\n \n \n \nImplication: More context creates MORE noise, not less. The needle gets buried deep.
\nData Characteristics:
\nTechnical Challenges:
\nWhat Fails:
\nWhat Works:
\nCritical Design Principles: Optimize attention allocation within context rot limit while preserving high local recall at filtering stage.
", "messages": [ { "id": "3304658", "postDate": "10/21/2025 03:24:46", "content": "Note: submission deadline in 1h, good luck to everyone.
\nI'd like to share with you some insights after ~200h, and $200 of API spending: Databricks/OpenAI/Parasail closing in about 1B tokens spent π
\nDocument Purpose: Foundation for Kaggle submission justification
\nDate: 2025-10-21
Document Ranking:
\nChunk Ranking:
\nAll tasks scored independently using three metrics at top-5 positions:
\nThe Leaderboard only shows MAP@5, which is not the full picture (also focus on nDCG@5 mentioned in Discussion topic by organizers)
\nQuery Message Characteristics:
\nScore Distribution Non-Uniformity:
\n| Score | \nFrequency | \nPercentage | \n
|---|---|---|
| 0 | \n6,064 | \n24.3% | \n
| 1 | \n5,367 | \n21.5% | \n
| 2 | \n4,951 | \n19.9% | \n
| 3 | \n4,605 | \n18.5% | \n
| 4 | \n3,943 | \n15.8% | \n
Scale & Context Window:
\nOn data labelling:
\nDistribution of Chunk Relevance dev-set:
\n| Metric | \nValue | \n
|---|---|
| Relevant chunks (score > 0) | \n5.1% of all chunks | \n
| Queries with only 1-2 relevant chunks | \n30.67% (needle-in-haystack) | \n
| Queries with >10 highly relevant chunks | \n4.25% | \n
Distribution of ground truth dev-set:
\n| Score | \nFrequency | \nPercentage | \n
|---|---|---|
| 0 (irrelevant) | \n3,029,827 | \n94.9% | \n
| 1 (relevant) | \n118,778 | \n3.7% | \n
| 2 (highly relevant) | \n44,847 | \n1.4% | \n
Token Distribution (eval-set):
\n| Context Window | \nQueries | \nPercentage | \n
|---|---|---|
| >32K | \n135 | \n67.5% | \n
| >60K | \n106 | \n53.0% | \n
| >128K | \n18 | \n9.0% | \n
Definition: Sparse signal detection - finding 1-2 relevant chunks among 100-300 candidates.
\nQuantitative Impact:
\nConcrete Example (Query q2e28fdef127e):
\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\nTotal chunks: \nRelevant: chunk (position / ~ % relative context)\nContent: [] State- loan distribution\n Washington , | Ohio , | York | Colorado ,\nFailure Mechanism:
\nImpact: Even advanced LLMs struggle when relevant chunks lack direct keyword matches and require financial domain inference. Position does matter, needles in early/last relative positions have better chance - \"lost in the middle\".
\nProblem: 67.5% of chunk queries exceed 32K tokens (refer to Context Rot recent findings), 10% exceed 120k tokens
\nEval Set Statistics:
\n| Percentile | \nMessage Tokens | \nChunks | \n
|---|---|---|
| P50 | \n63,756 | \n160 | \n
| P90 | \n120,451 | \n~300 | \n
| Max | \n224,000 | \n661 | \n
Processing Constraint:
\nTrade-off: More splits enable good processing early BUT create compound drop problem (Challenge 3).
\nDefinition: Local filtering loses globally relevant chunks in multi-stage ranking.
\nDataflow Example (5-part split):
\nQuery: chunks, relevant\n\nPart : chunks, relevant β Rank β top β If relevant ranks th β DROPPED\nPart : chunks, relevant β Rank β top β If relevant ranks th β DROPPED\n...\nPart : chunks, relevant β process\n\n: part creates opportunity\n Dropped chunks NEVER reach ranking stage\nExperimental Evidence (split size analysis, n=100):
\n| Split Size | \nLocal Recall | \nZero-Score Rate | \n
|---|---|---|
| 3K tokens/part (many splits) | \n75.35% | \n3.3% | \n
| 15K tokens/part (optimal) | \n75.35% | \n~15-18% | \n
| 30K tokens/part (fewer splits) | \n64.16% | \n28.6% | \n
Critical Finding: Splitting enable 'divide and conquer' (insight from the organizers), but too many splits -> global pool dilution (next).
\nDefinition: Chunks surviving local filtering still face overwhelming competition in global pool.
\nConcrete Example (Query q4054d9fe6d42):
\n: chunks, relevant, using top-k= (because each split now has few chunks)\n split β parts \n\n Stage:\n : relevant chunks\n : chunks\n : relevant chunks β Sent to global pool β\n\n Pool:\n candidates: parts Γ = candidates\n density: / = .% relevant\n : / = .% irrelevant\n\n Re-Ranking (top- selection):\n : relevant chunks\n : relevant chunks\n\n Drop Rate: β = % of surviving chunks lost at global stage\n Score: .\nMechanism: Surviving chunks compete in global pool. Stronger competition with 'distractors' (seemingly relevant, or weakly relevant).
\nComparison (same query):
\n split β parts\n\n Stage: ~ chunks survive ( dropped locally)\n Pool: Γ = candidates\n density: / = % relevant (vs .% for K)\n Selection: ~ relevant chunks\n Drop Rate: β = % (better than K's %)\n Score: .-. (x improvement)\nInsight: Signal-to-noise ratio in later-stage pool determines final performance. There in an inverted-u curve when it comes to splitting (assume fixed top-k, I also tried dynamic k but more issues there).
\nDefinition: queries require inference fail more when context is fragmented across splits.
\nExample Dataflow (Query q2e28fdef127e):
\n \n\nPart (chunks ): Contains chunk (state-level loan table)\n β ranks in isolation\n β No linking to \n β Chunk . (weak relevance)\n β Dropped from local top\n\nPart (chunks ): Contains chunks about (conceptual β part, never sees chunk data\n\nPart (chunks ): Contains chunks about (meeting transcript)\n β part, never sees chunk data\n never synthesizes connection () Geographic loan data [Part ]\n () Expansion strategy [Part ]\n () Investor view framing [Part ]\n β Final . (complete failure)\nImpact: Context fragmentation breaks holistic understanding needed for semantic inference in financial domain.
\nHypothesis: if chunks are small and uniform (say 50-200 tokens)
\nSingle-stage ranking ( chunks):\n β LLM sees chunk ( loan )\n β Also sees expansion strategy discussion\n β Also sees investor question framing\n β Can infer: \n β Makes final top-\nBut this is not the reality. Extreme variance found in chunks' token counts, leptokurtic (1-71K), flood context window.
\nDefinition: LLMs struggle to find needles when: (1) prompt is longer, and (2) relevant chunk is not in first-last 20% of context window.
\nWhy Extended Context Fails:
\n\n\n \n \n \n \n\n\n \n \n \n \n \nAttention Allocation Comparison:
\n| Strategy | \nChunks/Context | \nAttention/Chunk | \nGlobal Pool | \nGlobal Attention | \n
|---|---|---|---|---|
| 30-chunk parts | \n30 | \n3.3% | \n50 candidates | \n2.0% | \n
| 95-chunk parts | \n95 | \n1.1% | \n475 candidates | \n0.2% | \n
| Extended (full) | \n309 | \n0.3% | \nN/A | \n0.3% | \n
Empirical Validation: Long-context models struggle, perform worse than splitting. Splitting is critical, should NOT fit everything in one query.
\nIntuition: Fewer splits β Fewer local competitions β Less compound drop risk.
\nReality: Larger splits create saturated ranking tasks that degrade performance.
\nExperimental Evidence (2-split performance comparison):
\n| Threshold | \nChunks/Part | \nTokens/Part | \n2-Split Local Recall | \n
|---|---|---|---|
| 15K | \n~50-70 | \n15K | \n80.55% (BEST) | \n
| 30K | \n~80-100 | \n30K | \n66.07% (-14.5pts) | \n
| 45K | \n~100-150 | \n45K | \n54.41% (WORST, -26.1pts) | \n
Mechanism:
\nOptimal Point: ~15K threshold (50-70 chunks/part) balances all three factors.
\nNaive Solution: Increase local top-K from 5 to 10 to capture more relevant chunks.
\nWhy It Fails:
\nExample: =5 =10, given 10 equal splits\nLocal Ranking (Part-2, 18 chunks, =10):\n Position 6: Relevant chunk, score 0.73 β CAPTURED with =10 β (would drop with =5)\n\n\nGlobal Ranking : 50 candidates -> 100 candidates:\n Relevant chunk STILL has score 0.73 (unchanged)\n But now competes with 90 instead of 40 chunks.\n β Relevant chunk ranks ~45th globally\n β DROPPED final top-5 β\nMathematical Formula:
\n\n\n :\n \n \n : \nCritical Insight: We cannot improve global outcomes without changing the scores in the 2nd stage -> re-ranking using math formula rarely works. This agrees with the baseline setup from the organizers.
\nSolution: Global re-scoring assigns NEW scores in full context:
\nEmpirical Validation: Global re-scoring provides +5.8pts improvement over single-stage architecture.
\nProposed Approach: Use BM25/embeddings to filter chunks before LLM ranking.
\nWhy It Fails:
\nPipeline: BM25 β :\n: \nChunk : [TABLE] \n\nBM25 :\n Question : [, , , ]\n Chunk : [, , , , ]\n Lexical : MINIMAL ( weak match on geographic terms)\n BM25 : ~- out of chunks\n\nFilter to top-: chunks\n β Chunk in top-\n β DROPPED at Stage \n β Never reaches LLM\n β Final : \nFundamental Error: Financial questions require conceptual understanding:
\nBM25/embeddings cannot infer these semantic connections β Drops relevant chunks β Permanent information loss.
\nConclusion: Lexical filtering BEFORE deep reasoning is more harmful than multi-stage LLM architecture. Only LLM can perform required semantic inference.
\nAssumption: Longer context windows (200K) solve splitting problem by processing all chunks together.
\nReality: Attention degradation worse than multi-stage processing.
\nEvidence (predicted performance for 309-chunk query):
\n\n \n \n \n \n\n\n \n \n \nImplication: More context creates MORE noise, not less. The needle gets buried deep.
\nData Characteristics:
\nTechnical Challenges:
\nWhat Fails:
\nWhat Works:
\nCritical Design Principles: Optimize attention allocation within context rot limit while preserving high local recall at filtering stage.
", "rawMarkdown": "Note: submission deadline in 1h, good luck to everyone. \n\nI'd like to share with you some insights after ~200h, and $200 of API spending: Databricks/OpenAI/Parasail closing in about 1B tokens spent π \n\n# ACM ICAIF 2025 Competition: Design & Technical Challenges\n\n**Document Purpose**: Foundation for Kaggle submission justification\n**Date**: 2025-10-21\n\n---\n\n## 1. Competition Structure\n\n### Task Definitions\n\n**Document Ranking**:\n- Input: Financial question + 5 fixed SEC filing types (DEF14A, 10-K, 10-Q, 8-K, Earnings)\n- Output: Ranked list of all 5 document indices\n- Ground truth: Graded relevance 0-4 (4=most relevant)\n\n**Chunk Ranking**:\n- Input: Financial question + 100-300 paragraph-level text chunks\n- Output: Top-5 ranked list of most relevant chunk indices\n- Ground truth: Graded relevance 0-2 (2=highly relevant), *different than required output*\n\n\n### Evaluation Metrics\n\nAll tasks scored independently using three metrics at top-5 positions:\n- **MRR@5** (Mean Reciprocal Rank): Rewards first relevant result\n- **MAP@5** (Mean Average Precision): Measures ranking of all relevant items\n- **nDCG@5** (Normalized Discounted Cumulative Gain): Accounts for graded relevance with position discounting\n\nThe Leaderboard only shows MAP@5, which is not the full picture (also focus on nDCG@5 mentioned in Discussion topic by organizers)\n\n---\n\n## 2. Data Characteristics\n\n### Document Ranking\n\n**Query Message Characteristics**:\n- Consistent 155 tokens per message (range: 144-173)\n- 79.1% queries use all 5 scores [0,1,2,3,4]\n- **20.9% have ties** (should not happen as in competition instructions, but present in data)\n\n\n**Score Distribution Non-Uniformity**:\n| Score | Frequency | Percentage |\n|-------|-----------|------------|\n| 0 | 6,064 | 24.3% |\n| 1 | 5,367 | 21.5% |\n| 2 | 4,951 | 19.9% |\n| 3 | 4,605 | 18.5% |\n| 4 | 3,943 | 15.8% |\n\n\n### Chunk Ranking\n\n**Scale & Context Window**:\n- Mean: 190 chunks per query, 65K tokens\n- Range: 5-661 chunks, 2.7K-224K tokens\n- **Critical**: 67.5% queries exceed 32K tokens (eval set)\n- Individual chunks: Mean 335 tokens (range 1-71K), median 182 tokens\n\nOn *data labelling*:\n- About <1% of labels are extremely short 1-3 tokens: in data collection, experts see the full document (e.g. 1 relevant chunk with one token \"43\", among ~200 scored as not relevant) vs. this data, we only see chunks.\n- There are queries with >10 relevant chunks (max: 142)\n\n\n**Distribution of Chunk Relevance dev-set**:\n| Metric | Value |\n|--------|-------|\n| Relevant chunks (score > 0) | 5.1% of all chunks |\n| Queries with only 1-2 relevant chunks | 30.67% (needle-in-haystack) |\n| Queries with >10 highly relevant chunks | 4.25% |\n\n***\n\n**Distribution of ground truth dev-set**:\n\n| Score | Frequency | Percentage |\n|-------|-----------|------------|\n| 0 (irrelevant) | 3,029,827 | 94.9% |\n| 1 (relevant) | 118,778 | 3.7% |\n| 2 (highly relevant) | 44,847 | 1.4%|\n\n***\n\n**Token Distribution** (eval-set):\n| Context Window | Queries | Percentage |\n|----------------|---------|------------|\n| >32K | 135 | 67.5% |\n| >60K | 106 | 53.0% |\n| >128K | 18 | 9.0% |\n\n\n---\n\n## 3. Technical Challenges\n\n### Challenge 1: Needle-in-Haystack Problem\n\n**Definition**: Sparse signal detection - finding 1-2 relevant chunks among 100-300 candidates.\n\n**Quantitative Impact**:\n- 30.67% of chunk queries have only 1-2 relevant chunks\n- 5.1% overall chunk relevance rate (94.9% noise)\n- 12.5% zero-score rate observed in early experiments (fail to find the needles in top-5 = 0x3 scores)\n\n**Concrete Example** (Query q2e28fdef127e):\n```\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\nTotal chunks: 309\nRelevant: 1 chunk (position 52/309 ~ 17% relative context)\nContent: [TABLE] State-level loan distribution\n Washington $4,605 | Ohio $2,725 | New York $822 | Colorado $3,027\n```\n\n**Failure Mechanism**:\n- Lexical overlap: ZERO (question has \"expansion\", \"prospects\"; chunk has state names, numbers)\n- Required inference: \"Loan distribution by state\" β \"Geographic footprint\" β \"Expansion capability\"\n- Weak signal buried in noise β Dropped at filtering stage β Final score: 0.0\n\n**Impact**: Even advanced LLMs struggle when relevant chunks lack direct keyword matches and require financial domain inference. Position does matter, needles in early/last relative positions have better chance - \"lost in the middle\".\n\n---\n\n### Challenge 2: Context Window Constraints\n\n**Problem**: 67.5% of chunk queries exceed 32K tokens (refer to Context Rot recent findings), 10% exceed 120k tokens\n\n**Eval Set Statistics**:\n| Percentile | Message Tokens | Chunks |\n|------------|----------------|--------|\n| P50 | 63,756 | 160 |\n| P90 | 120,451 | ~300 |\n| Max | 224,000 | 661 |\n\n\n**Processing Constraint**:\n- Most production LLMs: 32K-128K context windows\n- Even 200K-1M context models face \"lost in the middle\" phenomenon\n- Requires splitting into multiple parts for processing\n\n**Trade-off**: More splits enable good processing early BUT create compound drop problem (Challenge 3).\n\n---\n\n### Challenge 3: Multi-Stage Compound Drop Problem\n\n**Definition**: Local filtering loses globally relevant chunks in multi-stage ranking.\n\n**Dataflow Example** (5-part split):\n```\nQuery: 300 chunks, 15 relevant\n\nPart 1: 60 chunks, 3 relevant β Rank all β Select top-10 β If relevant ranks 11-15th β DROPPED\nPart 2: 60 chunks, 2 relevant β Rank all β Select top-10 β If relevant ranks 11-15th β DROPPED\n...\nPart 5: 60 chunks, 4 relevant β Similar process\n\nResult: Each part creates drop opportunity\n Dropped chunks NEVER reach global ranking stage\n```\n\n**Experimental Evidence** (split size analysis, n=100):\n\n| Split Size | Local Recall | Zero-Score Rate |\n|-----------|--------------|-----------------|\n| 3K tokens/part (many splits) | 75.35% | 3.3% |\n| 15K tokens/part (optimal) | 75.35% | ~15-18% |\n| 30K tokens/part (fewer splits) | 64.16% | 28.6% |\n\n\n**Critical Finding**: Splitting enable 'divide and conquer' (insight from the organizers), but too many splits -> global pool dilution (next).\n\n---\n\n### Challenge 4: Global Dilution Problem\n\n**Definition**: Chunks surviving local filtering still face overwhelming competition in global pool.\n\n**Concrete Example** (Query q4054d9fe6d42):\n```\nConfiguration: 289 chunks, 28 relevant, using top-k=5 (because each split now has few chunks)\n3K split β 95 parts \n\nLocal Stage:\n Started: 28 relevant chunks\n Dropped: 2 chunks\n Survived: 26 relevant chunks β Sent to global pool β\n\nGlobal Pool:\n Total candidates: 95 parts Γ 5 = 475 candidates\n Signal density: 26/475 = 5.5% relevant\n Noise: 449/475 = 94.5% irrelevant\n\nFinal Re-Ranking (top-5 selection):\n Selected: 2 relevant chunks\n Dropped: 24 relevant chunks\n\nGlobal Drop Rate: 26 β 2 = 92% of surviving chunks lost at global stage\nFinal Score: 0.28\n```\n\n**Mechanism**: Surviving chunks compete in global pool. Stronger competition with 'distractors' (seemingly relevant, or weakly relevant).\n\n\n**Comparison** (same query):\n```\n30K split β 10 parts\n\nLocal Stage: ~20 chunks survive (8 dropped locally)\nGlobal Pool: 10 Γ 5 = 50 candidates\nSignal density: 20/50 = 40% relevant (vs 5.5% for 3K)\nFinal Selection: ~9 relevant chunks\nGlobal Drop Rate: 20 β 9 = 55% (better than 3K's 92%)\nFinal Score: 0.55-0.65 (2x improvement)\n```\n\n**Insight**: *Signal-to-noise ratio in later-stage pool* determines final performance. There in an inverted-u curve when it comes to splitting (assume fixed top-k, I also tried dynamic k but more issues there).\n\n---\n\n### Challenge 5: Semantic Gap\n\n**Definition**: queries require *inference* fail more when context is fragmented across splits.\n\n**Example Dataflow** (Query q2e28fdef127e):\n```\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\n\nPart 1 (chunks 0-60): Contains chunk 52 (state-level loan table)\n β LLM ranks in isolation\n β No context linking \"state distribution\" to \"expansion prospects\"\n β Chunk 52 score: 0.2 (weak relevance)\n β Dropped from local top-10\n\nPart 2 (chunks 61-120): Contains chunks about \"expansion strategy\" (conceptual discussion)\n β Different part, never sees chunk 52 data\n\nPart 3 (chunks 121-180): Contains chunks about \"investor questions\" (meeting transcript)\n β Different part, never sees chunk 52 data\n\nResult: LLM never synthesizes connection between:\n (1) Geographic loan data [Part 1]\n (2) Expansion strategy context [Part 2]\n (3) Investor view framing [Part 3]\n β Final score: 0.0 (complete failure)\n```\n\n**Impact**: Context fragmentation breaks holistic understanding needed for semantic inference in financial domain. \n\n**Hypothesis**: if chunks are small and uniform (say 50-200 tokens)\n```\nSingle-stage ranking (all 309 chunks):\n β LLM sees chunk 52 (state loan table)\n β Also sees expansion strategy discussion\n β Also sees investor question framing\n β Can infer: \"Table shows geographic presence β Relevant to expansion question\"\n β Makes final top-5\n```\nBut this is not the reality. Extreme variance found in chunks' token counts, leptokurtic (1-71K), flood context window.\n\n---\n\n### Challenge 6: Context Rot & \"Lost in the middle\" \n\n**Definition**: LLMs struggle to find needles when: (1) prompt is longer, and (2) relevant chunk is not in first-last 20% of context window.\n\n**Why Extended Context Fails**:\n```\nPrediction for Query q2e28fdef127e (309 chunks, chunk 52 relevant at position 17%):\n\nMulti-stage 30K (10 parts):\n - Chunk 52 in Part 1 with 30 chunks\n - Attention per chunk: ~3.3%\n - Score: 0.3 (low due to semantic gap)\n - Dropped locally β SCORE: 0.0\n\nExtended context (all 309 chunks):\n - Chunk 52 buried at position 52/309 (17%)\n - Attention per chunk: ~0.3% (10x dilution)\n - \"Lost in middle\" degradation + context rot\n - Score: 0.1 (even lower)\n - Dropped in final ranking β SCORE: 0.0 (NO BETTER)\n```\n\n**Attention Allocation Comparison**:\n| Strategy | Chunks/Context | Attention/Chunk | Global Pool | Global Attention |\n|----------|----------------|-----------------|-------------|------------------|\n| 30-chunk parts | 30 | 3.3% | 50 candidates | 2.0% |\n| 95-chunk parts | 95 | 1.1% | 475 candidates | 0.2% |\n| Extended (full) | 309 | 0.3% | N/A | 0.3% |\n\n\n**Empirical Validation**: Long-context models struggle, perform worse than splitting. Splitting is critical, should NOT fit everything in one query.\n\n---\n\n## 4. Architecture Trade-offs\n\n### Trade-off 1: Split Size Paradox\n\n**Intuition**: Fewer splits β Fewer local competitions β Less compound drop risk.\n\n**Reality**: Larger splits create saturated ranking tasks that degrade performance.\n\n**Experimental Evidence** (2-split performance comparison):\n| Threshold | Chunks/Part | Tokens/Part | 2-Split Local Recall |\n|-----------|-------------|-------------|----------------------|\n| 15K | ~50-70 | 15K | **80.55%** (BEST) |\n| 30K | ~80-100 | 30K | 66.07% (-14.5pts) |\n| 45K | ~100-150 | 45K | **54.41%** (WORST, -26.1pts) |\n\n\n\n**Mechanism**:\n1. **Prompt Length Effect**: 15K prompts preserve instruction retention; 45K prompts degrade clarity\n2. **Ranking Capacity**: K/N ratio critical - 10/50 (20%) captures needles better than 10/150 (7%)\n3. **Compound Drops**: More splits create more opportunities to drop, BUT better RECALL per split -> win\n\n**Optimal Point**: ~15K threshold (50-70 chunks/part) balances all three factors.\n\n---\n\n### Trade-off 2: Local Pool Paradox\n\n**Naive Solution**: Increase local top-K from 5 to 10 to capture more relevant chunks.\n\n**Why It Fails**:\n```\nExample: K=5 to K=10, given 10 equal splits\nLocal Ranking (Part-2, 18 chunks, K=10):\n Position 6: Relevant chunk, score 0.73 β CAPTURED with K=10 β (would drop with K=5)\n\n\nGlobal Ranking : 50 candidates -> 100 candidates:\n Relevant chunk STILL has score 0.73 (unchanged)\n But now competes with 90 instead of 40 chunks.\n β Relevant chunk ranks ~45th globally\n β DROPPED from final top-5 β\n```\n\n\n**Mathematical Formula**:\n```\nP(final success) = P(local survival) Γ P(global survival)\n\nIncreasing K:\n P(local survival) β (more chunks captured)\n P(global survival) β (same low scores in larger pool)\n Net effect: Minimal improvement\n```\n\n\n**Critical Insight**: We cannot improve global outcomes without changing the scores in the 2nd stage -> re-ranking using math formula rarely works. This agrees with the baseline setup from the organizers.\n\n**Solution**: Global re-scoring assigns NEW scores in full context:\n- Stage 1 (local): Chunk gets 0.73 in limited context (18 chunks)\n- Stage 2 (global): Model sees all 10 x 10 = 100 candidates, assigns NEW score 0.88 \n- Result: Chunk becomes competitive, makes final top-5\n\n**Empirical Validation**: Global re-scoring provides +5.8pts improvement over single-stage architecture.\n\n---\n\n### Trade-off 3: BM25 Pre-filtering Catastrophe\n\n**Proposed Approach**: Use BM25/embeddings to filter chunks before LLM ranking.\n\n\n**Why It Fails**:\n```\nPipeline: BM25 β Filter to top-30% β LLM ranking\n\nExample (Query q2e28fdef127e):\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\nChunk 52: [TABLE] \"Washington $4,605 | Ohio $2,725 | New York $822...\"\n\nBM25 Analysis:\n Question terms: [\"investor\", \"views\", \"expansion\", \"prospects\"]\n Chunk terms: [\"Washington\", \"Ohio\", \"New York\", \"$4,605\", \"$2,725\"]\n Lexical overlap: MINIMAL (only weak match on geographic terms)\n BM25 rank: ~200-250 out of 309 chunks\n\nFilter to top-30%: 93 chunks\n β Chunk 52 NOT in top-93\n β DROPPED at Stage 1\n β Never reaches LLM\n β Final score: 0.0\n```\n\n**Fundamental Error**: Financial questions require conceptual understanding:\n- \"Geographic expansion\" = loan distribution by state (tables)\n- \"Revenue concentration\" = customer segment breakdown (tables)\n- \"Margin pressure\" = cost structure details (financial data)\n\n**BM25/embeddings cannot infer these semantic connections** β Drops relevant chunks β Permanent information loss.\n\n**Conclusion**: Lexical filtering BEFORE deep reasoning is more harmful than multi-stage LLM architecture. Only LLM can perform required semantic inference.\n\n---\n\n### Trade-off 4: Extended Context Failure\n\n**Assumption**: Longer context windows (200K) solve splitting problem by processing all chunks together.\n\n**Reality**: Attention degradation worse than multi-stage processing.\n\n**Evidence** (predicted performance for 309-chunk query):\n```\nCurrent 30K split (10 parts):\n Local: 30 chunks/part, 3.3% attention per chunk\n Global: 50 candidates, 2% attention per chunk\n Average attention: ~2.5% per relevant chunk\n Observed score: ~0.4-0.6 (partial success)\n\nExtended context (309 chunks):\n Single stage: 309 chunks, 0.3% attention per chunk\n \"Lost in middle\" effect: chunks at 10-90% position get <0.2% attention\n Predicted score: ~0.3-0.4 (WORSE than multi-stage)\n```\n\n**Implication**: More context creates MORE noise, not less. The needle gets buried deep.\n\n---\n\n## Summary\n\n### Competition Constraints Drive Design\n\n**Data Characteristics**:\n- 67.5% queries exceed 32K tokens β Requires multi-part processing\n- 30.67% queries have 1-2 relevant chunks among ~200-300 candidates β Needles-in-haystack\n- 5.1% overall relevance rate β Overwhelming noisy\n\n**Technical Challenges**:\n1. Sparse signal detection in high-noise environments\n2. Context window constraints requiring splitting\n3. Multi-stage compound drops (local filtering loses global relevance)\n4. Global dilution (surviving chunks face 92% drop rate in noisy pools)\n5. Semantic gaps (inference breaks when context fragments)\n6. Context rot (attention degradation in extended contexts)\n\n### Key Trade-offs\n\n**What Fails**:\n- Extended context single-stage: Context rot degrades performance\n- BM25 pre-filtering: Drops inference-requiring chunks\n- Large split sizes: Saturated ranking (100-150 chunks/split) performs worse than moderate (50-70 chunks/split)\n- Pool expansion without re-scoring: Local scores determine global outcomes\n\n**What Works**:\n- Optimal split size (15K tokens, 50-70 chunks/part): Balances attention vs drops\n- Global re-scoring: NEW scores in full context (+5.8pts improvement)\n- Lexical-semantic fusion: *BM25 boost (not filter)* helps weak signals without dropping chunks\n- Multi-stage architecture: Better than extended context despite compound drops\n\n**Critical Design Principles**: Optimize *attention allocation* within *context rot limit* while preserving high *local recall* at filtering stage.", "votes": null }, { "id": "3304678", "postDate": "10/21/2025 04:28:02", "content": "Wow, $200 for 200 hours β thatβs dedication! Thank you for sharing! I didnβt spend a penny on tokens, just fine-tuned downstream task ,but my GPU is running so hot Iβm getting roasted, and my electricity bill feels it .π
", "rawMarkdown": "Wow, $200 for 200 hours β thatβs dedication! Thank you for sharing! I didnβt spend a penny on tokens, just fine-tuned downstream task ,but my GPU is running so hot Iβm getting roasted, and my electricity bill feels it .π", "votes": null }, { "id": "3304682", "postDate": "10/21/2025 04:41:23", "content": "Nice, fine-tune with your GPU is cool. For me it was a chance to test backends also, can brag later that I win -200$ :) Did you notice issues with the labels? Some labels are out-of-context, about <1% relevant chunks have 1-3 tokens
", "rawMarkdown": "Nice, fine-tune with your GPU is cool. For me it was a chance to test backends also, can brag later that I win -200$ :) Did you notice issues with the labels? Some labels are out-of-context, about <1% relevant chunks have 1-3 tokens", "votes": null }, { "id": "3305075", "postDate": "10/22/2025 00:24:42", "content": "Yeah, exactly β because the text is split into chunks, some parts seem irrelevant ,only make sense with more context and domain knowledge. judging relevance just from one chunk is tricky. I wanted to try adding chunk position and some context info to the modelβs encoding, but I didnβt have enough time before the deadline.
\nAnother challenge is that the content isnβt just about finance β it also includes topics like Apple, healthcare, and many other companies and area , So finance domain isnβt enough to understand the overall context.
", "rawMarkdown": "Yeah, exactly β because the text is split into chunks, some parts seem irrelevant ,only make sense with more context and domain knowledge. judging relevance just from one chunk is tricky. I wanted to try adding chunk position and some context info to the modelβs encoding, but I didnβt have enough time before the deadline.\n\nAnother challenge is that the content isnβt just about finance β it also includes topics like Apple, healthcare, and many other companies and area , So finance domain isnβt enough to understand the overall context.", "votes": null } ], "comments": [ { "id": 3304678, "author_name": "yuzhenhu", "author_url": "", "post_date": "10/21/2025 04:28:02", "content": "Wow, $200 for 200 hours β thatβs dedication! Thank you for sharing! I didnβt spend a penny on tokens, just fine-tuned downstream task ,but my GPU is running so hot Iβm getting roasted, and my electricity bill feels it .π
", "votes": null, "replies": [ { "id": 3304682, "author_name": "pandalikematcha", "author_url": "", "post_date": "10/21/2025 04:41:23", "content": "Nice, fine-tune with your GPU is cool. For me it was a chance to test backends also, can brag later that I win -200$ :) Did you notice issues with the labels? Some labels are out-of-context, about <1% relevant chunks have 1-3 tokens
", "votes": null, "replies": [ { "id": 3305075, "author_name": "yuzhenhu", "author_url": "", "post_date": "10/22/2025 00:24:42", "content": "Yeah, exactly β because the text is split into chunks, some parts seem irrelevant ,only make sense with more context and domain knowledge. judging relevance just from one chunk is tricky. I wanted to try adding chunk position and some context info to the modelβs encoding, but I didnβt have enough time before the deadline.
\nAnother challenge is that the content isnβt just about finance β it also includes topics like Apple, healthcare, and many other companies and area , So finance domain isnβt enough to understand the overall context.
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3304658": "Note: submission deadline in 1h, good luck to everyone. \n\nI'd like to share with you some insights after ~200h, and $200 of API spending: Databricks/OpenAI/Parasail closing in about 1B tokens spent π \n\n# ACM ICAIF 2025 Competition: Design & Technical Challenges\n\n**Document Purpose**: Foundation for Kaggle submission justification\n**Date**: 2025-10-21\n\n---\n\n## 1. Competition Structure\n\n### Task Definitions\n\n**Document Ranking**:\n- Input: Financial question + 5 fixed SEC filing types (DEF14A, 10-K, 10-Q, 8-K, Earnings)\n- Output: Ranked list of all 5 document indices\n- Ground truth: Graded relevance 0-4 (4=most relevant)\n\n**Chunk Ranking**:\n- Input: Financial question + 100-300 paragraph-level text chunks\n- Output: Top-5 ranked list of most relevant chunk indices\n- Ground truth: Graded relevance 0-2 (2=highly relevant), *different than required output*\n\n\n### Evaluation Metrics\n\nAll tasks scored independently using three metrics at top-5 positions:\n- **MRR@5** (Mean Reciprocal Rank): Rewards first relevant result\n- **MAP@5** (Mean Average Precision): Measures ranking of all relevant items\n- **nDCG@5** (Normalized Discounted Cumulative Gain): Accounts for graded relevance with position discounting\n\nThe Leaderboard only shows MAP@5, which is not the full picture (also focus on nDCG@5 mentioned in Discussion topic by organizers)\n\n---\n\n## 2. Data Characteristics\n\n### Document Ranking\n\n**Query Message Characteristics**:\n- Consistent 155 tokens per message (range: 144-173)\n- 79.1% queries use all 5 scores [0,1,2,3,4]\n- **20.9% have ties** (should not happen as in competition instructions, but present in data)\n\n\n**Score Distribution Non-Uniformity**:\n| Score | Frequency | Percentage |\n|-------|-----------|------------|\n| 0 | 6,064 | 24.3% |\n| 1 | 5,367 | 21.5% |\n| 2 | 4,951 | 19.9% |\n| 3 | 4,605 | 18.5% |\n| 4 | 3,943 | 15.8% |\n\n\n### Chunk Ranking\n\n**Scale & Context Window**:\n- Mean: 190 chunks per query, 65K tokens\n- Range: 5-661 chunks, 2.7K-224K tokens\n- **Critical**: 67.5% queries exceed 32K tokens (eval set)\n- Individual chunks: Mean 335 tokens (range 1-71K), median 182 tokens\n\nOn *data labelling*:\n- About <1% of labels are extremely short 1-3 tokens: in data collection, experts see the full document (e.g. 1 relevant chunk with one token \"43\", among ~200 scored as not relevant) vs. this data, we only see chunks.\n- There are queries with >10 relevant chunks (max: 142)\n\n\n**Distribution of Chunk Relevance dev-set**:\n| Metric | Value |\n|--------|-------|\n| Relevant chunks (score > 0) | 5.1% of all chunks |\n| Queries with only 1-2 relevant chunks | 30.67% (needle-in-haystack) |\n| Queries with >10 highly relevant chunks | 4.25% |\n\n***\n\n**Distribution of ground truth dev-set**:\n\n| Score | Frequency | Percentage |\n|-------|-----------|------------|\n| 0 (irrelevant) | 3,029,827 | 94.9% |\n| 1 (relevant) | 118,778 | 3.7% |\n| 2 (highly relevant) | 44,847 | 1.4%|\n\n***\n\n**Token Distribution** (eval-set):\n| Context Window | Queries | Percentage |\n|----------------|---------|------------|\n| >32K | 135 | 67.5% |\n| >60K | 106 | 53.0% |\n| >128K | 18 | 9.0% |\n\n\n---\n\n## 3. Technical Challenges\n\n### Challenge 1: Needle-in-Haystack Problem\n\n**Definition**: Sparse signal detection - finding 1-2 relevant chunks among 100-300 candidates.\n\n**Quantitative Impact**:\n- 30.67% of chunk queries have only 1-2 relevant chunks\n- 5.1% overall chunk relevance rate (94.9% noise)\n- 12.5% zero-score rate observed in early experiments (fail to find the needles in top-5 = 0x3 scores)\n\n**Concrete Example** (Query q2e28fdef127e):\n```\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\nTotal chunks: 309\nRelevant: 1 chunk (position 52/309 ~ 17% relative context)\nContent: [TABLE] State-level loan distribution\n Washington $4,605 | Ohio $2,725 | New York $822 | Colorado $3,027\n```\n\n**Failure Mechanism**:\n- Lexical overlap: ZERO (question has \"expansion\", \"prospects\"; chunk has state names, numbers)\n- Required inference: \"Loan distribution by state\" β \"Geographic footprint\" β \"Expansion capability\"\n- Weak signal buried in noise β Dropped at filtering stage β Final score: 0.0\n\n**Impact**: Even advanced LLMs struggle when relevant chunks lack direct keyword matches and require financial domain inference. Position does matter, needles in early/last relative positions have better chance - \"lost in the middle\".\n\n---\n\n### Challenge 2: Context Window Constraints\n\n**Problem**: 67.5% of chunk queries exceed 32K tokens (refer to Context Rot recent findings), 10% exceed 120k tokens\n\n**Eval Set Statistics**:\n| Percentile | Message Tokens | Chunks |\n|------------|----------------|--------|\n| P50 | 63,756 | 160 |\n| P90 | 120,451 | ~300 |\n| Max | 224,000 | 661 |\n\n\n**Processing Constraint**:\n- Most production LLMs: 32K-128K context windows\n- Even 200K-1M context models face \"lost in the middle\" phenomenon\n- Requires splitting into multiple parts for processing\n\n**Trade-off**: More splits enable good processing early BUT create compound drop problem (Challenge 3).\n\n---\n\n### Challenge 3: Multi-Stage Compound Drop Problem\n\n**Definition**: Local filtering loses globally relevant chunks in multi-stage ranking.\n\n**Dataflow Example** (5-part split):\n```\nQuery: 300 chunks, 15 relevant\n\nPart 1: 60 chunks, 3 relevant β Rank all β Select top-10 β If relevant ranks 11-15th β DROPPED\nPart 2: 60 chunks, 2 relevant β Rank all β Select top-10 β If relevant ranks 11-15th β DROPPED\n...\nPart 5: 60 chunks, 4 relevant β Similar process\n\nResult: Each part creates drop opportunity\n Dropped chunks NEVER reach global ranking stage\n```\n\n**Experimental Evidence** (split size analysis, n=100):\n\n| Split Size | Local Recall | Zero-Score Rate |\n|-----------|--------------|-----------------|\n| 3K tokens/part (many splits) | 75.35% | 3.3% |\n| 15K tokens/part (optimal) | 75.35% | ~15-18% |\n| 30K tokens/part (fewer splits) | 64.16% | 28.6% |\n\n\n**Critical Finding**: Splitting enable 'divide and conquer' (insight from the organizers), but too many splits -> global pool dilution (next).\n\n---\n\n### Challenge 4: Global Dilution Problem\n\n**Definition**: Chunks surviving local filtering still face overwhelming competition in global pool.\n\n**Concrete Example** (Query q4054d9fe6d42):\n```\nConfiguration: 289 chunks, 28 relevant, using top-k=5 (because each split now has few chunks)\n3K split β 95 parts \n\nLocal Stage:\n Started: 28 relevant chunks\n Dropped: 2 chunks\n Survived: 26 relevant chunks β Sent to global pool β\n\nGlobal Pool:\n Total candidates: 95 parts Γ 5 = 475 candidates\n Signal density: 26/475 = 5.5% relevant\n Noise: 449/475 = 94.5% irrelevant\n\nFinal Re-Ranking (top-5 selection):\n Selected: 2 relevant chunks\n Dropped: 24 relevant chunks\n\nGlobal Drop Rate: 26 β 2 = 92% of surviving chunks lost at global stage\nFinal Score: 0.28\n```\n\n**Mechanism**: Surviving chunks compete in global pool. Stronger competition with 'distractors' (seemingly relevant, or weakly relevant).\n\n\n**Comparison** (same query):\n```\n30K split β 10 parts\n\nLocal Stage: ~20 chunks survive (8 dropped locally)\nGlobal Pool: 10 Γ 5 = 50 candidates\nSignal density: 20/50 = 40% relevant (vs 5.5% for 3K)\nFinal Selection: ~9 relevant chunks\nGlobal Drop Rate: 20 β 9 = 55% (better than 3K's 92%)\nFinal Score: 0.55-0.65 (2x improvement)\n```\n\n**Insight**: *Signal-to-noise ratio in later-stage pool* determines final performance. There in an inverted-u curve when it comes to splitting (assume fixed top-k, I also tried dynamic k but more issues there).\n\n---\n\n### Challenge 5: Semantic Gap\n\n**Definition**: queries require *inference* fail more when context is fragmented across splits.\n\n**Example Dataflow** (Query q2e28fdef127e):\n```\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\n\nPart 1 (chunks 0-60): Contains chunk 52 (state-level loan table)\n β LLM ranks in isolation\n β No context linking \"state distribution\" to \"expansion prospects\"\n β Chunk 52 score: 0.2 (weak relevance)\n β Dropped from local top-10\n\nPart 2 (chunks 61-120): Contains chunks about \"expansion strategy\" (conceptual discussion)\n β Different part, never sees chunk 52 data\n\nPart 3 (chunks 121-180): Contains chunks about \"investor questions\" (meeting transcript)\n β Different part, never sees chunk 52 data\n\nResult: LLM never synthesizes connection between:\n (1) Geographic loan data [Part 1]\n (2) Expansion strategy context [Part 2]\n (3) Investor view framing [Part 3]\n β Final score: 0.0 (complete failure)\n```\n\n**Impact**: Context fragmentation breaks holistic understanding needed for semantic inference in financial domain. \n\n**Hypothesis**: if chunks are small and uniform (say 50-200 tokens)\n```\nSingle-stage ranking (all 309 chunks):\n β LLM sees chunk 52 (state loan table)\n β Also sees expansion strategy discussion\n β Also sees investor question framing\n β Can infer: \"Table shows geographic presence β Relevant to expansion question\"\n β Makes final top-5\n```\nBut this is not the reality. Extreme variance found in chunks' token counts, leptokurtic (1-71K), flood context window.\n\n---\n\n### Challenge 6: Context Rot & \"Lost in the middle\" \n\n**Definition**: LLMs struggle to find needles when: (1) prompt is longer, and (2) relevant chunk is not in first-last 20% of context window.\n\n**Why Extended Context Fails**:\n```\nPrediction for Query q2e28fdef127e (309 chunks, chunk 52 relevant at position 17%):\n\nMulti-stage 30K (10 parts):\n - Chunk 52 in Part 1 with 30 chunks\n - Attention per chunk: ~3.3%\n - Score: 0.3 (low due to semantic gap)\n - Dropped locally β SCORE: 0.0\n\nExtended context (all 309 chunks):\n - Chunk 52 buried at position 52/309 (17%)\n - Attention per chunk: ~0.3% (10x dilution)\n - \"Lost in middle\" degradation + context rot\n - Score: 0.1 (even lower)\n - Dropped in final ranking β SCORE: 0.0 (NO BETTER)\n```\n\n**Attention Allocation Comparison**:\n| Strategy | Chunks/Context | Attention/Chunk | Global Pool | Global Attention |\n|----------|----------------|-----------------|-------------|------------------|\n| 30-chunk parts | 30 | 3.3% | 50 candidates | 2.0% |\n| 95-chunk parts | 95 | 1.1% | 475 candidates | 0.2% |\n| Extended (full) | 309 | 0.3% | N/A | 0.3% |\n\n\n**Empirical Validation**: Long-context models struggle, perform worse than splitting. Splitting is critical, should NOT fit everything in one query.\n\n---\n\n## 4. Architecture Trade-offs\n\n### Trade-off 1: Split Size Paradox\n\n**Intuition**: Fewer splits β Fewer local competitions β Less compound drop risk.\n\n**Reality**: Larger splits create saturated ranking tasks that degrade performance.\n\n**Experimental Evidence** (2-split performance comparison):\n| Threshold | Chunks/Part | Tokens/Part | 2-Split Local Recall |\n|-----------|-------------|-------------|----------------------|\n| 15K | ~50-70 | 15K | **80.55%** (BEST) |\n| 30K | ~80-100 | 30K | 66.07% (-14.5pts) |\n| 45K | ~100-150 | 45K | **54.41%** (WORST, -26.1pts) |\n\n\n\n**Mechanism**:\n1. **Prompt Length Effect**: 15K prompts preserve instruction retention; 45K prompts degrade clarity\n2. **Ranking Capacity**: K/N ratio critical - 10/50 (20%) captures needles better than 10/150 (7%)\n3. **Compound Drops**: More splits create more opportunities to drop, BUT better RECALL per split -> win\n\n**Optimal Point**: ~15K threshold (50-70 chunks/part) balances all three factors.\n\n---\n\n### Trade-off 2: Local Pool Paradox\n\n**Naive Solution**: Increase local top-K from 5 to 10 to capture more relevant chunks.\n\n**Why It Fails**:\n```\nExample: K=5 to K=10, given 10 equal splits\nLocal Ranking (Part-2, 18 chunks, K=10):\n Position 6: Relevant chunk, score 0.73 β CAPTURED with K=10 β (would drop with K=5)\n\n\nGlobal Ranking : 50 candidates -> 100 candidates:\n Relevant chunk STILL has score 0.73 (unchanged)\n But now competes with 90 instead of 40 chunks.\n β Relevant chunk ranks ~45th globally\n β DROPPED from final top-5 β\n```\n\n\n**Mathematical Formula**:\n```\nP(final success) = P(local survival) Γ P(global survival)\n\nIncreasing K:\n P(local survival) β (more chunks captured)\n P(global survival) β (same low scores in larger pool)\n Net effect: Minimal improvement\n```\n\n\n**Critical Insight**: We cannot improve global outcomes without changing the scores in the 2nd stage -> re-ranking using math formula rarely works. This agrees with the baseline setup from the organizers.\n\n**Solution**: Global re-scoring assigns NEW scores in full context:\n- Stage 1 (local): Chunk gets 0.73 in limited context (18 chunks)\n- Stage 2 (global): Model sees all 10 x 10 = 100 candidates, assigns NEW score 0.88 \n- Result: Chunk becomes competitive, makes final top-5\n\n**Empirical Validation**: Global re-scoring provides +5.8pts improvement over single-stage architecture.\n\n---\n\n### Trade-off 3: BM25 Pre-filtering Catastrophe\n\n**Proposed Approach**: Use BM25/embeddings to filter chunks before LLM ranking.\n\n\n**Why It Fails**:\n```\nPipeline: BM25 β Filter to top-30% β LLM ranking\n\nExample (Query q2e28fdef127e):\nQuestion: \"What investor views emerged on KeyCorp's geographic expansion prospects?\"\nChunk 52: [TABLE] \"Washington $4,605 | Ohio $2,725 | New York $822...\"\n\nBM25 Analysis:\n Question terms: [\"investor\", \"views\", \"expansion\", \"prospects\"]\n Chunk terms: [\"Washington\", \"Ohio\", \"New York\", \"$4,605\", \"$2,725\"]\n Lexical overlap: MINIMAL (only weak match on geographic terms)\n BM25 rank: ~200-250 out of 309 chunks\n\nFilter to top-30%: 93 chunks\n β Chunk 52 NOT in top-93\n β DROPPED at Stage 1\n β Never reaches LLM\n β Final score: 0.0\n```\n\n**Fundamental Error**: Financial questions require conceptual understanding:\n- \"Geographic expansion\" = loan distribution by state (tables)\n- \"Revenue concentration\" = customer segment breakdown (tables)\n- \"Margin pressure\" = cost structure details (financial data)\n\n**BM25/embeddings cannot infer these semantic connections** β Drops relevant chunks β Permanent information loss.\n\n**Conclusion**: Lexical filtering BEFORE deep reasoning is more harmful than multi-stage LLM architecture. Only LLM can perform required semantic inference.\n\n---\n\n### Trade-off 4: Extended Context Failure\n\n**Assumption**: Longer context windows (200K) solve splitting problem by processing all chunks together.\n\n**Reality**: Attention degradation worse than multi-stage processing.\n\n**Evidence** (predicted performance for 309-chunk query):\n```\nCurrent 30K split (10 parts):\n Local: 30 chunks/part, 3.3% attention per chunk\n Global: 50 candidates, 2% attention per chunk\n Average attention: ~2.5% per relevant chunk\n Observed score: ~0.4-0.6 (partial success)\n\nExtended context (309 chunks):\n Single stage: 309 chunks, 0.3% attention per chunk\n \"Lost in middle\" effect: chunks at 10-90% position get <0.2% attention\n Predicted score: ~0.3-0.4 (WORSE than multi-stage)\n```\n\n**Implication**: More context creates MORE noise, not less. The needle gets buried deep.\n\n---\n\n## Summary\n\n### Competition Constraints Drive Design\n\n**Data Characteristics**:\n- 67.5% queries exceed 32K tokens β Requires multi-part processing\n- 30.67% queries have 1-2 relevant chunks among ~200-300 candidates β Needles-in-haystack\n- 5.1% overall relevance rate β Overwhelming noisy\n\n**Technical Challenges**:\n1. Sparse signal detection in high-noise environments\n2. Context window constraints requiring splitting\n3. Multi-stage compound drops (local filtering loses global relevance)\n4. Global dilution (surviving chunks face 92% drop rate in noisy pools)\n5. Semantic gaps (inference breaks when context fragments)\n6. Context rot (attention degradation in extended contexts)\n\n### Key Trade-offs\n\n**What Fails**:\n- Extended context single-stage: Context rot degrades performance\n- BM25 pre-filtering: Drops inference-requiring chunks\n- Large split sizes: Saturated ranking (100-150 chunks/split) performs worse than moderate (50-70 chunks/split)\n- Pool expansion without re-scoring: Local scores determine global outcomes\n\n**What Works**:\n- Optimal split size (15K tokens, 50-70 chunks/part): Balances attention vs drops\n- Global re-scoring: NEW scores in full context (+5.8pts improvement)\n- Lexical-semantic fusion: *BM25 boost (not filter)* helps weak signals without dropping chunks\n- Multi-stage architecture: Better than extended context despite compound drops\n\n**Critical Design Principles**: Optimize *attention allocation* within *context rot limit* while preserving high *local recall* at filtering stage.", "3304678": "Wow, $200 for 200 hours β thatβs dedication! Thank you for sharing! I didnβt spend a penny on tokens, just fine-tuned downstream task ,but my GPU is running so hot Iβm getting roasted, and my electricity bill feels it .π", "3304682": "Nice, fine-tune with your GPU is cool. For me it was a chance to test backends also, can brag later that I win -200$ :) Did you notice issues with the labels? Some labels are out-of-context, about <1% relevant chunks have 1-3 tokens", "3305075": "Yeah, exactly β because the text is split into chunks, some parts seem irrelevant ,only make sense with more context and domain knowledge. judging relevance just from one chunk is tricky. I wanted to try adding chunk position and some context info to the modelβs encoding, but I didnβt have enough time before the deadline.\n\nAnother challenge is that the content isnβt just about finance β it also includes topics like Apple, healthcare, and many other companies and area , So finance domain isnβt enough to understand the overall context." }, "source": "meta" }