Technical notes on AI, large language models, and research methods for finance and accounting researchers.
Wed, Jun 3, 2026
Temperature zero is not the same thing as reproducibility. Repeated LLM API calls can produce different labels even with the same prompt, model, seed, and decoding parameters. LLM-based measurement needs stability checks and disclosure.
Read more →
Tue, May 5, 2026
The things we add to AI systems to make them safer can make them worse. OpenAI’s goblin problem and Anthropic’s Claude Code postmortem show why hidden guardrails need versioning, regression testing, and monitoring.
Read more →
Fri, Mar 20, 2026
I asked eight AI models to count subsidiaries in GE Aerospace’s SEC filing. Answers ranged from 14 to 228. The correct answer is 80. The failures reveal how retrieval, chunking, and attention windows silently corrupt financial data extraction.
Read more →
Wed, Dec 10, 2025
Adding timestamps for audit trails drops accuracy by 10%. The compliance mechanism undermines the output being audited. This counterintuitive finding reveals how irrelevant context—even seemingly harmless metadata—can degrade model performance. For regulated industries requiring audit logs, this creates a fundamental tension.
Read more →
Wed, Nov 26, 2025
Users complain GPT models get worse over time. OpenAI denies it. I designed 43 tests and ran them repeatedly. The results show a clear U-shaped pattern: GPT-4 started at 86% accuracy, dropped to 46.5% with GPT-4-turbo, then recovered. Teams need continuous evaluation, not blind trust in a model label.
Read more →
Wed, Nov 19, 2025
LLMs ace easy sums but fail at 4-digit multiplication. This isn’t a bug—it’s how tokenization works. For finance, audit, and tax teams, understanding these limitations is critical. The model that writes your report may silently miscalculate totals. I tested GPT-4 on arithmetic and the results reveal systematic patterns of failure.
Read more →
Sun, Nov 9, 2025
Chain-of-Thought prompting encourages LLMs to generate intermediate reasoning steps before producing a final answer. This technique, pioneered by Wei et al. (2022), allows large models to solve complex problems more accurately by breaking them into manageable steps. For finance and accounting researchers, CoT is particularly valuable for tasks requiring multi-step analysis.
Read more →