Teppei Nakano — Reading notes

Teppei Nakano — Reading notes https://kekeke29341.github.io/reading/en.html One takeaway per paper for local LLMs and medical ML en Lost in the Middle: How Language Models Use Long Contexts https://kekeke29341.github.io/reading/note.html?slug=lost-in-the-middle&lang=en https://kekeke29341.github.io/reading/note.html?slug=lost-in-the-middle&lang=en Sat, 15 Aug 2026 12:00:00 +0000 The middle of a long context is unused. RAG either puts evidence at the ends or shortens the context. The shaky foundations of large language models and foundation models for electronic health records https://kekeke29341.github.io/reading/note.html?slug=wornow-shaky-foundations&lang=en https://kekeke29341.github.io/reading/note.html?slug=wornow-shaky-foundations&lang=en Sat, 15 Aug 2026 12:00:00 +0000 Before the task name, I check whether EHR foundation-model labels are billing or annotation, and whether the split is by patient. Large language models encode clinical knowledge https://kekeke29341.github.io/reading/note.html?slug=med-palm&lang=en https://kekeke29341.github.io/reading/note.html?slug=med-palm&lang=en Fri, 14 Aug 2026 12:00:00 +0000 A board-exam score is not faithfulness to a ward protocol. A medical LLM decides citations and what it will not emit first. AI for radiographic COVID-19 detection selects shortcuts over signal https://kekeke29341.github.io/reading/note.html?slug=degrave-covid-shortcuts&lang=en https://kekeke29341.github.io/reading/note.html?slug=degrave-covid-shortcuts&lang=en Fri, 14 Aug 2026 12:00:00 +0000 Imaging models pick laterality markers and site style before disease. Before saliency, I ask whether the embedding predicts the hospital. Do We Still Need Clinical Language Models? https://kekeke29341.github.io/reading/note.html?slug=lehman-clinical-lms&lang=en https://kekeke29341.github.io/reading/note.html?slug=lehman-clinical-lms&lang=en Thu, 13 Aug 2026 12:00:00 +0000 A general LLM can win a clinical task and still be unusable if notes cannot leave. The boundary precedes the model name. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs https://kekeke29341.github.io/reading/note.html?slug=zech-pneumonia-2018&lang=en https://kekeke29341.github.io/reading/note.html?slug=zech-pneumonia-2018&lang=en Thu, 13 Aug 2026 12:00:00 +0000 In-hospital AUROC dies next door. I read pneumonia papers for scanner and protocol text before the leaderboard. Capabilities of GPT-4 on Medical Challenge Problems https://kekeke29341.github.io/reading/note.html?slug=nori-gpt4-medical&lang=en https://kekeke29341.github.io/reading/note.html?slug=nori-gpt4-medical&lang=en Wed, 12 Aug 2026 12:00:00 +0000 A challenge-problem score is not EHR input and not an accountability story. I read it as an upper bound on a demo. Dissecting racial bias in an algorithm used to manage the health of populations https://kekeke29341.github.io/reading/note.html?slug=obermeyer-bias-2019&lang=en https://kekeke29341.github.io/reading/note.html?slug=obermeyer-bias-2019&lang=en Wed, 12 Aug 2026 12:00:00 +0000 Cost as a proxy for risk learns access. I write what the teacher stands in for before I write the loss. GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records https://kekeke29341.github.io/reading/note.html?slug=gatortron&lang=en https://kekeke29341.github.io/reading/note.html?slug=gatortron&lang=en Tue, 11 Aug 2026 12:00:00 +0000 The point of a large clinical LM is notes trained inside the institution. I look at weight release and the teacher first. The myth of generalisability in clinical research and machine learning in health care https://kekeke29341.github.io/reading/note.html?slug=futoma-generalisability&lang=en https://kekeke29341.github.io/reading/note.html?slug=futoma-generalisability&lang=en Tue, 11 Aug 2026 12:00:00 +0000 Generalisation is a report of which shifts you survived, not a property. I do not call a single AUROC generalisable. Publicly Available Clinical BERT Embeddings https://kekeke29341.github.io/reading/note.html?slug=clinical-bert-alsentzer&lang=en https://kekeke29341.github.io/reading/note.html?slug=clinical-bert-alsentzer&lang=en Mon, 10 Aug 2026 12:00:00 +0000 Public clinical BERT leans toward MIMIC prose. I do not freeze the embedding without a local-note re-eval. Scalable and accurate deep learning with electronic health records https://kekeke29341.github.io/reading/note.html?slug=rajkomar-ehr-2018&lang=en https://kekeke29341.github.io/reading/note.html?slug=rajkomar-ehr-2018&lang=en Mon, 10 Aug 2026 12:00:00 +0000 Ingesting the EHR as a sequence is not the result. The result is whether anything after time t leaked in. BioBERT: a pre-trained biomedical language representation model for biomedical text mining https://kekeke29341.github.io/reading/note.html?slug=biobert&lang=en https://kekeke29341.github.io/reading/note.html?slug=biobert&lang=en Sun, 09 Aug 2026 12:00:00 +0000 BERT on PubMed. That is paper English, not a ward note. I do not confuse corpora by name. MIMIC-III, a freely accessible critical care database https://kekeke29341.github.io/reading/note.html?slug=mimic-iii&lang=en https://kekeke29341.github.io/reading/note.html?slug=mimic-iii&lang=en Sun, 09 Aug 2026 12:00:00 +0000 The default public ICU set. I read which MIMIC version, whether patients collide, and whether clocks agree. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging https://kekeke29341.github.io/reading/note.html?slug=oakden-hidden-stratification&lang=en https://kekeke29341.github.io/reading/note.html?slug=oakden-hidden-stratification&lang=en Sat, 08 Aug 2026 12:00:00 +0000 Rare subtypes sink under mean AUROC. If the stratum is not a label, the mean hides it. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison https://kekeke29341.github.io/reading/note.html?slug=chexpert-irvin&lang=en https://kekeke29341.github.io/reading/note.html?slug=chexpert-irvin&lang=en Sat, 08 Aug 2026 12:00:00 +0000 The uncertain label (pos / uncertain / neg) is the dataset. Collapse it to positive and the model never learns radiologist doubt. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports https://kekeke29341.github.io/reading/note.html?slug=mimic-cxr&lang=en https://kekeke29341.github.io/reading/note.html?slug=mimic-cxr&lang=en Fri, 07 Aug 2026 12:00:00 +0000 A public set with images and reports. Split by patient. Do not mix acquisition time and report time. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets https://kekeke29341.github.io/reading/note.html?slug=saito-pr-2015&lang=en https://kekeke29341.github.io/reading/note.html?slug=saito-pr-2015&lang=en Fri, 07 Aug 2026 12:00:00 +0000 On imbalance, PR comes first. AUROC can look high while almost every page is a miss. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations https://kekeke29341.github.io/reading/note.html?slug=seyyed-underdiagnosis&lang=en https://kekeke29341.github.io/reading/note.html?slug=seyyed-underdiagnosis&lang=en Thu, 06 Aug 2026 12:00:00 +0000 Under mean AUROC, false negatives rise in under-served groups. I do not ship a chest model I cannot slice. Tackling the widespread and critical impact of batch effects in high-throughput data https://kekeke29341.github.io/reading/note.html?slug=leek-batch-2010&lang=en https://kekeke29341.github.io/reading/note.html?slug=leek-batch-2010&lang=en Thu, 06 Aug 2026 12:00:00 +0000 Batch can outrun biology. I put a cross-study split before the correction. Deep learning predicts hip fracture using confounding patient and healthcare variables https://kekeke29341.github.io/reading/note.html?slug=badgeley-hip-confound&lang=en https://kekeke29341.github.io/reading/note.html?slug=badgeley-hip-confound&lang=en Wed, 05 Aug 2026 12:00:00 +0000 It looks like fracture and is hospital and scanner. I plot confounders before I print accuracy. Adjusting batch effects in microarray expression data using empirical Bayes methods https://kekeke29341.github.io/reading/note.html?slug=combat-johnson-2007&lang=en https://kekeke29341.github.io/reading/note.html?slug=combat-johnson-2007&lang=en Wed, 05 Aug 2026 12:00:00 +0000 ComBat shrinks batch mean and variance. Re-estimating on the test study leaks the evaluation. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) https://kekeke29341.github.io/reading/note.html?slug=tripod-collins&lang=en https://kekeke29341.github.io/reading/note.html?slug=tripod-collins&lang=en Tue, 04 Aug 2026 12:00:00 +0000 Reporting items come first. A paper without split, calibration, and population cannot be reproduced. A simple algorithm for identifying negated findings and diseases in discharge summaries https://kekeke29341.github.io/reading/note.html?slug=chapman-negex&lang=en https://kekeke29341.github.io/reading/note.html?slug=chapman-negex&lang=en Tue, 04 Aug 2026 12:00:00 +0000 Half of clinical text is absence. Extraction that ignores negation doubles the findings. Efficient Memory Management for Large Language Model Serving with PagedAttention https://kekeke29341.github.io/reading/note.html?slug=pagedattention&lang=en https://kekeke29341.github.io/reading/note.html?slug=pagedattention&lang=en Mon, 03 Aug 2026 12:00:00 +0000 Contiguous KV leaves holes. Paging the cache is how the same VRAM holds a batch. Decision Curve Analysis: A Novel Method for Evaluating Prediction Models https://kekeke29341.github.io/reading/note.html?slug=vickers-dca&lang=en https://kekeke29341.github.io/reading/note.html?slug=vickers-dca&lang=en Mon, 03 Aug 2026 12:00:00 +0000 Plot net benefit against the treat / do-not-treat threshold. A high AUROC that loses at the operating threshold does not ship. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness https://kekeke29341.github.io/reading/note.html?slug=flashattention&lang=en https://kekeke29341.github.io/reading/note.html?slug=flashattention&lang=en Sun, 02 Aug 2026 12:00:00 +0000 Attention is slow from HBM traffic, not from approximation. Tile it; keep it exact. Multitask learning and benchmarking with clinical time series data https://kekeke29341.github.io/reading/note.html?slug=harutyunyan-mimic&lang=en https://kekeke29341.github.io/reading/note.html?slug=harutyunyan-mimic&lang=en Sun, 02 Aug 2026 12:00:00 +0000 The MIMIC benchmark is the task definition and the split. I do not silently copy that preprocessing onto my own task. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints https://kekeke29341.github.io/reading/note.html?slug=gqa&lang=en https://kekeke29341.github.io/reading/note.html?slug=gqa&lang=en Sat, 01 Aug 2026 12:00:00 +0000 Sharing K/V heads cuts KV bytes before it cuts quality. That is the local-inference estimate. Do no harm: a roadmap for responsible machine learning for health care https://kekeke29341.github.io/reading/note.html?slug=wiens-do-no-harm&lang=en https://kekeke29341.github.io/reading/note.html?slug=wiens-do-no-harm&lang=en Sat, 01 Aug 2026 12:00:00 +0000 Responsibility is who can stop it, not a model card. I write the off-switch before deployment. RoFormer: Enhanced Transformer with Rotary Position Embedding https://kekeke29341.github.io/reading/note.html?slug=roformer&lang=en https://kekeke29341.github.io/reading/note.html?slug=roformer&lang=en Fri, 31 Jul 2026 12:00:00 +0000 Positions are rotated, not added. Relative offset enters the dot product. Long-context work is about those periods. The false hope of current approaches to explainable artificial intelligence in health care https://kekeke29341.github.io/reading/note.html?slug=ghassemi-xai-false-hope&lang=en https://kekeke29341.github.io/reading/note.html?slug=ghassemi-xai-false-hope&lang=en Fri, 31 Jul 2026 12:00:00 +0000 Saliency is not evidence of generalization. I print external-site and slice failures before an explanation figure. Efficient Streaming Language Models with Attention Sinks https://kekeke29341.github.io/reading/note.html?slug=attention-sinks&lang=en https://kekeke29341.github.io/reading/note.html?slug=attention-sinks&lang=en Thu, 30 Jul 2026 12:00:00 +0000 A sliding window breaks unless you keep the first tokens. The sink is a lemma of the window. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2 https://kekeke29341.github.io/reading/note.html?slug=deseq2&lang=en https://kekeke29341.github.io/reading/note.html?slug=deseq2&lang=en Thu, 30 Jul 2026 12:00:00 +0000 Shrink dispersion, then test a difference. I do not hand raw counts to a learner. Fast Inference from Transformers via Speculative Decoding https://kekeke29341.github.io/reading/note.html?slug=speculative-decoding&lang=en https://kekeke29341.github.io/reading/note.html?slug=speculative-decoding&lang=en Wed, 29 Jul 2026 12:00:00 +0000 A small model drafts; the large one verifies. Wall time shrinks only when rejects stay rare. Fast, sensitive and accurate integration of single-cell data with Harmony https://kekeke29341.github.io/reading/note.html?slug=harmony-korsunsky&lang=en https://kekeke29341.github.io/reading/note.html?slug=harmony-korsunsky&lang=en Wed, 29 Jul 2026 12:00:00 +0000 Integration pulls batches together and should leave biology. I do not put the test study into the integration fit. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale https://kekeke29341.github.io/reading/note.html?slug=llm-int8&lang=en https://kekeke29341.github.io/reading/note.html?slug=llm-int8&lang=en Tue, 28 Jul 2026 12:00:00 +0000 Even at 8-bit, outlier channels stay in 16-bit. Quantization is not a uniform shave. Current best practices in single-cell RNA-seq analysis: a tutorial https://kekeke29341.github.io/reading/note.html?slug=luecken-theis-scrna&lang=en https://kekeke29341.github.io/reading/note.html?slug=luecken-theis-scrna&lang=en Tue, 28 Jul 2026 12:00:00 +0000 In single-cell, the preprocessing order is the result. I freeze QC, normalize, integrate, cluster before the learner. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration https://kekeke29341.github.io/reading/note.html?slug=awq&lang=en https://kekeke29341.github.io/reading/note.html?slug=awq&lang=en Mon, 27 Jul 2026 12:00:00 +0000 Weight importance follows activation magnitude. MSE alone drops the channels you needed. Characterizing genetic intra-tumor heterogeneity in 500 tumors from 38 tissues https://kekeke29341.github.io/reading/note.html?slug=dentro-ith&lang=en https://kekeke29341.github.io/reading/note.html?slug=dentro-ith&lang=en Mon, 27 Jul 2026 12:00:00 +0000 One biopsy is a piece of the tumor. I split heterogeneity into diversity inside the sample and extrapolation to unsampled regions. MIMIC-IV, a freely accessible electronic health record dataset https://kekeke29341.github.io/reading/note.html?slug=mimic-iv&lang=en https://kekeke29341.github.io/reading/note.html?slug=mimic-iv&lang=en Sun, 26 Jul 2026 12:00:00 +0000 The successor to III. Write version and module. Do not silently reuse III preprocessing on IV. LoRA: Low-Rank Adaptation of Large Language Models https://kekeke29341.github.io/reading/note.html?slug=lora&lang=en https://kekeke29341.github.io/reading/note.html?slug=lora&lang=en Sun, 26 Jul 2026 12:00:00 +0000 Add a low-rank delta. Choose which matrices before you choose rank. QLoRA: Efficient Finetuning of Quantized Language Models https://kekeke29341.github.io/reading/note.html?slug=qlora&lang=en https://kekeke29341.github.io/reading/note.html?slug=qlora&lang=en Sat, 25 Jul 2026 12:00:00 +0000 Add LoRA while the base sits in 4-bit. Close to the air-gapped default when full finetune VRAM is gone. Direct Preference Optimization: Your Language Model is Secretly a Reward Model https://kekeke29341.github.io/reading/note.html?slug=dpo&lang=en https://kekeke29341.github.io/reading/note.html?slug=dpo&lang=en Sat, 25 Jul 2026 12:00:00 +0000 Given preference pairs, you can update the policy without a reward model. Pair quality precedes the loss. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers https://kekeke29341.github.io/reading/note.html?slug=gptq&lang=en https://kekeke29341.github.io/reading/note.html?slug=gptq&lang=en Fri, 24 Jul 2026 12:00:00 +0000 Quantize weights with no training. Calibration text precedes the bit width. Training language models to follow instructions with human feedback https://kekeke29341.github.io/reading/note.html?slug=instructgpt&lang=en https://kekeke29341.github.io/reading/note.html?slug=instructgpt&lang=en Fri, 24 Jul 2026 12:00:00 +0000 SFT, then reinforce from human preference. The template for “aligning” a local stack. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning https://kekeke29341.github.io/reading/note.html?slug=flashattention-2&lang=en https://kekeke29341.github.io/reading/note.html?slug=flashattention-2&lang=en Thu, 23 Jul 2026 12:00:00 +0000 On top of FA1’s IO cut, fix how work is partitioned. I check this is the default kernel before I talk about approximating attention. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity https://kekeke29341.github.io/reading/note.html?slug=switch-transformers&lang=en https://kekeke29341.github.io/reading/note.html?slug=switch-transformers&lang=en Thu, 23 Jul 2026 12:00:00 +0000 Each token picks one expert. Compute is sparse; memory still pays for every expert. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks https://kekeke29341.github.io/reading/note.html?slug=rag-lewis&lang=en https://kekeke29341.github.io/reading/note.html?slug=rag-lewis&lang=en Wed, 22 Jul 2026 12:00:00 +0000 Retrieve before you generate. Do not bake the knowledge into weights. Score retrieval and generation apart. Fast Transformer Decoding: One Write-Head is All You Need https://kekeke29341.github.io/reading/note.html?slug=mqa-shazeer&lang=en https://kekeke29341.github.io/reading/note.html?slug=mqa-shazeer&lang=en Wed, 22 Jul 2026 12:00:00 +0000 One K/V head slashes KV bytes. The prehistory of GQA. I write the quality trade next to the memory estimate. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation https://kekeke29341.github.io/reading/note.html?slug=alibi&lang=en https://kekeke29341.github.io/reading/note.html?slug=alibi&lang=en Tue, 21 Jul 2026 12:00:00 +0000 Add a linear bias on distance. A different path to length extrapolation than RoPE. I do not mix the recipes. Dense Passage Retrieval for Open-Domain Question Answering https://kekeke29341.github.io/reading/note.html?slug=dpr&lang=en https://kekeke29341.github.io/reading/note.html?slug=dpr&lang=en Tue, 21 Jul 2026 12:00:00 +0000 Put questions and passages in one space. Before dropping BM25, I look at what dense retrieval catches and drops. YaRN: Efficient Context Window Extension of Large Language Models https://kekeke29341.github.io/reading/note.html?slug=yarn&lang=en https://kekeke29341.github.io/reading/note.html?slug=yarn&lang=en Mon, 20 Jul 2026 12:00:00 +0000 Stretch RoPE frequencies by band. In operations I try this before raising the base in one shot. Attention Is All You Need https://kekeke29341.github.io/reading/note.html?slug=vaswani-2017&lang=en https://kekeke29341.github.io/reading/note.html?slug=vaswani-2017&lang=en Mon, 20 Jul 2026 12:00:00 +0000 Drop recurrence; make distance constant with attention. Current decoders keep one side of this paper. Training Compute-Optimal Large Language Models https://kekeke29341.github.io/reading/note.html?slug=chinchilla&lang=en https://kekeke29341.github.io/reading/note.html?slug=chinchilla&lang=en Sun, 19 Jul 2026 12:00:00 +0000 For a fixed compute budget, more tokens beat an oversized model. I use it to size an air-gapped pretrain. Decoupled Weight Decay Regularization https://kekeke29341.github.io/reading/note.html?slug=adamw&lang=en https://kekeke29341.github.io/reading/note.html?slug=adamw&lang=en Sun, 19 Jul 2026 12:00:00 +0000 Decouple decay from the gradient. That is not Adam plus L2. Leave Norm γ out of decay. Scaling Laws for Neural Language Models https://kekeke29341.github.io/reading/note.html?slug=kaplan-scaling&lang=en https://kekeke29341.github.io/reading/note.html?slug=kaplan-scaling&lang=en Sat, 18 Jul 2026 12:00:00 +0000 Loss falls as a power of size, data, and compute. The optimal split was revised later by Chinchilla. Root Mean Square Layer Normalization https://kekeke29341.github.io/reading/note.html?slug=rmsnorm&lang=en https://kekeke29341.github.io/reading/note.html?slug=rmsnorm&lang=en Sat, 18 Jul 2026 12:00:00 +0000 No mean subtraction; divide by RMS. With pre-norm, that is current decoder normalization. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models https://kekeke29341.github.io/reading/note.html?slug=zero-deepspeed&lang=en https://kekeke29341.github.io/reading/note.html?slug=zero-deepspeed&lang=en Fri, 17 Jul 2026 12:00:00 +0000 Shard optimizer state across GPUs. Whether an air-gapped train fits is the ZeRO stage, not the parameter count. On Layer Normalization in the Transformer Architecture https://kekeke29341.github.io/reading/note.html?slug=preln-xiong&lang=en https://kekeke29341.github.io/reading/note.html?slug=preln-xiong&lang=en Fri, 17 Jul 2026 12:00:00 +0000 Pre-norm leaves the residual as an identity. That is one reason a deep decoder trains. The Curious Case of Neural Text Degeneration https://kekeke29341.github.io/reading/note.html?slug=nucleus-sampling&lang=en https://kekeke29341.github.io/reading/note.html?slug=nucleus-sampling&lang=en Thu, 16 Jul 2026 12:00:00 +0000 Argmax loops. Temperature is not enough. Cut the tail with a nucleus (top-p). Mixed Precision Training https://kekeke29341.github.io/reading/note.html?slug=mixed-precision&lang=en https://kekeke29341.github.io/reading/note.html?slug=mixed-precision&lang=en Thu, 16 Jul 2026 12:00:00 +0000 Compute in FP16; keep an FP32 copy for stability. I do not mix this with inference quantization. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT https://kekeke29341.github.io/reading/note.html?slug=colbert&lang=en https://kekeke29341.github.io/reading/note.html?slug=colbert&lang=en Wed, 15 Jul 2026 12:00:00 +0000 Late interaction over tokens. Higher than one vector, heavier to store. A middle option for internal search. Distilling the Knowledge in a Neural Network https://kekeke29341.github.io/reading/note.html?slug=hinton-distill&lang=en https://kekeke29341.github.io/reading/note.html?slug=hinton-distill&lang=en Wed, 15 Jul 2026 12:00:00 +0000 Move the shape of the teacher’s logits, with a temperature. Relations survive better than hard labels. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering https://kekeke29341.github.io/reading/note.html?slug=fid-izacard&lang=en https://kekeke29341.github.io/reading/note.html?slug=fid-izacard&lang=en Tue, 14 Jul 2026 12:00:00 +0000 Encode passages separately; fuse in the decoder. Attribution is easier than one long concatenated prompt. On Calibration of Modern Neural Networks https://kekeke29341.github.io/reading/note.html?slug=guo-calibration&lang=en https://kekeke29341.github.io/reading/note.html?slug=guo-calibration&lang=en Tue, 14 Jul 2026 12:00:00 +0000 Deeper, stronger nets are worse calibrated. I put ECE and temperature scaling next to AUROC. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena https://kekeke29341.github.io/reading/note.html?slug=llm-as-judge&lang=en https://kekeke29341.github.io/reading/note.html?slug=llm-as-judge&lang=en Mon, 13 Jul 2026 12:00:00 +0000 LLM judges are biased by position and self-preference. A partial substitute for humans, not the only KPI. Representation Learning with Contrastive Predictive Coding https://kekeke29341.github.io/reading/note.html?slug=cpc-infonce&lang=en https://kekeke29341.github.io/reading/note.html?slug=cpc-infonce&lang=en Mon, 13 Jul 2026 12:00:00 +0000 Pick the positive out of noise. InfoNCE became the name of that loss. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback https://kekeke29341.github.io/reading/note.html?slug=hh-rlhf&lang=en https://kekeke29341.github.io/reading/note.html?slug=hh-rlhf&lang=en Sun, 12 Jul 2026 12:00:00 +0000 Helpful and harmless do not rise on one reward. I write which preference the data prioritized. Neural Machine Translation of Rare Words with Subword Units https://kekeke29341.github.io/reading/note.html?slug=bpe-sennrich&lang=en https://kekeke29341.github.io/reading/note.html?slug=bpe-sennrich&lang=en Sun, 12 Jul 2026 12:00:00 +0000 Split rare words into subwords. The tokenizer is part of the model, not preprocessing. Constitutional AI: Harmlessness from AI Feedback https://kekeke29341.github.io/reading/note.html?slug=constitutional-ai&lang=en https://kekeke29341.github.io/reading/note.html?slug=constitutional-ai&lang=en Sat, 11 Jul 2026 12:00:00 +0000 Write a constitution first; let the model critique itself. That cuts human pairs. RLHF without principles becomes a department’s taste. Language Models are Few-Shot Learners https://kekeke29341.github.io/reading/note.html?slug=gpt3-brown&lang=en https://kekeke29341.github.io/reading/note.html?slug=gpt3-brown&lang=en Fri, 10 Jul 2026 12:00:00 +0000 In-context examples steer without a weight update. Demos are strong and weak to contamination and secret exemplars. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding https://kekeke29341.github.io/reading/note.html?slug=bert-devlin&lang=en https://kekeke29341.github.io/reading/note.html?slug=bert-devlin&lang=en Thu, 09 Jul 2026 12:00:00 +0000 A bidirectional understanding model. Masking and KV growth are not a decoder’s. I do not mix the names. Measuring Massive Multitask Language Understanding https://kekeke29341.github.io/reading/note.html?slug=mmlu&lang=en https://kekeke29341.github.io/reading/note.html?slug=mmlu&lang=en Wed, 08 Jul 2026 12:00:00 +0000 A broad multiple-choice set is both a contamination risk and a poor proxy for the floor task. Not the only hiring number. Llama 2: Open Foundation and Fine-Tuned Chat Models https://kekeke29341.github.io/reading/note.html?slug=llama2&lang=en https://kekeke29341.github.io/reading/note.html?slug=llama2&lang=en Tue, 07 Jul 2026 12:00:00 +0000 A public-weight starting point for an air gap. I read the license and the chat post-train recipe before the pretrain story. Mistral 7B https://kekeke29341.github.io/reading/note.html?slug=mistral-7b&lang=en https://kekeke29341.github.io/reading/note.html?slug=mistral-7b&lang=en Mon, 06 Jul 2026 12:00:00 +0000 GQA and a sliding window make 7B cheaper. One comparison point for a small air-gapped model. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models https://kekeke29341.github.io/reading/note.html?slug=chain-of-thought&lang=en https://kekeke29341.github.io/reading/note.html?slug=chain-of-thought&lang=en Sun, 05 Jul 2026 12:00:00 +0000 Writing intermediate steps helps arithmetic and logic. In medicine a hallucinated procedure can still look well-formed. GLU Variants Improve Transformer https://kekeke29341.github.io/reading/note.html?slug=swiglu&lang=en https://kekeke29341.github.io/reading/note.html?slug=swiglu&lang=en Sat, 04 Jul 2026 12:00:00 +0000 Gate the FFN. Llama-family MLPs are this line. I do not borrow a ReLU-FFN learning rate. Evaluating Large Language Models Trained on Code https://kekeke29341.github.io/reading/note.html?slug=humaneval&lang=en https://kekeke29341.github.io/reading/note.html?slug=humaneval&lang=en Fri, 03 Jul 2026 12:00:00 +0000 Unit tests passing is the code-generation eval. Internally those tests can be the secret.