lora learns less of a target domain than full fine-tuning (fft) (evaluated on code and math; humaneval and gsm8k) but this difference is not consistent across domain or training budget. naturally, lora forgets less of the source domain. lora seemingly represents a smaller perturbance of the original weights. given the above; does LoRA and fft represent different tradeoffs between learning and forgetting? no. they occupy the same pareto curve. i would have liked to see a compute comparison at this stage, to complement the trade-off analysis. depending on your desired...

Many people often ask me why I chose to become a research engineer rather than a software engineer, and what the difference between the two roles is. Besides the obvious point that we read a lot of papers—which, by the way, is something I think software engineers should get more accustomed to— the aspect I value most about research engineering is our relationship to code. Now, I say this as someone someone with a background in software engineering. My bachelor is a classic CS degree and I only discovered my...

--- A model with 100M parameters storing 220M bits of knowledge has a capacity ratio of 2.2 bits per parameter. This knowledge does not refer to word-for-word memorization but rather knowledge that is extractable and applicable to downstream tasks. --- Statement 1. GPT-2 consistently achieves a 2bit/param capacity ratio across all data settings after sufficient training. This holds across a wide variety of model configurations (size, depth, width, etc..) Statement 2. Most LLM architectures achieve this 2bit/param ratio if sufficiently trained, even with MLP layers removed, this is universal....