Boosting Large Language Model Results with Human Comparison Data

Capabilities of large language models (LLMs) have progressed rapidly, but the key ideas surrounding LLMs have their roots in research from years before! In fact, the training pipeline used by LLMs (pretraining -> SFT -> RLHF) was originally proposed in 2020 for training abstractive summarization models... Supervised learning for summarization: Prior to the popularization of LLMs, most summarization models were trained in a supervised fashion where we: 1. Perform self-supervised pretraining. 2. Perform supervised finetuning (SFT) over summary examples. Humans don't always write perfect summaries and summarization is a very open-ended task (i.e., one document can be summarized in many different ways that are equally valid). As such, training a summarization model to exactly match a human-written summary is a flawed approach! Learning from human feedback: Inspired by these issues, authors in [1] propose a three-part framework that enables LLMs to be finetuned based on human preference data. The LLM is first pretrained then finetuned with SFT. Afterwards, we collect data for reinforcement learning from human feedback (RLHF) as follows: 1. Take a textual input (source document) from the training dataset. 2. Use several policies (e.g., pretrained model, supervised baseline, current model, or the human reference summary) to sample summaries of the input. 3. Pick two summaries from the set of samples responses. 4. Ask a human annotator to identify the better of the two summaries. Once the data has been collected, we use this comparison data to train a reward model (using a pairwise ranking loss) that accurately predicts a human preference score given a summary produced by the LLM. From here, we use PPO (an RL algorithm) to finetune the model based on preference scores outputted by the reward model. Avoiding drift: The authors in [1] add a KL divergence term to the objective being optimized by PPO, which penalizes the policy from becoming too different from the supervised baseline policy during RLHF. Such an approach, which is now commonly used (e.g., see Eq. 4 in the LLaMA-2 report), encourages exploration without mode collapse and prevents summaries written by the LLM from becoming too different from those that are seen during training. Is learning from feedback effective? After training a 1.3B and 6.7B GPT-style LLM with this approach, we see that 1.3B human feedback model outperforms a 10× larger model trained with supervised learning alone, and the 6.7B human feedback model performs even better than the 1.3B model. These models are consistently preferred to those trained via SFT alone. From summarization to ChatGPT: The approach proposed in [1] is not just useful for summarization! This strategy for learning from human feedback was extended by InstructGPT [2] (the sister model and predecessor to ChatGPT) to align foundation language models. The training strategy used in [2] is nearly identical to the training framework proposed in [1]!

Exciting Research Alert: Improving RAG with Self-Generated Demonstrations I just came across a fascinating paper that addresses a critical challenge in Retrieval-Augmented Generation (RAG) systems. The research team from USC and Meta has developed a novel approach called Self-Demo Retrieval-Augmented Instruction Tuning (SD-RA-IT) that significantly improves how Large Language Models (LLMs) handle retrieved information. >> The Problem They're Solving When fine-tuning LLMs for RAG, we typically use human-written responses that weren't created with the retrievals in mind. This creates two major issues: - Misalignment between retrievals and responses - Training on out-of-distribution (OOD) text that the model wouldn't naturally generate These issues lead to hallucinations and poor performance in RAG systems. >> Their Innovative Solution The researchers propose a brilliantly simple yet effective approach: 1. Generate multiple response candidates from the LLM itself using the instruction and retrievals 2. Filter these self-generated responses for correctness against gold answers 3. Train the model on these in-distribution self-demos instead of human-written responses 4. When no good response is found, train the model to generate a refusal This method ensures the training data matches the model's own distribution while still providing accurate supervision. >> Technical Implementation Details Their implementation uses Llama-3-8B-Instruct and Llama-3-70B-Instruct models with fairseq2 for training and vLLM for inference. They employ automatic prompt optimization to generate diverse response candidates and use a tournament-style filtering process to select the best responses. The team evaluated two training objectives: - Supervised fine-tuning (SFT) with cross-entropy loss - Direct preference optimization (DPO) using rejected responses as negative examples >> Impressive Results The results speak for themselves: - Higher precision (accuracy on attempted questions) - Higher recall (successful attempts on answerable questions) - Lower counterfactual accuracy (better at refusing questions it would get wrong) - Minimal degradation in non-RAG settings - Superior performance across different numbers of retrievals Most importantly, SD-RA-IT models are significantly better at avoiding hallucinations by refusing to answer questions they're likely to get wrong, while still correctly extracting answers from relevant retrievals. This research provides valuable insights for anyone working with RAG systems. By training on self-generated demonstrations, we can create more reliable and accurate RAG-enabled language models.

🧪 New Machine Learning Research: exploring new methods in AI alignment! Karel D'Oosterlinck, Chris Develder, Thomas Demeester, Amanpreet Singh and their colleagues from Ghent University, in collaboration with Stanford University and Contextual AI have introduced two novel approaches, CLAIR and APO, to improve the alignment of large language models (LLMs) with human preferences. - Research goal: The study aims to address the challenges in aligning LLMs by enhancing the contrastiveness of preference data and refining alignment objectives to achieve better performance and more human-like outputs. - Research methodology: The team developed Contrastive Learning from AI Revisions (CLAIR), which creates minimally contrasting preference pairs, and Anchored Preference Optimization (APO), a set of alignment objectives that better control the alignment process. They evaluated these methods on the Llama-3-8B-Instruct model using different alignment datasets and objectives. - Key findings: The CLAIR method improved performance by 7.65% on the MixEval-Hard benchmark, closing the gap with GPT4-turbo by 45%. APO consistently outperformed traditional objectives, showcasing significant improvements in preference alignment. - Practical implications: These advancements can reduce the training cost and time for aligning LLMs while improving their adherence to human preferences. For instance, using CLAIR and APO can enhance applications like AI-assisted writing tools, customer service bots, and personalized content generation, making them more reliable and effective. #LabelYourData #LanguageModels #AIAlignment #MachineLearning #Innovation #AIResearch #MLResearch 

𝘔𝘢𝘯𝘺 "𝘰𝘱𝘦𝘯" 𝘝𝘪𝘴𝘪𝘰𝘯-𝘓𝘢𝘯𝘨𝘶𝘢𝘨𝘦 𝘔𝘰𝘥𝘦𝘭𝘴 𝘢𝘳𝘦 𝘫𝘶𝘴𝘵 𝘦𝘤𝘩𝘰𝘦𝘴 𝘰𝘧 𝘱𝘳𝘰𝘱𝘳𝘪𝘦𝘵𝘢𝘳𝘺 𝘮𝘰𝘥𝘦𝘭𝘴, 𝘵𝘳𝘢𝘪𝘯𝘦𝘥 𝘣𝘺 𝘥𝘪𝘴𝘵𝘪𝘭𝘭𝘪𝘯𝘨 𝘵𝘩𝘦𝘪𝘳 𝘴𝘺𝘯𝘵𝘩𝘦𝘵𝘪𝘤 𝘥𝘢𝘵𝘢. 𝘞𝘩𝘢𝘵 𝘪𝘧 𝘸𝘦 𝘤𝘰𝘶𝘭𝘥 𝘣𝘳𝘦𝘢𝘬 𝘵𝘩𝘢𝘵 𝘤𝘺𝘤𝘭𝘦 𝘢𝘯𝘥 𝘣𝘶𝘪𝘭𝘥 𝘢 𝘵𝘰𝘱-𝘵𝘪𝘦𝘳 𝘝𝘓𝘔 𝘧𝘳𝘰𝘮 𝘴𝘤𝘳𝘢𝘵𝘤𝘩, 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 𝘵𝘩𝘦 𝘣𝘭𝘢𝘤𝘬 𝘣𝘰𝘹𝘦𝘴? A new paper from the Allen Institute, "𝐌𝐨𝐥𝐦𝐨 𝐚𝐧𝐝 𝐏𝐢𝐱𝐌𝐨: 𝐎𝐩𝐞𝐧 𝐖𝐞𝐢𝐠𝐡𝐭𝐬 𝐚𝐧𝐝 𝐎𝐩𝐞𝐧 𝐃𝐚𝐭𝐚 𝐟𝐨𝐫 𝐒𝐭𝐚𝐭𝐞-𝐨𝐟-𝐭𝐡𝐞-𝐀𝐫𝐭 𝐕𝐢𝐬𝐢𝐨𝐧-𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬," provides a powerful new playbook. They tackle the data problem by creating 𝘗𝘪𝘹𝘔𝘰, a massive, high-quality dataset built entirely without using external VLMs. Their innovations in data collection are the key: ✅ Instead of typed captions, they used spoken descriptions transcribed to text, yielding far more detailed training data. ✅ They built an instruction-following dataset by having humans interactively edit answers from a language-only LLM. ✅ They introduced a unique 2D pointing dataset with 2.3M annotations, enabling "chain-of-thought" counting where the model points to objects as it counts. The results: Their best open-weight, open-data model, Molmo-72B, outperforms proprietary models like Claude 3.5 Sonnet and Gemini 1.5 Pro. In both academic benchmarks and a large-scale human evaluation, it ranked second only to GPT-4o.

If you are wondering how RLHF works, and how we can teach large language models to be helpful, harmless, and honest, read along 👇 The key isn’t just in scaling up model size, it’s in aligning models with human intent. The InstructGPT paper (2022) introduced a three-step process called Reinforcement Learning from Human Feedback (RLHF). And even today, it remains the foundation of how we build instruction-following models like ChatGPT. Let me walk you through the workflow in plain terms, based on the now-famous diagram below 👇 𝟭. 𝗦𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗲𝗱 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 (𝗦𝗙𝗧) → Start by showing the model examples of great answers to real prompts, written by humans. → These examples help the model learn how to respond: clear, direct, and grounded. → Think of this as training a junior writer by giving them a stack of perfect first drafts. → Even with a small dataset (13k samples), this creates a solid instruction-following base. 𝟮. 𝗥𝗲𝘄𝗮𝗿𝗱 𝗠𝗼𝗱𝗲𝗹 (𝗥𝗠) → Next, we collect several outputs for the same prompt and ask humans to rank them from best to worst. → We then train a separate model- the reward model, to predict those rankings. → Now, we’ve turned human preferences into a numerical score the model can optimize for. → This is the real magic: turning subjective feedback into something that can guide learning. 𝟯. 𝗥𝗲𝗶𝗻𝗳𝗼𝗿𝗰𝗲𝗺𝗲𝗻𝘁 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (𝗣𝗣𝗢) → Now the model generates new answers, gets scored by the reward model, and adjusts its behavior to maximize reward. → We use Proximal Policy Optimization (PPO), an RL algorithm that gently nudges the model in the right direction without making it forget what it already knows. → A “KL penalty” keeps it from straying too far, like a seatbelt keeping it grounded. 𝗪𝗵𝘆 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀❓ ✅ A small 1.3B model trained with this pipeline outperformed GPT-3 (175B) in human evaluations. ✅ It generalized to unseen domains with little extra supervision. ✅ And it required orders of magnitude less data than pre-training. 𝗪𝗵𝗮𝘁 𝘁𝗵𝗶𝘀 𝗺𝗲𝗮𝗻𝘀 𝗳𝗼𝗿 𝗯𝘂𝗶𝗹𝗱𝗲𝗿𝘀❓ → Bigger isn’t always better. Better feedback leads to better behavior. → Pairwise comparisons are often more scalable than manual ratings. → RLHF lets us teach models values, not just vocabulary. If you're building AI systems, aligning them with human preferences isn’t just a safety concern- it’s a product strategy. --------- Share this with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI insights.

Reinforcement learning from human feedback (RLHF)—the technique for getting large language models to follow instructions—usually operates in two separate phases: collect a large set of human judgments comparing pairs of model responses, then train a reward model (a network that scores responses) and use those scores to push the language model toward higher-rated outputs. Google DeepMind researchers instead interleave the two, updating both models as each new comparison arrives, and pick which response pairs to send to humans based on which choice would be most informative. Informativeness rests on a small ensemble of networks attached to the reward model, which produces a spread of reward predictions instead of a single number; pairs where ensemble members disagree most about the winner are the ones routed to humans. They report needing roughly 10x fewer human labels than standard offline RLHF at the 20K-comparison scale, and project a 1000x reduction at one million comparisons. Read with an AI tutor on ChapterPal: https://lnkd.in/eJK9aFER PDF: https://lnkd.in/ecQFNxcp

ChatGPT owes much of its "intelligence" to us humans. Despite the widespread belief that these models are purely machine-trained, the human touch has truly guided them to their current heights of sophistication. 🔥🧠 Now, this sparks an intriguing conversation about Reinforcement Learning from Human Feedback (RLHF) - a crucial technology behind these models. In the world of LLMs, RLHF is not an alien concept, but the degree to which it shapes models like GPT is underappreciated. RLHF isn't an elite club limited to the playground of tech giants. Even smaller models and tight-budget teams can harness its powers to remarkable effects.🚀🌍 Our latest blog post delves into the synergy between RLHF and Large Language Models (LLMs) and how you can use it to your advantage, regardless of your project's scale. Here's a sneak peek into what you'll discover: 1️⃣ The blend of RL and NLP to create RLHF for enhanced language understanding. 2️⃣ The essential stages of RLHF and how it's akin to teaching a parrot to communicate. 3️⃣ Real-world applications and challenges of RLHF, including the issue of hallucinations in LLMs. 4️⃣ A practical guide on implementing RLHF, even with limited resources.

Using LLMs for market research and consumer understanding is a smart idea. Large FMCG companies have a long history of effectively using MR to build their brands. However these can be very expensive and time consuming. New age companies (Specially in Auto, Tech, Finance, even D2C and others) have shied away from this rigour precisely for the same reason. They need to turn around products and innovation much faster and fear the costs associated. Therefore they simply wing it. This is one of the reasons for high failure rate of new companies. Not knowing your customers well before building your business. Role play with LLMs can boost customer understanding and behaviour significantly, even if it is not 100%. A recent paper co-authored by PyMC Labs and Colgate-Palmolive teams demonstrates how LLMs can be used to replace sophisticated Market Research with 90% accuracy. Not just qualitative but quantitate. The paper "LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings" summarises "Consumer research costs companies billions annually yet suffers from panel biases and limited scale. Large language models (LLMs) offer an alternative by simulating synthetic consumers, but produce unrealistic response distributions when asked directly for numerical ratings. We present semantic similarity rating (SSR), a method that elicits textual responses from LLMs and maps these to Likert distributions using embedding similarity to reference statements. Testing on an extensive dataset comprising 57 personal care product surveys conducted by a leading corporation in that market (9,300 human responses), SSR achieves 90% of human test–retest reliability while maintaining realistic response distributions (KS similarity > 0.85). Additionally, these synthetic respondents provide rich qualitative feedback explaining their ratings. This framework enables scalable consumer research simulations while preserving traditional survey metrics and interpretability." I have been using some of these techniques. Saves a lot of time and some results are quite surprising (never thought of). Before taking your company, product, brand or campaign to market, use LLMs to garner better consumer understanding and test. It can save a lot of heartburn.

In an interesting new paper from Colgate-Palmolive and PyMC Labs, researchers explored whether large language models (LLMs) could accurately replicate traditional consumer concept testing. Turns out, maybe they can! Instead of assigning LLMs a Likert score (the popular 1-5 rating survey rating system), researchers developed a new system they’re calling a “Semantic-similarity rating (SSR).” For the study, the team used GPT-4o and Gemini-2.0-flash to impersonate consumers from different demographic groups like age and income. The LLMs then were asked give text-based answers 57 real consumer surveys about personal care products, which the team then analyzed with semantic analysis and then map to Likert-style ratings. When compared with the 9,300 real human answers for the same surveys, the LLMs’ answers mirrored the human answers more than 90% of the time — and even provided richer responses and more nuanced opinions about products and pricing. This paper also builds on another study I remember reading back in 2023, when researchers from Harvard and Microsoft examined whether LLMs could predict whether someone was likely to buy a product based on various scenarios and factors like brand, product, price and income level. Turns out that study — which includes comparisons of Colgate vs Crest for toothpaste — even part of the inspiration for the new paper. (I'll link to both papers in the comments.)