February 2024

True Zero-shot MT

True Zero-shot MT

⋅

Little over a week ago, Gemini 1.5 reported close to human-level performance on MTOB, a recent challenging translation dataset. In this post, we’ll dig into this result, explore true zero-shot machine translation (MT), and consider how to teach LLMs a new language like humans. This post was first published in NLP News. Low-resource MT To set the scene, let’s first consider what it means for a language to be considered “low-resource”. As with LLMs, the performance of MT models depends on […]

Ver mais

Liked Liked

Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch

⋅

Low-rank adaptation (LoRA) is a machine learning technique that modifies a pretrained model (for example, an LLM or vision transformer) to better suit a specific, often smaller, dataset by adjusting only a small, low-rank subset of the model’s parameters. In this article, we will take a look at both LoRA and DoRA, which is a new promising alternative to LoRA.

Ver mais

Liked Liked
Thoughts on the 2024 AI Job Market

Thoughts on the 2024 AI Job Market

⋅

It’s crazy how the AI and NLP landscape has evolved over the last five years. 5 years ago, around the time I finished my PhD, if you wanted to work on cutting-edge natural language processing (NLP), your choice was relatively limited. Recently, I decided to go on the job market again, which has become much more diverse. In this post, I want to highlight some macro trends that I observed and the reasons that I joined my new […]

Ver mais

Liked Liked
I Completed The David Goggins Challenge And Asked My Garmin How I Did

I Completed The David Goggins Challenge And Asked My Garmin How I Did

⋅

David Goggins – just your typical advocate for a leisurely life of comfort and avoidance of challenge. His philosophy? Life sucks, so embrace discomfort, work hard, and push boundaries – because who needs an easy, stress-free existence anyway? David Goggins is an interesting personality. From my perspective, he’s one of those figures you either love or hate. I’m impressed by his perseverance and drive, but his life and fitness philosophy don’t appeal to me much. I guess my […]

Ver mais

Liked Liked

Thinking about High-Quality Human Data

⋅

[Special thank you to Ian Kivlichan for many useful pointers (E.g. the 100+ year old Nature paper “Vox populi”) and nice feedback. 🙏 ] High-quality data is the fuel for modern data deep learning model training. Most of the task-specific labeled data comes from human annotation, such as classification task or RLHF labeling (which can be constructed as classification format) for LLM alignment training. Lots of ML techniques in the post can help with data quality, but fundamentally […]

Ver mais

Liked Liked

↑

Digitado © 2025