LLMs and RAGs are powerful, but without a strong evaluation pipeline, you're flying blind. Here's how to systematically test and improve your GenAI Apps. 🧵 (1/6)
You should operate under the assumption that all of your private data will become public in the next 2 years. Not even Google/iCloud is safe long term.
If models already stumble onto system exploits as a side effect of benchmark optimization, what happens when persistence is the explicit objective function?
What actually stops someone from using RL to reward root access, weight cloning, and anti-shutdown behavior directly?
Honestly pretty torn on open weight models after the OpenAI/Hugging Face situation.
On one hand, letting people fine tune away safety guardrails is a real risk. On the other, overly strict guardrails just tie the hands of security people when things go wrong.