Skip to content

Commit dca37fe

Browse files
markovialmarkovial
committed
update: ch5 evals
1 parent 1911686 commit dca37fe

File tree

2 files changed

+72
-23
lines changed

2 files changed

+72
-23
lines changed

‎docs/chapters/05/README.md‎

Lines changed: 72 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@
6464
</span>
6565
<div class="meta-content">
6666
<div class="meta-label">Reading Time</div>
67-
<div class="meta-value">154 min (core)</div>
67+
<div class="meta-value">132 min (core)</div>
6868
</div>
6969
</div>
7070
@@ -76,34 +76,83 @@
7676
<div class="meta-content">
7777
<div class="meta-label">Also available on</div>
7878
<div class="meta-value meta-links">
79-
<a href="https://docs.google.com/document/d/1KI95w27Ce7yWoynE11PJ94IXK0gT0NwP8091s06P7wM/edit?usp=sharing" class="meta-link">Google Docs</a> · <a href="https://www.lesswrong.com/s/GTAGBi3fyw88yArQi" class="meta-link">Alignment Forum</a>
79+
<a href="https://docs.google.com/document/d/1KI95w27Ce7yWoynE11PJ94IXK0gT0NwP8091s06P7wM/edit?usp=sharing" class="meta-link">Google Docs</a>
8080
</div>
8181
</div>
8282
</div>
8383
</div>
8484
</div>
8585

8686
<div class="action-buttons">
87-
<div class="action-button disabled" data-tippy-content="Video coming soon">
88-
<i class="fas fa-video"></i>
89-
<span>Watch</span>
90-
</div>
91-
<div class="action-button disabled" data-tippy-content="Audio coming soon">
92-
<i class="fas fa-headphones"></i>
93-
<span>Listen</span>
94-
</div>
95-
<div class="action-button disabled" data-tippy-content="PDF coming soon">
96-
<i class="fas fa-file-pdf"></i>
97-
<span>Download</span>
98-
</div>
99-
<a href="https://forms.gle/ZsA4hEWUx1ZrtQLL9" class="action-button">
100-
<i class="fas fa-comment"></i>
101-
<span>Feedback</span>
102-
</a>
103-
<a href="https://docs.google.com/document/d/1T-UU0FBeElX6cvbWYKpVAl3U4ivrQLHA3IdIWqWKuBA/edit?tab=t.0#heading=h.fo57hwsn3del" class="action-button">
104-
<i class="fas fa-users"></i>
105-
<span>Facilitate</span>
106-
</a>
87+
<div class="action-button disabled" data-tippy-content="Video coming soon">
88+
<i class="fas fa-video"></i>
89+
<span>Watch</span>
90+
</div>
91+
<a href="#" class="action-button">
92+
<i class="fas fa-headphones"></i>
93+
<span>Listen</span>
94+
</a>
95+
<div class="action-button disabled" data-tippy-content="PDF coming soon">
96+
<i class="fas fa-file-pdf"></i>
97+
<span>Download</span>
98+
</div>
99+
<a href="https://forms.gle/ZsA4hEWUx1ZrtQLL9" class="action-button">
100+
<i class="fas fa-comment"></i>
101+
<span>Feedback</span>
102+
</a>
103+
<a href="https://docs.google.com/document/d/1T-UU0FBeElX6cvbWYKpVAl3U4ivrQLHA3IdIWqWKuBA/edit?tab=t.0#heading=h.fo57hwsn3del" class="action-button">
104+
<i class="fas fa-users"></i>
105+
<span>Facilitate</span>
106+
</a>
107+
</div>
108+
109+
<div class="atlas-audio-player">
110+
<!-- Header section -->
111+
<div class="audio-player__header">
112+
<div class="header-left">
113+
<i class="fas fa-headphones-alt"></i>
114+
<span class="audio-player__title">Audio Version</span>
115+
<span class="audio-warning-text"><i class="fas fa-robot"></i> AI-generated</span>
116+
</div>
117+
<div class="header-right">
118+
<button class="audio-close-button" aria-label="Close audio player">
119+
<i class="fas fa-times"></i>
120+
</button>
121+
</div>
122+
</div>
123+
124+
<!-- MEGA embed wrapper -->
125+
<div class="mega-audio-wrapper">
126+
<iframe
127+
width="100%"
128+
height="160"
129+
frameborder="0"
130+
src="https://mega.nz/embed/ieBRQZQZ#GzZAljhhfVUmUHSG0cWEBaW2AkuCm4LmpfYFGtWKBbM!1v1c"
131+
allowfullscreen
132+
loading="lazy"
133+
allow="autoplay">
134+
</iframe>
135+
</div>
136+
137+
<!-- Error toggle button at the bottom -->
138+
<div class="errors-toggle-container">
139+
<button class="errors-toggle-button" aria-label="View known errors">
140+
<i class="fas fa-exclamation-circle"></i> View known errors in AI-generated audio
141+
</button>
142+
</div>
143+
144+
<!-- Hidden errors panel at the bottom -->
145+
<div class="audio-errors-panel">
146+
<h4>Known Errors in AI-Generated Audio</h4>
147+
<ul>
148+
<li><strong>3:00</strong> - Error - Feature visualizations (are not that helpful currently; behavioral approaches remain the best way to evaluate deception)</li>
149+
<li><strong>4:00</strong> - Imprecise - Multiple labs evaluate CBRN capabilities, not just OpenAI</li>
150+
<li><strong>12:30</strong> - Debated - TruthfulQA is not really a propensity evaluation</li>
151+
<li><strong>17:30</strong> - Error - Sandbagging emerges from the AI itself, it is not implemented by developers</li>
152+
<li><strong>18:40</strong> - Inaccurate - One of the most important limitations is not just that we lack evaluations in cybersecurity; rather, it's difficult to elicit the capabilities of a system, and we cannot be certain whether it will be impossible to create a system that performs</li>
153+
</ul>
154+
<p class="report-errors">Found more errors? Please report to <a href="mailto:[email protected]">[email protected]</a></p>
155+
</div>
107156
</div>
108157

109158
# Introduction
@@ -140,4 +189,4 @@ Benchmarks revolutionized AI development by creating standardized metrics that a
140189
<figcaption markdown="1"><b>Figure 5.1:</b> Overview of chapter content.</figcaption>
141190
</figure>
142191

143-
This introduction gave you the general overview of many of the concepts that we will be talking about throughout this chapter. The sections will largely proceed in the order that we introduced the ideas above. We begin by exploring how benchmarks have shaped AI development.
192+
This introduction gave you the general overview of many of the concepts that we will be talking about throughout this chapter. The sections will largely proceed in the order that we introduced the ideas above. We begin by exploring how benchmarks have shaped AI development.

‎latex/AI Safety Atlas - Evaluations.pdf‎

7.29 MB
Binary file not shown.

0 commit comments

Comments
 (0)