sciclaimeval.github.io/faq.html at main · SciClaimEval/sciclaimeval.github.io · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
---
layout: none
title: FAQ
permalink: /faq.html
---
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>SciClaimEval FAQ</title>
    <!-- explicit favicon so the browser doesn't 404 on /favicon.ico -->
    <link rel="icon" href="/assets/favicon.png" type="image/png">
    <style>
        body {
            font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
            max-width: 1100px;
            margin: 0 auto;
            padding: 20px;
            background-color: #f5f5f5;
        }

        .header {
            text-align: center;
            margin-bottom: 30px;
            background-color: white;
            padding: 28px;
            border-radius: 8px;
            box-shadow: 0 2px 4px rgba(0,0,0,0.1);
        }

        h1 {
            color: #2c3e50;
            margin-bottom: 5px;
            font-size: 30px;
        }

        .header p {
            margin: 5px 0;
            font-size: 13px;
            color: #666;
        }

        .back-link {
            display: inline-block;
            margin-top: 10px;
            font-size: 16px;
            color: #2980b9;
            text-decoration: none;
            font-weight: 600;
        }

        .back-link:hover { text-decoration: underline; }

        .faq-section {
            background-color: white;
            padding: 16px 32px;
            margin-bottom: 20px;
            border-radius: 8px;
            box-shadow: 0 2px 4px rgba(0,0,0,0.1);
        }

        /* ── Accordion item ── */
        .faq-item {
            border-bottom: 1px solid #eee;
        }

        .faq-item:last-child { border-bottom: none; }

        .faq-question {
            display: flex;
            align-items: center;
            justify-content: space-between;
            width: 100%;
            background: none;
            border: none;
            color: #2c3e50;
            font-size: 19px;
            font-weight: 600;
            text-align: left;
            padding: 18px 4px;
            margin: 0;
            cursor: pointer;
            font-family: inherit;
            transition: color 0.2s;
        }

        .faq-question:hover { color: #2980b9; background: none; }

        /* ── Category tag ── */
        .faq-question .q-text {
            display: flex;
            align-items: center;
            gap: 10px;
        }

        .faq-tag {
            display: inline-block;
            padding: 2px 10px;
            border-radius: 10px;
            font-size: 11px;
            font-weight: 700;
            text-transform: uppercase;
            letter-spacing: 0.5px;
            flex-shrink: 0;
        }

        .faq-tag.task {
            background-color: #e8e3fb;
            color: #5b3fae;
        }

        .faq-tag.data {
            background-color: #d4edda;
            color: #155724;
        }

        .faq-tag.submission {
            background-color: #fce8d4;
            color: #a4520a;
        }

        .faq-tag.paper {
            background-color: #d6ecf7;
            color: #155b80;
        }

        .faq-question .toggle-icon {
            flex-shrink: 0;
            margin-left: 16px;
            font-size: 20px;
            color: #3498db;
            transition: transform 0.2s;
            line-height: 1;
        }

        .faq-item.open .toggle-icon { transform: rotate(45deg); }

        .faq-answer {
            display: none;
            padding: 0 4px 18px;
            font-size: 17px;
            line-height: 1.7;
            color: #555;
        }

        .faq-item.open .faq-answer { display: block; }

        .faq-answer a {
            color: #2980b9;
            text-decoration: none;
            font-weight: 600;
        }

        .faq-answer a:hover { text-decoration: underline; }

        .faq-answer code {
            background-color: #ecf0f1;
            padding: 2px 6px;
            border-radius: 4px;
            font-size: 13px;
            color: #c0392b;
        }

        .faq-answer ul {
            margin: 8px 0;
            padding-left: 22px;
        }

        .faq-answer li { margin-bottom: 6px; }

        .contact-note {
            text-align: center;
            color: #666;
            font-size: 16px;
            margin-top: 10px;
        }

        .contact-note a {
            color: #2980b9;
            font-weight: 600;
            text-decoration: none;
        }

        .contact-note a:hover { text-decoration: underline; }
    </style>
</head>
<body>

    <div class="header">
        <h1>SciClaimEval — Frequently Asked Questions</h1>
        <a class="back-link" href="/">← Back to the task home page</a>
    </div>

    <div class="faq-section" id="faqSection">

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag data">Data</span>What should I do if I find errors in the dataset samples?</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                As mentioned in our paper, all samples in the dataset were created by humans. However, human
                annotators can occasionally make mistakes due to workload, oversight, or other factors. To
                date, we are aware of only a few (two at the moment) problematic samples in the dataset.
                To ensure a fair comparison among all participating teams, we have decided to keep the dataset
                unchanged at this time. If you encounter any errors, please don't hesitate to inform us.
            </div>
        </div>

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag data">Data</span>Some claims contain ambiguous words such as "it" or "they" and I'm not sure what they refer to. Are these considered problematic samples?</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                If you encounter such ambiguous claims, please refer to the <code>use_context</code> field in
                our dataset. This field can have three values: <code>no</code> (no additional context
                required), <code>yes</code> (requires the <code>context</code> field for disambiguation), or
                <code>other sources</code> (requires the full paper for disambiguation). Based on our
                experience creating and reviewing the dataset, we have found that most ambiguous claims can be
                clarified using the context information (either the provided short paragraph or the full
                paper).
            </div>
        </div>

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag submission">Submission</span>Is there a limitation on how many submissions we can make?</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                No, there is no limitation on how many submissions you can make.
            </div>
        </div>

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag submission">Submission</span>Is there a live leaderboard available?</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                No, we do not provide a leaderboard before the run submission deadline (July 19th). However, on our website
                (<a href="https://sciclaimeval.github.io/#evaluations--baselines">Evaluations &amp; Baselines</a>), you can compare your
                results against the baseline scores. When you upload a new submission, our evaluation bot (EvalBot) will send out an e-mail
                to your full team containing the results (or an error message if something went wrong). You are encouraged to upload more
                runs to further improve your scores but you cannot see other participant's scores before the deadline.
                We publish a full leaderboard after the run submission deadline.
            </div>
        </div>

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag submission">Submission</span>Will evaluation results be provided on our submissions during the active phase (i.e., before the final submission deadline on July 19th)?</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                Yes, after you submit your file, our evaluation bot (EvalBot) sends your team an e-mail containing your results.
                If you haven't received an e-mail after 30min, please send an e-mail to the organizers (<a href="mailto:sciclaimeval@gmail.com">sciclaimeval (at) gmail.com</a>) as there was likely an error during the evaluation.
            </div>
        </div>

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag submission">Submission</span>I have uploaded a submission but did not receive an e-mail notification.</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                We deployed an automated evaluation bot (EvalBot) that automatically verifies and evaluates your result, but it may take up to 30min for you to receive the e-mail. The e-mail will have the subject 'SciClaimEval Submission Update' and
                come from sciclaimeval@gmail.com. If you haven't received an e-mail after 30min, please make sure you entered the correct Group ID assigned to your team. If you do not have a group ID yet, you need to register for the task first to obtain one.

                If you are sure you entered the correct Group ID, please send an e-mail to the organizers (<a href="mailto:sciclaimeval@gmail.com">sciclaimeval (at) gmail.com</a>) as there was likely an error during the evaluation.
            </div>
        </div>

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag submission">Submission</span>Which submission do you select, if we upload multiple submissions?</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                The best run on the primary metric (task 1: pair accuracy, task 2: accuracy) decides your team rank versus other team's best submission regardless of the submission date/order.
                However, we will still show and analyse all other submissions if they differ sufficiently (for example, different architecture/model differs sufficiently; hyperparameter tuning does not).
                Hence, we encourage all teams to explain each submission during the upload in the Google form and explore different ideas. The final leaderboard will contain all submissions eventually.
            </div>
        </div>

        <div class="faq-item">
            <button class="faq-question" onclick="toggleFaq(this)">
                <span class="q-text"><span class="faq-tag paper">Paper Submission</span>Is there a specific template for the paper submission, and what are the formal requirements (e.g., page limit)?</span>
                <span class="toggle-icon">+</span>
            </button>
            <div class="faq-answer">
                Information regarding the paper template and formal requirements (including the page limit and
                formatting guidelines) is available on the NTCIR-19 submission instructions page. Please refer
                to the
                <a href="https://research.nii.ac.jp/ntcir/ntcir-19/papers.html" target="_blank" rel="noopener">NTCIR-19 Paper Submission Instructions</a>
                for details.
            </div>
        </div>

    </div>

    <p class="contact-note">
        Still have questions? Email us at
        <a href="mailto:sciclaimeval@gmail.com">sciclaimeval (at) gmail.com</a>.
    </p>

    <script>
        function toggleFaq(btn) {
            const item = btn.parentElement;
            item.classList.toggle('open');
            btn.querySelector('.toggle-icon').textContent =
                item.classList.contains('open') ? '−' : '+';
        }
    </script>
</body>
</html>