Benchmark Discussion

[mldiego]

The initial plan is to reuse benchmarks from 2024, but we welcome everyone to propose new benchmarks or modifications to existing ones.

We also welcome any suggestions to improve the competition. Please share in this thread so all participants are aware of ongoing discussions.

[ttj]

Prior benchmarks are here: https://github.com/verivital/ARCH-COMP2024

[toladnertum]

Dear all,

We again use our submission system to evaluate your tool.
In case there are any issues and questions, please leave them here: #2

@ttj: In case other categories want to use the submission system as well, please let me know. I know that the liner and nonlinear category used it as well last year, not sure about the others.

Best,
Tobias

[xiangruzh]

Hi all,

We have a question regarding the benchmark TORA. According to the benchmark description (link), the outputs from both controllers are post-processed as $u=11\cdot f(x)$, consistent with previous years' reports. However, the specification file for the system using sigmoid controller appears to use a different post-processing expression (link).

This inconsistency also existed in last year's competition. Just want to confirm which one we should follow for this year.

Thanks!
Xiangru Zhong

[mldiego]

@xiangruzh We are using 0 and 11 for both. I have corrected that in the txt file of the controller. We are using the ones consistent with the report and specification description.

[Akash-Harapanahalli]

Hi all,

My advisor Dr. Samuel Coogan and I are excited to contribute this year with our toolbox immrax.

For now, our framework requires the neural network controller to be in continuous feedback with the plant. So for the benchmarks we were able to complete, we have interpreted them without the sample-and-hold present in the benchmark descriptions. Most of the benchmarks we completed verify the same specifications, but the SinglePendulum benchmark violates the specification when the neural network is in continuous feedback.

We will make these points clear in this year's report.

Thanks,
Akash Harapanahalli

[toladnertum]

Dear @Akash-Harapanahalli,

Glad to hear you are contributing to the competition this year!
We always have tools that consider slightly different use-cases or settings, and we are fine with it as long as it is clearly stated in the report.
For example, we had GoTube participate last year, which provides probabilistic guarantees rather than formal ones as provided by the other tools.

Best,
Tobias