Initially I aimed to test with at least 10 formulas for each model for SAT/UNSAT, but it turned out to be more expensive than I expected, so I tested ~5 formulas for each case/model. First, I used the openrouter API to automate the process, but I experienced response stops in the middle due to long reasoning process, so I reverted to using the chat interface (I don't if this was a problem from the model provider or if it's an openrouter issue). For this reason I don't have standard outputs for each testing, but I linked to the output for each case I mentioned in results.
长和系三家公司发布联合公告，宣布出售英国电网业务，套现逾1100亿港元。下载安装 谷歌浏览器 开启极速安全的 上网之旅。是该领域的重要参考
Cancer screening。旺商聊官方下载是该领域的重要参考
Following the president's Friday afternoon announcement, OpenAI CEO Sam Altman appeared on CNBC and voiced support for Anthropic. "For all the differences I have with Anthropic, I mostly trust them as a company and I think they really do care about safety, and I’ve been happy that they’ve been supporting our war fighters," Altman said, according to a clip of the appearance posted to X.