• AIPressRoom
  • Posts
  • SmartGPT: Main Benchmark Damaged – 89.0% on MMLU + Examination’s Many Errors

SmartGPT: Main Benchmark Damaged – 89.0% on MMLU + Examination’s Many Errors

September 15, 2023

Has GPT4, utilizing a SmartGPT system, damaged a significant benchmark, the MMLU, in additional methods than one? 89.0% is an unofficial report, however will we urgently want a brand new, authoritative benchmark, particularly within the gentle of in the present day’s insider data of 5x compute for Gemini than for GPT 5?

Study all in regards to the energy of exemplars, self-consistency and how one can tangibly profit in actual world examples. You may be taught extra about the whole lot from leading edge benchmarking to AGI forecasting.

Unique SmartGPT Video: https://www.youtube.com/watch?v=wVzuvf9D9BU&list=PPSVMMLU: https://arxiv.org/pdf/2009.03300.pdfGemini 5x GPT 4, Semianalysis: https://www.semianalysis.com/p/google-gemini-eats-the-world-geminiWizardCoder Overfitting? https://twitter.com/Shahules786/status/1695493641610133600Let’s Do a Thought Experiment: https://arxiv.org/pdf/2306.14308.pdfLegalBench: https://arxiv.org/pdf/2308.11462.pdfSciBench: https://arxiv.org/pdf/2307.10635.pdfAGIEval: https://arxiv.org/pdf/2304.06364.pdfMMLU Grading Points: https://huggingface.co/blog/evaluating-mmlu-leaderboardOxford College Press Query Instance: https://global.oup.com/uk/orc/chemistry/chechik/student/mcqs/ch04/Fall 2011 Epidemiology Instance: https://www.docsity.com/en/final-exam-fall-2011-4/8308030/HellaSwag: https://arxiv.org/pdf/1905.07830.pdfGPT 4 Technical Report: https://arxiv.org/pdf/2303.08774.pdfMinerva, Fixing Quantitative Reasoning: https://arxiv.org/pdf/2206.14858.pdfUnique Scratchpads Paper: https://arxiv.org/pdf/2112.00114.pdfIs ChatGPT Behaviour Altering Over Time? https://arxiv.org/pdf/2307.09009.pdfPaul Christiano: https://www.lesswrong.com/posts/fRSj2W4Fjje8rQWm9/thoughts-on-sharing-information-about-language-modelMetaculus Forecasting: https://www.metaculus.com/ai/ https://www.lesswrong.com/posts/SdkexhiynayG2sQCC/ai-forecasting-two-years-inMIT Paper: https://twitter.com/jeremyphoward/status/1669588857149612033?lang=en-GBSnowballing Hallucinations: https://arxiv.org/pdf/2305.13534.pdfSelf Consistency: https://arxiv.org/pdf/2203.11171.pdfOpenLLM Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboardNHS Query from ‘Prolonged Matching Questions’Graph of Ideas: https://arxiv.org/pdf/2308.09687.pdfDario Amodei Interview – Dwarkesh Patel: https://www.youtube.com/watch?v=Nlkk3glap_U

Joshua Stapleton is a Machine Studying Engineer who has labored within the healthcare and defence sectors. He not too long ago pivoted into AI capabilities and security, with a focus on LLMs. He now works as a analysis engineer, consults on the purposes of AI throughout numerous industries, and is pursuing his Masters in Machine Studying and Information Science at Imperial Faculty London.Be at liberty to succeed in out to Josh through his electronic mail, [email protected], or take a look at his new Patreon: https://patreon.com/JoshuaStapleton.