Falcon 180B: Complete Benchmark Performance Guide

other falcon-180b 2023-09-06 TII Falcon LLM License

In-depth analysis of Falcon 180B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionfalcon-180b
Release Date2023-09-06
Context Window2048 tokens
Input Modalitiestext
Output Modalitiestext
LicenseTII Falcon LLM License
Documentationhttps://huggingface.co/models

Benchmark Performance

BenchmarkScoreUnitEvaluated AtNotesSource
MMLU76.7%2023-09-065-shotview
HUMANEVAL68.9pass@12023-09-06—view
GSM8K77.5%2023-09-060-shot CoTview
MATH44.8%2023-09-060-shot CoTview
BBH76.4%2023-09-063-shot CoTview
GPQA43.2%2023-09-060-shotview
IFEVAL70.5%2023-09-06prompt_strictview
ARC91.6%2023-09-06challengeview
MUSR59.1%2023-09-060-shotview
WINOGRANDE80%2023-09-060-shotview

Pricing

TierPriceCurrency
Input$1 / MtokUSD
Output$1 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2023-09-06

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Falcon 180B

Modeloverzicht

TII Falcon 180B 大型开源模型, 2K 上下文, 1800 亿参数, 在开源模型中曾排名第一, 排在 GPT-4 之后。

Kernspecificaties

LeverancierVersieReleasedatumContextvensterInvoermodaliteitenUitvoermodaliteitenLicentie
Otherfalcon-180b2023-09-062KtexttextTII Falcon LLM License

Benchmarkprestaties

BenchmarkScoreEenheidNotities
MMLU (Massive Multitask Language Understanding)76.7%5-shot
HumanEval68.9pass@1—
GSM8K (Grade School Math 8K)77.5%0-shot CoT
MATH44.8%0-shot CoT
BBH (BIG-Bench Hard)76.4%3-shot CoT
GPQA43.2%0-shot
IFEval70.5%prompt_strict
ARC91.6%challenge
MUSR59.1%0-shot
WinoGrande80.0%0-shot

Prijzen

InvoerUitvoerCache-lezenCache-schrijven
————

per miljoen tokens

Sterktes

  • 可靠的通用模型。

Zwaktes

  • 闭源专有模型，不支持自托管。
  • 上下文窗口 2K 偏小。

Gebruiksscenario’s

  • 通用对话与问答

Referenties