Lexicostatistics

Language Models

Release Date	Vendor	Model	Prompt Herdan	Prompt Zipf	Reasoning Herdan	Reasoning Zipf
2024-01-25	OpenAI	gpt-3.5-turbo-0125	0.454	1.006	0.586	0.992
2024-07-18	OpenAI	gpt-4o-mini	0.568	1.108	0.712	0.841
2024-10-22	Anthropic	claude-3-5-haiku-20241022	0.700	0.888	0.677	0.867
2024-12-05	OpenAI	o1	0.626	0.956	0.685	0.870
2024-12-06	Meta	llama3.3:latest	0.325	0.628	0.537	0.831
2024-12-17	TII (UAE)	falcon3:10b	0.185	0.333	0.173	0.445
2025-01-29	OpenAI	gpt-4o-2025-01-29	0.490	1.013	0.565	0.778
2025-01-30	Google DeepMind	gemini-2.0-flash	0.384	1.331	0.647	0.963
2025-02-05	Google DeepMind	gemini-2.0-pro-exp	0.464	1.137	0.670	0.875
2025-02-19	Anthropic	claude-3-7-sonnet-20250219	0.613	1.005	0.691	0.842
2025-02-27	OpenAI	gpt-4.5-preview	0.500	1.182	0.675	0.834
2025-03-12	Google DeepMind	gemma3:27b	0.534	1.134	0.631	0.918
2025-03-25	Google DeepMind	gemini-2.5-pro-exp-03-25	0.499	1.074	0.700	0.818
2025-03-27	OpenAI	gpt-4o-2025-03-26	0.566	1.115	0.693	0.827
2025-04-14	OpenAI	gpt-4.1	0.566	0.998	0.710	0.807
2025-04-15	Deep Cogito	cogito:70b	0.542	0.962	0.639	0.839
2025-04-16	OpenAI	o3	0.647	0.895	0.769	0.711
2025-05-14	Anthropic	claude-opus-4-20250514	0.531	1.209	0.635	0.959
2025-05-14	Anthropic	claude-sonnet-4-20250514	0.529	1.130	0.637	0.967

Herdan Coefficient (Prompts)

Slope 0.0001, intercept 0.4591, p=0.56186

Zipf Coefficient (Prompts)

Slope 0.0004, intercept 0.8742, p=0.43684

Herdan Coefficient (Reasoning)

Slope 0.0002, intercept 0.5630, p=0.45814

Zipf Coefficient (Reasoning)

Slope -0.0001, intercept 0.8834, p=0.64529

Best Ensembles

Date	Ensemble	Prompt Herdan	Prompt Zipf	Reasoning Herdan	Reasoning Zipf
2024-07-18	openailong,openai35,random	0.610	1.021	0.724	0.839
2024-07-18	openailong,openai35,random	0.610	1.021	0.724	0.839
2024-07-18	openailong,openai35,random	0.610	1.021	0.724	0.839
2024-07-18	openailong,openai35,random	0.610	1.021	0.724	0.839
2024-07-18	openailong,openai35,random	0.610	1.021	0.724	0.839
2024-07-18	openailong,openai35,random	0.610	1.021	0.724	0.839
2024-10-22	anthropic,openai35,random	0.656	0.967	0.666	0.894
2024-10-22	anthropic10,openailong,openai35	0.630	1.000	0.737	0.828
2024-10-22	anthropic,openai35,random	0.656	0.967	0.666	0.894
2024-10-22	anthropic,anthropic10,openai35	0.686	0.885	0.687	0.862
2024-10-22	anthropic,anthropic10,openailong	0.639	0.984	0.727	0.814
2024-10-22	anthropic,anthropic10,random	0.688	0.896	0.684	0.864
2024-10-22	anthropic,anthropic10,openai35	0.686	0.885	0.687	0.862
2024-11-20	anthropic,openai,openailong	0.627	0.996	0.728	0.827
2024-11-20	openai10,openai,random	0.645	0.977	0.727	0.789
2024-11-20	openai10,openai,openai10	0.645	0.977	0.727	0.789
2024-11-20	anthropic,openai,random	0.679	0.923	0.689	0.841
2024-11-20	anthropic,openai,openai35	0.684	0.912	0.709	0.836
2024-11-20	openai,openai,openai10	0.645	0.977	0.727	0.789
2024-11-20	anthropic,openai,openai10	0.694	0.918	0.733	0.795
2024-11-20	anthropic,anthropic10,openai10	0.691	0.871	0.728	0.809
2024-11-20	anthropic,openai10,openailong	0.632	0.989	0.731	0.822
2024-11-20	anthropic10,openai,openai35	0.718	0.804	0.730	0.792
2024-11-20	openai,openai,openai35	0.650	0.946	0.704	0.818
2024-12-05	anthropic,anthropic10,openai10o1	0.674	0.896	0.724	0.814
2024-12-05	anthropic10,openai10o1,openaio1	0.696	0.895	0.734	0.782
2024-12-05	openai10o1,openailong,openaio1	0.640	0.985	0.723	0.828
2024-12-05	openai,openai10o1,openaio1	0.667	0.934	0.745	0.778
2024-12-05	anthropic,openai10o1,openaio1	0.673	0.928	0.710	0.812
2024-12-05	anthropic,openai10o1,openailong	0.643	0.981	0.726	0.814
2024-12-05	anthropic10,openai,openai10o1	0.718	0.864	0.738	0.769
2024-12-05	anthropic,openaio1,random	0.645	0.937	0.691	0.853
2024-12-05	anthropic,openai10o1,openai	0.672	0.925	0.731	0.793
2024-12-05	openai10,openai10o1,openai	0.650	0.962	0.751	0.771
2025-01-29	gpt-4o-legacy10,openailong,openai35	0.604	1.016	0.713	0.852
2025-01-30	gemini,openai,openai10o1	0.642	0.967	0.723	0.824
2025-02-05	anthropic10,gemini,geminipro10	0.703	0.843	0.716	0.818
2025-02-05	geminipro10,openai10o1,openai35	0.633	0.993	0.725	0.800
2025-02-19	anthropic37,anthropic3710,openai35	0.624	1.012	0.696	0.841
2025-02-19	anthropic,anthropic37,openai10o1	0.665	0.925	0.728	0.795
2025-02-19	anthropic37,gemini,openai10o1	0.620	0.978	0.704	0.809
2025-02-19	anthropic3710,openai10o1,openaio1	0.644	0.982	0.714	0.795
2025-02-19	anthropic3710,openaio1,openai35	0.627	1.007	0.720	0.819
2025-02-27	geminipro,openai10o1,openai4510	0.619	1.008	0.718	0.807
2025-02-27	anthropic,openai4510,openaio1	0.649	0.940	0.706	0.822
2025-03-25	anthropic,gemini25,gemini2510	0.652	0.974	0.713	0.818
2025-04-14	gemini25,openai10,openai4110	0.614	0.954	0.762	0.741
2025-04-14	gemini25,openai10o1,openai4110	0.629	0.941	0.750	0.760
2025-04-14	gemini,gemini25,openai4110	0.596	0.967	0.739	0.778
2025-04-16	anthropic37,gemini,openaio310	0.619	0.963	0.744	0.743
2025-04-16	geminipro,openai10o1,openaio3	0.662	0.937	0.749	0.751
2025-04-16	gemini,openaio3,openaio310	0.649	0.890	0.766	0.716
2025-04-16	openai10,openaio3,openaio310	0.692	0.841	0.789	0.684
2025-04-16	anthropic37,openaio3,openaio310	0.650	0.939	0.774	0.699
2025-04-16	anthropic3710,openaio3,openaio310	0.655	0.908	0.775	0.707
2025-04-16	gemini10,openaio3,openaio310	0.663	0.903	0.770	0.716
2025-04-16	anthropic3710,gemini,openaio3	0.650	0.994	0.744	0.763
2025-04-16	anthropic3710,openaio3,openai35	0.654	0.970	0.741	0.755
2025-05-14	anthropic,opus4010,opus4010	0.628	1.032	0.678	0.921
2025-05-14	anthropic3710,opus40,sonnet4010	0.617	1.069	0.675	0.916
2025-05-14	sonnet40,sonnet4010,openai35	0.590	1.044	0.661	0.941
2025-05-14	openai10o1,opus4010,sonnet4010	0.612	1.034	0.670	0.920
2025-05-14	geminipro10,openaio1,opus4010	0.626	1.051	0.676	0.899
2025-05-14	openaio3,opus4010,openai35	0.620	1.045	0.706	0.843
2025-05-14	openaio1,opus4010,opus4010	0.618	1.052	0.659	0.918
2025-05-14	gemini,openaio3,opus40	0.605	1.089	0.718	0.828
2025-05-14	opus40,sonnet4010,random	0.557	1.150	0.649	0.942
2025-05-14	anthropic37,gemini25,opus40	0.619	1.034	0.703	0.845

Herdan Coefficient (Ensembles - Prompts)

Slope -0.0001, intercept 0.6568, p=0.10183

Zipf Coefficient (Ensembles - Prompts)

Slope 0.0001, intercept 0.9449, p=0.14888

Herdan Coefficient (Ensembles - Reasoning)

Slope -0.0000, intercept 0.7190, p=0.97455

Zipf Coefficient (Ensembles - Reasoning)

Slope -0.0001, intercept 0.8282, p=0.41614