v1.56.5

What's Changed

• Refactor: move all bedrock invoke providers to BaseConfig by @krrishdholakia in #7463
• (fix) litellm.amoderation - support using model=openai/omni-moderation-latest, model=omni-moderation-latest, model=None by @ishaan-jaff in #7475
• [Bug Fix]: rerank restfulapi response parse still too strict by @ishaan-jaff in #7476
• Litellm dev 12 30 2024 p1 by @krrishdholakia in #7480
• HumanLoop integration for Prompt Management by @krrishdholakia in #7479

Full Changelog: v1.56.4...v1.56.5

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.56.5

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	230.0	268.0630784626629	6.174316845767241	0.0	1848	0	212.08500100010497	3189.481879000027
Aggregated	Passed ✅	230.0	268.0630784626629	6.174316845767241	0.0	1848	0	212.08500100010497	3189.481879000027

v1.56.4

What's Changed

• Update model_prices_and_context_window.json by @superpoussin22 in #7452
• (Refactor) 🧹 - remove deprecated litellm server by @ishaan-jaff in #7456
• 📖 Docs - Using LiteLLM with 1M rows in spend logs by @ishaan-jaff in #7461
• (Admin UI - 1) - added the model used either directly before or after the "Assistant" so that it's clear which model provided the given assistant output by @ishaan-jaff in #7459
• (Admin UI - 2) UI chat should render the output in markdown by @ishaan-jaff in #7460
• (Security fix) - Upgrade to fastapi==0.115.5 by @ishaan-jaff in #7447
• fix OR deepseek by @paul-gauthier in #7425
• (Bug Fix) Add health check support for realtime models by @ishaan-jaff in #7453
• (Refactor) - Re use litellm.completion/litellm.embedding etc for health checks by @ishaan-jaff in #7455
• Litellm dev 12 28 2024 p3 by @krrishdholakia in #7464
• Fireworks AI - document inlining support + model access groups for wildcard models by @krrishdholakia in #7458

Full Changelog: v1.56.3...v1.56.4

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.56.4

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	240.0	268.74238744669225	6.116896356155644	0.0	1829	0	214.29422199992132	1969.7571099999323
Aggregated	Passed ✅	240.0	268.74238744669225	6.116896356155644	0.0	1829	0	214.29422199992132	1969.7571099999323

v1.56.3

What's Changed

• Update Documentation - Gemini Embedding by @igorlima in #7436
• (Bug fix) missing model_group field in logs for aspeech call types by @ishaan-jaff in #7392
• (Feat) - new endpoint GET /v1/fine_tuning/jobs/{fine_tuning_job_id:path} by @ishaan-jaff in #7427
• Update model_prices_and_context_window.json by @superpoussin22 in #7345
• LiteLLM Minor Fixes & Improvements (12/27/2024) - p1 by @krrishdholakia in #7448
• Litellm dev 12 27 2024 p2 1 by @krrishdholakia in #7449

New Contributors

• @igorlima made their first contribution in #7436

Full Changelog: v1.56.2...v1.56.3

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.56.3

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	276.9724297749999	6.148940938190872	0.003341815727277648	1840	1	112.37049800001842	1700.1428350000083
Aggregated	Passed ✅	250.0	276.9724297749999	6.148940938190872	0.003341815727277648	1840	1	112.37049800001842	1700.1428350000083

v1.56.2

What's Changed

• Litellm dev 12 24 2024 p2 by @krrishdholakia in #7400
• (feat) Support Dynamic Params for guardrails by @ishaan-jaff in #7415
• docs: cleanup docker compose comments by @marcoscannabrava in #7414
• (Security fix) UI - update next version by @ishaan-jaff in #7418
• (security fix) - fix docs snyk vulnerability by @ishaan-jaff in #7419
• LiteLLM Minor Fixes & Improvements (12/25/2024) - p1 by @krrishdholakia in #7411
• LiteLLM Minor Fixes & Improvements (12/25/2024) - p2 by @krrishdholakia in #7420
• Ensure 'disable_end_user_cost_tracking_prometheus_only' works for new prometheus metrics by @krrishdholakia in #7421
• (security fix) - bump fast api, fastapi-sso, python-multipart - fix snyk vulnerabilities by @ishaan-jaff in #7417
• docs - batches cost tracking by @ishaan-jaff in #7422
• Add /openai pass through route on litellm proxy by @ishaan-jaff in #7412
• (Feat) Add logging for POST v1/fine_tuning/jobs by @ishaan-jaff in #7426
• (docs) - show all supported Azure OpenAI endpoints in overview by @ishaan-jaff in #7428
• (docs) - custom guardrail show how to use dynamic guardrail params by @ishaan-jaff in #7430
• Support budget/rate limit tiers for keys by @krrishdholakia in #7429
• (fix) initializing OTEL Logging on LiteLLM Proxy - ensure OTEL logger is initialized only once by @ishaan-jaff in #7435
• Litellm dev 12 26 2024 p3 by @krrishdholakia in #7434
• fix(key_management_endpoints.py): enforce user_id / team_id checks on key generate by @krrishdholakia in #7437
• LiteLLM Minor Fixes & Improvements (12/26/2024) - p4 by @krrishdholakia in #7439
• Refresh VoyageAI models, prices and context by @fzowl in #7443
• Revert "Refresh VoyageAI models, prices and context" by @krrishdholakia in #7446
• (feat) /guardrails/list show guardrail info params by @ishaan-jaff in #7442
• add openrouter o1 by @paul-gauthier in #7424
• ✨ (Feat) Log Guardrails run, guardrail response on logging integrations by @ishaan-jaff in #7445

New Contributors

• @marcoscannabrava made their first contribution in #7414
• @fzowl made their first contribution in #7443

Full Changelog: v1.55.12...v1.56.2

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.56.2

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	275.3240164096845	6.143891773397197	0.0	1838	0	224.26387399997338	1437.5524760000076
Aggregated	Passed ✅	250.0	275.3240164096845	6.143891773397197	0.0	1838	0	224.26387399997338	1437.5524760000076

v1.55.12

What's Changed

• Add 'end_user', 'user' and 'requested_model' on more prometheus metrics by @krrishdholakia in #7399
• (feat) /batches Add support for using /batches endpoints in OAI format by @ishaan-jaff in #7402
• (feat) /batches - track user_api_key_alias, user_api_key_team_alias etc for /batch requests by @ishaan-jaff in #7401
• Litellm dev 12 24 2024 p3 by @krrishdholakia in #7403
• (Feat) add `"/v1/batches/{batch_id:path}/cancel" endpoint by @ishaan-jaff in #7406
• Litellm dev 12 24 2024 p4 by @krrishdholakia in #7407

Full Changelog: v1.55.11...v1.55.12

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.12

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	220.0	241.51418849604215	6.334659319234715	0.0	1895	0	191.11329300005764	3854.987871999924
Aggregated	Passed ✅	220.0	241.51418849604215	6.334659319234715	0.0	1895	0	191.11329300005764	3854.987871999924

v1.55.11

What's Changed

• LiteLLM Minor Fixes & Improvements (12/23/2024) - p3 by @krrishdholakia in #7394

Full Changelog: v1.55.10...v1.55.11

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.11

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	250.0	290.3865391657403	6.034920682874279	0.0	1804	0	229.06071099987457	2909.605226000167
Aggregated	Passed ✅	250.0	290.3865391657403	6.034920682874279	0.0	1804	0	229.06071099987457	2909.605226000167

v1.55.10

What's Changed

• (Admin UI) - Test Key Tab - Allow typing in model name + Add wrapping for text response by @ishaan-jaff in #7347
• (Admin UI) - Test Key Tab - Allow using UI Session instead of manually creating a virtual key by @ishaan-jaff in #7348
• (refactor) - fix from enterprise.utils import ui_get_spend_by_tags by @ishaan-jaff in #7352
• (chore) - enforce model budgets on virtual keys as enterprise feature by @ishaan-jaff in #7353
• (Admin UI) correctly render provider name in /models with wildcard routing by @ishaan-jaff in #7349
• (Admin UI) - maintain history on chat UI by @ishaan-jaff in #7351
• Litellm enforce enterprise features by @krrishdholakia in #7357
• Document team admins + Enforce assigning team admins as an enterprise feature by @krrishdholakia in #7359
• Litellm docs update by @krrishdholakia in #7365
• Complete 'requests' library removal by @krrishdholakia in #7350
• (chore) remove unused code files by @ishaan-jaff in #7363
• (security fix) - update base image for all docker images to python:3.13.1-slim by @ishaan-jaff in #7388
• LiteLLM Minor Fixes & Improvements (12/23/2024) - p1 by @krrishdholakia in #7383
• LiteLLM Minor Fixes & Improvements (12/23/2024) - P2 by @krrishdholakia in #7386
• [Bug Fix]: Errors in LiteLLM When Using Embeddings Model with Usage-Based Routing by @ishaan-jaff in #7390
• (Feat) Add input_cost_per_token_batches, output_cost_per_token_batches for OpenAI cost tracking Batches API by @ishaan-jaff in #7391
• (feat) Add basic logging support for /batches endpoints by @ishaan-jaff in #7381
• (feat) Add cost tracking for /batches requests OpenAI by @ishaan-jaff in #7384
• dd logger fix - handle objects that can't be JSON dumped by @ishaan-jaff in #7393

Full Changelog: v1.55.9...v1.55.10

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.10

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	200.0	218.24862748744047	6.256831142894005	0.0	1871	0	177.71721199983403	1940.1571020000574
Aggregated	Passed ✅	200.0	218.24862748744047	6.256831142894005	0.0	1871	0	177.71721199983403	1940.1571020000574

v1.55.8-stable

Full Changelog: v1.55.8...v1.55.8-stable

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:litellm_stable_release_branch-v1.55.8-stable

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	230.0	256.12454035233407	6.158450406948531	0.0	1842	0	207.30311900001652	2232.342858000038
Aggregated	Passed ✅	230.0	256.12454035233407	6.158450406948531	0.0	1842	0	207.30311900001652	2232.342858000038

v1.55.9

What's Changed

• Controll fallback prompts client-side by @krrishdholakia in #7334
• [Bug fix ]: Triton /infer handler incompatible with batch responses by @ishaan-jaff in #7337
• Litellm dev 12 20 2024 p3 by @krrishdholakia in #7339
• Litellm dev 2024 12 20 p1 by @krrishdholakia in #7335
• (fix) LiteLLM Proxy fix GET /files/{file_id:path}/content" endpoint by @ishaan-jaff in #7342
• (Bug fix) Azure cost calculation - dall-e-3 by @ishaan-jaff in #7343

Full Changelog: v1.55.8...v1.55.9

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.9

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	240.0	270.2192842992925	6.152704904591068	0.0	1841	0	213.0105499999786	2430.5650640000067
Aggregated	Passed ✅	240.0	270.2192842992925	6.152704904591068	0.0	1841	0	213.0105499999786	2430.5650640000067

v1.55.8

What's Changed

• fix(proxy_server.py): pass model access groups to get_key/get_team mo… by @krrishdholakia in #7281
• Litellm security fixes by @krrishdholakia in #7282
• Added sambanova cloud models by @rodrigo-92 in #7187
• Re-add prompt caching based model filtering (route to previous model) by @krrishdholakia in #7299
• (Fix) deprecated Pydantic Config class with model_config BerriAI/li… by @ishaan-jaff in #7300
• (feat - proxy) Add status_code to litellm_proxy_total_requests_metric_total by @ishaan-jaff in #7293
• fix(hosted_vllm/transformation.py): return fake api key, if none give… by @krrishdholakia in #7301
• LiteLLM Minor Fixes & Improvements (2024/12/18) p1 by @krrishdholakia in #7295
• (feat proxy) v2 - model max budgets by @ishaan-jaff in #7302
• (proxy admin ui) - show Teams sorted by Team Alias by @ishaan-jaff in #7296
• (Refactor) use separate file for track_cost_callback by @ishaan-jaff in #7304
• o1 - add image param handling by @krrishdholakia in #7312
• (code quality) run ruff rule to ban unused imports by @ishaan-jaff in #7313
• [Bug Fix]: ImportError: cannot import name 'T' from 're' by @ishaan-jaff in #7314
• (code refactor) - Add BaseRerankConfig. Use BaseRerankConfig for cohere/rerank and azure_ai/rerank by @ishaan-jaff in #7319
• (feat) add infinity rerank models by @ishaan-jaff in #7321
• Litellm dev 12 19 2024 p2 by @krrishdholakia in #7315
• Langfuse Prompt Management Support by @krrishdholakia in #7322
• Fix LiteLLM Fireworks AI Documentation by @jravi-fireworks in #7333

New Contributors

• @rodrigo-92 made their first contribution in #7187
• @jravi-fireworks made their first contribution in #7333

Full Changelog: v1.55.4...v1.55.8

Docker Run LiteLLM Proxy

docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.55.8

Don't want to maintain your internal proxy? get in touch 🎉

Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

Name	Status	Median Response Time (ms)	Average Response Time (ms)	Requests/s	Failures/s	Request Count	Failure Count	Min Response Time (ms)	Max Response Time (ms)
/chat/completions	Passed ✅	220.0	237.6551034099362	6.125601230624555	0.0	1832	0	193.92061900009594	1182.1513959999947
Aggregated	Passed ✅	220.0	237.6551034099362	6.125601230624555	0.0	1832	0	193.92061900009594	1182.1513959999947