- AllenNLP Interpret: A library for interpreting and visualizing LLM predictions, assisting in model explanation and debugging. It works for any model of your choice.
- LangKit: An open-source toolkit for monitoring Large Language Models (LLMs). Features include assessing text quality and relevance, hallucinations check, sentiment and toxicity analysis.
- BERTViz: Specifically designed for visualizing and interpreting BERT-based LLMs. Helps visualize attention in NLP Models (BERT, GPT2, BART, etc.).
- SHAP (SHapley Additive exPlanations): A game theoretic approach to explain the output of any machine learning model. Allows users to use models from the transformers library by HuggingFace.
- AI Fairness 360: An extensible open-source toolkit can help you examine, report, and mitigate discrimination and bias in machine learning models throughout the AI application lifecycle
- Prometheus: An open-source monitoring toolkit for collecting and querying metrics from LLMs in real time.
- Grafana: Integrates with tools like Prometheus and Elasticsearch to provide visualization and analysis of LLM metrics and logs.
Learn more about LLM monitoring and observability at https://www.turingpost.com/p/monitoring
