AI Security Tools

The open-source toolkit for attacking and defending AI systems, ranked by GitHub stars — 66 tools across 6 categories.

66 tools

• promptfoopromptfooTest & red-team LLM apps, agents, and RAG pipelines
  26.0k
• SkillSpectorNVIDIASecurity scanner for AI agent skills — detect prompt injection, data exfiltration and supply-chain risks
  20.3k
• evalsopenaiFramework for evaluating LLMs and LLM systems — open-source benchmark registry
  19.6k
• PentestGPTGreyDGLAutomated penetration testing framework powered by LLMs
  16.0k
• lm-evaluation-harnessEleutherAIFew-shot evaluation framework for language models
  14.2k
• PresidioMicrosoftDetect, redact and anonymize PII in text, images and structured data
  11.3k
• PySyftOpenMinedPerform data science on data in someone else's server — federated learning and secure computation
  10.0k
• garakNVIDIALLM vulnerability scanner — probes for jailbreaks, prompt injection, data leakage
  9.6k
• guardrailsguardrails-aiAdd programmable guardrails to LLM outputs
  7.5k
• OpenLLMetrytraceloopOpen-source observability for GenAI/LLM applications based on OpenTelemetry
  7.5k
• NeMo GuardrailsNVIDIAToolkit for adding programmable rails to LLM conversations
  7.3k
• AI-Infra-GuardTencentFull-stack AI red teaming — agent scan, skills scan, MCP scan, LLM jailbreak evaluation
  6.9k
• Superagentsuperagent-aiProtect AI apps against prompt injections, data leaks and harmful outputs
  6.8k
• cleverhanscleverhans-labAdversarial-example library for benchmarking ML robustness
  6.5k
• ARTTrusted-AIAdversarial ML — evasion, poisoning, extraction, inference attacks and defenses
  6.3k
• FATEFederatedAIIndustrial-grade federated learning framework for privacy-preserving ML
  6.1k
• GiskardGiskard-AIEvaluation and testing library for LLM agents — detect hallucination, bias, security
  5.9k
• llm-attacksllm-attacksUniversal & transferable adversarial attacks on aligned LLMs
  4.8k
• PyRITmicrosoftPython Risk Identification Tool for generative AI red teaming
  4.6k
• dotsensity-aiDeepfake Offensive Toolkit for red-teaming face verification systems
  4.6k
• PurpleLlamameta-llamaMeta's tools to assess and improve LLM security (CyberSecEval)
  4.4k
• FedMLFedML-AIUnified federated learning library — distributed training with privacy
  4.1k
• safetensorshuggingfaceSafe way to store and distribute model tensors — prevent deserialization attacks
  3.9k
• jailbreak_llmsverazuo15,140 ChatGPT jailbreak prompts dataset from Reddit, Discord and websites
  3.8k
• TextAttackQDataAdversarial attacks, data augmentation and model training for NLP
  3.5k
• differential-privacygoogleGoogle's differential privacy algorithm libraries
  3.4k
• llm-guardprotectaiSecurity toolkit for LLM interactions — input/output scanning
  3.2k
• DeepTeamconfident-aiRed team LLMs and AI agents with 40+ attack methods — jailbreaking, RAG poisoning, PII leakage
  3.0k
• FoolboxbethgelabCreate adversarial examples that fool neural networks in PyTorch, TensorFlow and JAX
  3.0k
• HELMstanford-crfmHolistic Evaluation of Language Models (HELM) — reproducible LLM benchmarking
  2.9k
• whylogswhylabsData logging library for ML model monitoring — privacy-preserving profiling
  2.8k
• FaceForensicsondyariLarge-scale face forgery detection benchmark dataset (FaceForensics++)
  2.8k
• alibi-detectSeldonIOOutlier, adversarial and data drift detection algorithms
  2.5k
• torchattacksHarry24kPyTorch adversarial attack implementations (torchattacks) — FGSM, PGD, C&W and more
  2.2k
• llm-securitygreshakeResearch on indirect prompt injection — new ways of breaking app-integrated LLMs
  2.1k
• AI-Red-Teaming-PlaygroundmicrosoftAI red teaming playground labs with training infrastructure from Microsoft
  2.1k
• CheckListmarcotcrBehavioral testing of NLP models beyond accuracy — checklist methodology
  2.0k
• TensorFlow PrivacytensorflowTrain ML models with differential privacy for training data
  2.0k
• agentic_securitymsoedovAgentic LLM vulnerability scanner and AI red teaming kit
  2.0k
• OpacuspytorchTrain PyTorch models with differential privacy
  2.0k
• open-appsecopenappsecML-based web application and API security engine — preemptive threat prevention
  1.7k
• rebuffprotectaiPrompt-injection detector for LLM applications
  1.5k
• promptmaputkusenSecurity scanner for custom LLM applications
  1.3k
• Safety-Promptsthu-coaiChinese safety prompts for evaluating and improving LLM safety
  1.2k
• DeepfakeBenchSCLBDComprehensive deepfake detection benchmark — standardized comparison of methods
  1.1k
• HarmBenchcenterforaisafetyStandardized evaluation framework for automated red teaming and robust refusal
  1.1k
• agentic-radarsplx-aiSecurity scanner for LLM agentic workflows — detect risks in agent pipelines
  1.1k
• TenSEALOpenMinedHomomorphic encryption operations on tensors for private ML inference
  1.0k
• LangKitwhylabsToolkit for monitoring LLMs — safety, security and text quality metrics
  1.0k
• NVFlareNVIDIANVIDIA Federated Learning Application Runtime Environment
  981
• counterfitAzureCLI for assessing security of ML models — evasion, extraction, poisoning
  945
• diffprivlibIBMIBM Differential Privacy Library — mechanisms, models, and tools
  920
• JailbreakingLLMspatrickrchaoResearch on jailbreaking LLMs using adversarial attacks
  799
• RobustBenchRobustBenchStandardized adversarial robustness benchmark (NeurIPS 2021)
  790
• modelscanprotectaiScan ML models for unsafe code (deserialization attacks)
  784
• OpenAttackthunlpOpen-source package for textual adversarial attack research
  781
• ML Privacy MeterprivacytrustlabAudit data privacy in statistical and machine learning algorithms
  728
• lm-watermarkingjwkirchenbauerLLM output watermarking — embed and detect watermarks in model-generated text
  704
• jailbreakbenchJailbreakBenchOpen robustness benchmark for LLM jailbreaking
  687
• ficklingtrailofbitsPython pickling decompiler and static analyzer — detect malicious model files
  672
• ai-safety-gridworldsgoogle-deepmindRL environments illustrating AI agent safety properties from DeepMind
  639
• TrustLLMHowieHwongTrustworthiness evaluation of LLMs across safety, fairness, robustness and privacy (ICML 2024)
  632
• deep-pwningcchioMetasploit for machine learning — adversarial attack toolkit
  570
• PromptInjectagencyenterpriseFramework for quantitative analysis of LLM prompt injection robustness
  528
• ARESthu-mlBenchmarking adversarial robustness on image classifiers
  528
• BlindAImithril-securityConfidential AI deployment with secure enclaves
  512

Star counts from the GitHub API, refreshed daily by the pipeline. Inclusion is editorial — a curated set, not an exhaustive index.