generic database assignment exposed on github

Search code, repositories, users, issues, pull requests...

Provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warning email from GitGuardian about SECRET_KEY #1687

jwhitlock commented Nov 14, 2020

• 👍 30 reactions
• 👀 6 reactions

nandish1199 commented Jul 24, 2021

Sorry, something went wrong.

waseyhasankhan commented Feb 12, 2022

ghost commented Apr 21, 2022

das-g commented Apr 21, 2022

Marderon123 commented May 29, 2022

• 👀 2 reactions

jayumaks commented Oct 14, 2022

mnemonico commented Dec 22, 2022

• 👍 2 reactions

das-g commented Dec 22, 2022

mcdwayne commented Jan 23, 2023

• 👍 1 reaction

das-g commented Jan 23, 2023

Mcdwayne commented jan 24, 2023.

amastaneh commented Apr 9, 2023 • edited

Mcdwayne commented apr 9, 2023 • edited, amastaneh commented apr 10, 2023, mcdwayne commented apr 10, 2023 • edited.

• ❤️ 1 reaction

fragmuffin commented Jun 10, 2023

Das-g commented sep 6, 2023.

thibaudcolas commented Oct 8, 2023

Thibaudcolas commented nov 18, 2023 • edited, mcdwayne commented nov 18, 2023, mcdwayne commented nov 20, 2023.

• 🎉 1 reaction

No branches or pull requests

Generic database assignment

Description ​.

The generic database assignment detector aims at catching any quadruple host, port, username, and password that are database credentials for which it wasn't possible to infer the database type.

This statement is pretty wide, therefore to avoid raising many false alerts, GitGuardian has come up with a range of validation steps and specifications to refine the perimeter to look at.

Specifications ​

The four components of the quadruple that the detector catches are referred as host, port, username, and password . The detector keeps only the combination of matched element that form a quadruple which are the closest matches inside the document. Another version of this detector exists for cases where the port is attached to the host name.

For this detector, each element must follow a specific set of rules to be considered as sensitive and therefore valid:

• The document must contain the string db or database (see whitelist hereunder ).
• Must be an assigned value except for the port, namely of the form {assigned_variable} {assignment_token} {value} , where {value} is either host , username , or password . The port can be either an assigned value or present in the host (for example my_host:some_port ).
• The document must contain the string host (see whitelist hereunder ).
• Caught hosts should be sensitive ones as well as their corresponding {assigned_variable} . Therefore, a set of common hosts are banned such as localhost , test/example hosts, or dummy IPs such as 1.2.3.4 and host assigned variables such as proxy (see banlist hereunder ).
• The document must contain the string port (see whitelist hereunder ).
• Caught ports should be sensitive ones as well as their corresponding {assigned_variable} . Therefore, a set of common ports are banned such as 8080 and port assigned variables such as support (see banlist hereunder ).
• The document must contain the string user (see whitelist hereunder ).
• Caught usernames should be sensitive ones as well as their corresponding {assigned_variable} . Therefore, a set of common usernames are banned such as db_user and username assigned variables such as user-agent (see banlist hereunder ).
• The document must contain the string pwd or pass (see whitelist hereunder ).
• Caught passwords should be sensitive ones as well as their corresponding {assigned_variable} . Therefore, a set of common password are banned such as encrypted or hashed ones and password assigned variables such as getpass (see banlist hereunder ).

Revoke the secret ​

This detector catches generic database credentials, hence GitGuardian cannot infer the type of database concerned. To properly revoke the secret:

• Understand what type of database is concerned.
• Refer to the corresponding database documentation to know how to revoke and rotate the credentials.

Examples that WILL be caught

Examples that WILL NOT be caught

• Host name is not a sensitive one.
• The IP is not sensitive.
• The username is not a sensitive one.
• The password is hashed.

Details for Generic database assignment ​

High Recall: False

Validity Check: False

Minimum Number of Matches: 3

Occurrences found for one million commits: 244

Prefixed: False

PreValidators :

• PostValidators

Was this page helpful?

• Specifications
• Revoke the secret
• Details for Generic database assignment

Something we didn’t cover?

See our roadmap, subscribe on github, submit a request, subscribe to our newsletter.

By submitting this form, I agree to GitGuardian’s Privacy Policy

Exposing secrets on GitHub: What to do after leaking credentials and API keys

If you have discovered that you have just exposed a sensitive file or secrets to a public git repository, there are some very important steps to follow.

Mackenzie Jackson

Mackenzie is the developer advocate at GitGuardian, he is passionate about technology and building a community of engaged developers to shape future tools and systems.

More posts by Mackenzie Jackson.

Liked this article?

Subscribe to the GitGuardian blog to receive all future articles directly to your mailbox.

As a developer, if you have discovered that you have just exposed a sensitive file or secrets to a public git repository, there are some very important steps to follow.

What is a secret? In this document when we use the term secret, we are referring to anything used to authenticate or authorize ourselves, most common are API keys, SSH keys, database credentials, or security certificates.

The first step, breathe: in most circumstances, if you follow this guide carefully, it will only take a few minutes for you to nullify most of the potential damage. This post will go through the four steps needed to remove the risk and make sure it doesn't happen in the future.

If I delete the file or repository, then I’m safe right?

Unfortunately, no. If you make the repository private or delete the files, you can reduce the risk of someone new discovering your leak, but the reality is that it is very likely your files will still exist for those that know where to look (bug bounty hunters, or worse...). Git keeps a history of everything you do, so even if you delete the file, it will still exist in your git history. Even making the repository private, deleting your history, or even the entire repository, your secrets are still compromised .

There are easy ways to monitor secret leaks; for instance, GitHub has a public API where you can monitor every single git commit that is made. That means that anyone can (and they do) monitor this API to find leaking secrets.

For this reason, it is best to assume that if you have leaked a secret, it is compromised forever.

Leaking secrets in public repositories on GitHub and then removing them, is just like accidentally posting an embarrassing tweet, deleting it and just hoping no one saw it or took a screenshot.

Step 1. Revoke the secret and remove the risk

The first thing we need to do is make sure that the secret you have exposed is no longer active so no one can exploit it.

If you need specific advice on how to revoke keys, you can see instructions within our API Best Practices Document .

If this key belongs to an organization you work for, then it is important to speak to the senior developers at your organization. It can be scary to let your company know you have leaked sensitive information, particularly if it has happened on a personal repository. But honesty is the best approach: remember that mistakes can be forgiven if the problem is resolved and you genuinely care.

Don’t hope that the problem will go away - mistakes happen, and it is best to be honest and upfront.

Did you know? Slack keys are among the rare API tokens that have the ability to auto-revoke themselves! As simple as using the auth.revoke endpoint!

Step 2. (Optional) Permanently delete all evidence of the leak

Once the secret has been revoked, it cannot be used anymore. Nonetheless, having credentials, even expired ones, could look unprofessional and raise concerns. Additionally, some secrets cannot be revoked (for example, database records), or credentials no one can guarantee were properly revoked (for example, SSH keys that can be used in many different places). While we will go through the steps of how to remove secrets from the git history, please note that this is not a trivial task, and it is advised to seek the guidance of a senior developer.

a. Either delete your repository or make it privat e

Should it become apparent that the entire repository had no place on public GitHub, navigate to your GitHub repository, then click

Then, all the way down to the "Danger Zone" and click " Make Private " to hide the repository from the public.

Note: if you wish to make a backup, then click on "Delete this repository." You will push it back later.

b. Rewrite the git history

I try to never rewrite git history but when @GitGuardian mailed me today about leaked keys I was on that rewrite like there is no tomorrow! Thanks GitGuardian! — Jeroen Claassens (@Favna_) May 11, 2018

Rewriting the git history can be very cumbersome, we strongly recommend that you follow the step-by-step instructions below before doing anything at the risk of messing up your team's work!

Step 3. Check your access logs!

This is very important, depending on the leaked secret. Sometimes when one access key leaks, it creates a domino effect and leads to exposing new secrets. For example, a Slack access key may give a bad actor access to messages containing new credentials and access codes, so it is very important to make sure that there is no suspicious-looking data!

Checking access logs really depends on the type of credential that was leaked. For example, AWS logs are sometimes centralized into Cloudwatch . Slack has a dedicated API endpoint that allows access to audit logs. This is probably a good time for you to reach out to your SRE or Application Security team to make sure everything’s fine!

So the credential has been revoked, and the repositories’ history has been cleaned. What should I do now?

Now you have had a good scare, it is the ideal time to start implementing some valuable security practices.

1. Get Protected with GitGuardian

GitGuardian detects secrets in your source code and integrates natively with GitHub, GitLab, Bitbucket, and Azure Repos. You can read more here and learn how it can benefit you as a developer.

GitGuardian is your code security platform to find and fix hardcoded secrets in your VCS, CI/CD tools, and infrastructure-as-code configurations – with GitGuardian’s code security policy engine.

If you want to shift your security left, prevent secrets leaks, and integrate GutGuardian anywhere in your software development cycle, learn how to install and use ggshield, the GitGuardian CLI:

Want to learn what the GitGuardian platform is all about? Watch the webinar replay below:

2. Review API Best Practices

To better protect your secrets in the future, we advise that you look at our API Best practice guide . It has lots of helpful tips on how to make sure you don't accidentally leak secrets in the future.

GitGuardian is helping developers secure software development with automated secrets detection & remediation for private or public source code. We scan public GitHub for secrets such as API keys or credentials in real-time. If you received an email from GitGuardian after committing to a public repository, it means we were able to find a secret in your code. Don’t panic! Just click here to find the steps that will help you remediate this potential leak. If this is a false positive, we apologize for the alert. Secrets detection is probabilistic: our algorithms achieve a tradeoff between not raising false alerts and not missing exposed credentials.

You can find all pricing information on our pricing page .

Although Secrets detection is a new category, we are currently gathering vetted customer and user reviews of GitGuardian. You can find reviews on the following sites: Peerspot Capterra Gartner Peer Insights G2 Trustradius You can also find out what developers have to say about us and how customers such as Talend, Bokeh, Mirantis or Genesys use and benefit from GitGuardian

Related Articles

What to do if you expose a secret: how to stay calm and respond to an incident.

Learn how to respond to a secret leak incident effectively. Follow our step-by-step guide to understand the impact, rotate secrets safely, and prevent future leaks.

Dwayne McDaniel

11 Nov 2023 – 11 min read

Webinar Recap: Hands-on guide to Runtime Security for CI/CD Pipelines with StepSecurity

A condensed recap of our hands-on runtime security webinar from September. Get the juiciest knowledge nuggets and pointers to more.

Greg Bulmash

7 Nov 2023 – 3 min read

How to Handle Secrets in Helm

Learn step-by-step techniques and best practices to handle secrets in Helm charts safely and effectively. Level up your Helm deployments today!

Guest Expert

25 Oct 2023 – 16 min read

Secrets management takes more than just tools

Can you just purchase a tool to give you good security posture? Discover how People, Processes, and Tools elevate code security to protect against data breaches.

27 Sep 2023 – 12 min read

Start your journey to secrets-free source code

And keep your secrets out of sight

GitGuardian is the code security platform for the DevOps generation. With automated secrets detection and remediation, our platform enables Dev, Sec, and Ops to advance together towards the Secure Software Development Lifecycle.

Subscribe to our newsletter to receive the latest content and updates from GitGuardian.

By submitting this form, I agree to GitGuardian's Privacy Policy

• PLATFORM Secrets Detection Honeytoken Infra as Code Security Public Monitoring GITGUARDIAN VS GitHub Advanced Security GitLab Secret Detection truffleHog v3 Gitleaks More alternatives
• DEVELOPERS GitGuardian CLI Documentation API Documentation Good Samaritan Labs GitHub Roadmap API Status
• RESOURCES Blog Learning Center Dev & Sec resources State of Secrets Sprawl Events Free GitHub Secrets Audit PRICING Plans Value Calculator
• COMPANY About Us Wall of fame Careers Contact us FAQ Partners Newsroom

© 2023 GitGuardian. All Rights Reserved.

DEV Community

Posted on Oct 15, 2021 • Updated on Mar 11, 2022 • Originally published at blog.gitguardian.com

Why detecting generic credentials is a game changer

At our core, you probably know that we are a company specializing in detecting secrets (if you don't know what a secret is, please take a moment here and come back). Very early on, we had to address the question: what would be a good way to categorize secrets?

Take a look at this:

You can spot the difference: the first one is tied to a well identified service, AWS, while for the second, things are a bit blurrier: we immediately understand that it has to do with a database connection, but, without further context it doesn't tell us what doors it opens.

That's the most basic distinction we can make when scanning source code: some secrets are specific , since they are somehow self-revealing, while others are said to be generic , because we cannot be so sure what they give access to..

In this article, we intend to explore why detecting generic credentials is an absolute must have for a secrets detection engine. We will also explain how we addressed this topic at GitGuardian, and give some insights on our findings.

Specific detection has advantages but is not sufficient

Detecting specific credentials has at least two big advantages over detecting generic ones.

First of all, we are often able to test the validity of specific credentials, which can give us 100% confidence in the secret's validity. This ensures a very good precision overall.

Second, they are often associated with very well known patterns, sometimes even prefixed ones, or at least a very specific context. This means a very good recall is easy to obtain.

But don't be fooled: being "the easy part of the game" doesn't mean that they are any less valuable: eventually, the user can be provided with very detailed information about the exposed secret, the risks incurred or the correct way to revoke credentials. That's what we thrive to do in our public documentation for the 300+ detectors we are covering.

Although we have drastically improved our average time to develop a new specific detector, shrinking it from 2 days to 2 hours, scaling this list is not easy. The reason is straightforward: the number of API providers is growing at a very fast rate.

But what about the other category? As said earlier, some of the credentials we found are simply not linkable to any particular service: think about contextless passwords, combinations of usernames and passwords for an internal service, or just an API key with a very generic name. We estimate that almost half of the secrets we find belong to this category.

You may be wondering what happens if some credentials are detected by both generic detectors and specific detectors. In that particular case, GitGuardian always gives priority to the specific detector for the reasons we listed above. But note that this is not an issue and even rather a clue that our generic detection performs well and can act as a failsafe in case something went wrong with the concerned specific detector.

You get where this is going: if a secrets detection engine wants to achieve the best possible precision AND recall, it needs a tailored and powerful detection for generic credentials.

How it's done at GitGuardian

Why generic detection is not so easy to do….

As the name suggests, when looking for generic credentials, the contextual information we are looking for is… generic. Narrowing down candidates is therefore a bit more complicated. For instance, targeting all the password keywords is obviously not as effective a filter as targeting files containing aws and client_secret .

Generic credentials detection difficulty results from 3 factors . First, they are widely different, being made from very broad patterns: charset and length can be almost anything. Second, how they are supposed to be used is also unknown. Third, even when the credential is clearly identified, we have no way to check its validity.

By the way, a quick reminder on the importance of having both a good precision and a good recall. Take for example this valid, generic, secret:

We could certainly catch this one by filtering for all the random looking strings in our engine (namely, high entropy strings). But we would also certainly catch a lot of random strings that are not secrets (think UUIDs, hashes...), ruining our precision rate. So entropy alone is not a sufficient criteria if we want to limit noise and save the engineers from alert fatigue.

On the other hand, if the engine only targets very specific assignments like apikey = abc , we would miss a lot of generic credentials that are valid secrets resulting in low recall. Worse, we never know for sure what the proportion of missed secrets is (e.g. the rate of false negatives). For the user, it means a low level of confidence in the tool.

Generic detection is a real challenge that requires techniques of its own. At GitGuardian, our approach is twofold: first, the idea is to maximize recall and avoid blind spots by looking for very broad assignments in source code. Second, we want to have powerful tools to sort the results and discard false positives in an efficient way, so to guarantee a high precision and avoid alert fatigue.

GitGuardian's arsenal and tools

As mentioned earlier, an important and first part of our approach is to detect a wide variety of assignments in source code. To do so, we came up with a wide variety of possible assignments inspired from many languages. Here are some tricky examples that we can detect:

Having this capability significantly improves our recall. Then an important part of our work is to discard false positives as early as possible in the process. At GitGuardian, we designed a wide arsenal of post validation steps to decide whether a secret should be processed any further or not. Here are details about some of these so-called post-validators.

ContextWindowPostValidator : This post validator bans irrelevant matches based on contextual information. For instance, we consider that a match that contains pubkey in its close context can safely be discarded.

CommonValuesBanlist : This PostValidator leverages dynamic banlists that are computed and adjusted according to the live monitoring of GitHub. More specifically, we are looking for example keys, or patterns that are so common that we consider them as invalid secrets. Here are some simple examples:

And many other common values for passwords, usernames or high entropy values.

AssignmentBanlistPostValidator : That's a very powerful and unique feature of GitGuardian's secrets detection engine. For each language, we are able to identify the variable to which a secret was assigned, if it exists. We can then ban some patterns in the assignment variable. For instance all assigned variables containing "uuid" suggest that the value matched is not a secret but an identifier.

Key Figures and Insights

At the end of the day, GitGuardian has developed more than ten generic detectors, scoring between 85% and 95% for precision according to our benchmarks. These are the most common:

Overall, generic detectors account for 45.4% of all the secrets we detect. This means that any secrets detection solution that does not implement generic detection algorithms, misses at least half of the secrets present out there.

Another interesting metric : close to 25% of secrets found by specific detectors would have also been found by generic detectors. This indicates that generic detectors appear to be a very good fallback in case a specific detector behaves badly or simply does not exist yet.

Finally, our efforts to improve generic detection brought up interesting side-effects that we were able to exploit:

• Detecting pattern drift : When we detect that a specific detector yields less credentials over time, and if in the meantime, we witness the appearance of generic credentials with the name of the concerned provider in their context, we can conclude that a change occurred in the pattern for this provider. This has proven very useful to constantly be up to date with vendor's changes.
• Detecting new candidates for specific detectors : If all of a sudden a lot of generic credentials mention a given word in their context, we can conclude that this corresponds to a new API provider gaining notoriety. We even have an internal tool to infer the pattern for the concerned credentials and be up to date with developer's practices as fast as possible.

Implementing solid generic detection capabilities is a significant improvement for recall while keeping a very good precision. It is therefore a huge competitive advantage compared to other tools. What's more, generic detection offers some serenity for our customers: we may not have a specific detector targeting this very special kind of secret, but, in most cases, our generic detectors have our customer's back, and we keep getting better at detecting generic credentials.

Top comments (0)

Templates let you quickly answer FAQs or store snippets for re-use.

Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink .

Hide child comments as well

For further actions, you may consider blocking this person and/or reporting abuse

[Alien Hunter Series Pt. 1] Mastering the COBOL Programming Language

Saurabh Rai - Nov 6

24 Open-Source Projects for Developers in 2023 🔥👍

Madza - Oct 17

Web5 Meets Generative AI: Reimagining the Future of Digital Identity with Netonomy

Rizèl Scarlett - Nov 14

My first npm package

Rafael Magalhaes - Nov 4

Once suspended, gitguardian will not be able to comment or publish posts until their suspension is removed.

Once unsuspended, gitguardian will be able to comment and publish posts again.

Once unpublished, all posts by gitguardian will become hidden and only accessible to themselves.

If gitguardian is not suspended, they can still re-publish their posts from their dashboard.

Once unpublished, this post will become invisible to the public and only accessible to Pierre Lalanne.

They can still re-publish the post if they are not suspended.

Thanks for keeping DEV Community safe. Here is what you can do to flag gitguardian:

gitguardian consistently posts content that violates DEV Community's code of conduct because it is harassing, offensive or spammy.

Unflagging gitguardian will restore default visibility to their posts.

We're a place where coders share, stay up-to-date and grow their careers.

• Generative AI
• Business Operations
• IT Leadership
• Application Security
• Business Continuity
• Cloud Security
• Critical Infrastructure
• Identity and Access Management
• Network Security
• Physical Security
• Risk Management
• Security Infrastructure
• Vulnerabilities
• Software Development
• United States
• United Kingdom
• Newsletters
• Foundry Careers
• Privacy Policy
• Cookie Policy
• Member Preferences
• About AdChoices
• E-commerce Links
• Your California Privacy Rights

Our Network

• Computerworld
• Network World

HasMySecretLeaked finds exposed secrets in the GitHub repository

GitGuardian's new free service lets organizations check for exposed secrets from a database of 20 million records.

Exposing hard-coded credentials and sensitive secrets through public code repositories has been a major security risk for organizations for years, with over 10 million new instances of credential leaks detected on GitHub alone in 2022. A new free service called HasMySecretLeaked now allows organizations to securely and privately check if any of their secrets are in a database of 20 million exposed records collected by security firm GitGuardian since 2020.

GitHub already has its own free service that notifies repository owners if secrets are detected in their public repositories, but the types of secrets that are monitored are typically cloud API access keys or other access token formats provided by partners. GitGuardian’s HasMySecretLeaked covers many more types of hard-coded secrets , both service-specific and generic ones, including database passwords, encryption keys, username and password combinations, messaging tokens, SSH credentials, and email passwords.

The company has been scanning every public code commit on GitHub for hard-coded secrets for the past several years, refining its detection algorithms, expanding the list of supported credential formats, and lowering false-positive rates. In 2020 it uncovered 3 million exposed secrets on GitHub, in 2021 it found 6 million, and in 2022 over 10 million.

GitGuardian used its research to release an annual report called The State of Secrets Sprawl as well as to build and enhance its own code security platform that prevents developers and engineers from accidentally leaking secrets in their code, build scripts, Docker images, configuration files and so on.

Search your own repositories vs. searching all

Secret-detection services have generally been built with the goal of serving repository owners. GitHub will notify the repository owner if a secret is detected in a repository they own and will also notify a partner service like AWS if the secret is an AWS key so that Amazon can make the decision to revoke it before it’s abused. GitGuardian’s own security platform will notify the organization if a secret is found anywhere in their software development pipeline: code, Docker images, DevOps environment, etc.

However, HasMySecretLeaked was built with another goal: to let organizations check if any of their known secrets were leaked anywhere on GitHub, including repositories owned by other parties. External leaks are not unusual. For example, one of the company’s developers might decide to publish a piece of code in his own public repository and accidentally forgets to scrub one of the organization’s tokens. Or a company’s developers are allowed to contribute to a community project but forget to remove a private database URL that includes credentials.

In fact, HasMySecretLeaked is similar in approach and takes a lot of inspiration from HaveIBeenPwned , a service by security researcher Troy Hunt that allows users to check if their emails and passwords have been leaked in publicly known data breaches. In both cases, care had to be taken to prevent attackers from abusing the service and to perform the searches without exposing the secrets to the service owner.

Building an API search for secrets without leaking secrets

After deciding they want to build a service that lets users search their huge database of leaked secret incidents, the GitGuardian researchers came across the first implementation problem: How will users submit their secrets to GitGuardian’s service without GitGuardian actually seeing them in plain text and creating a privacy and security problem?

The answer might seem straightforward: Use hashing. Hashing is a cryptographic representation of a string that is supposed to be irreversible — although some older and weak hashing schemes can be cracked using brute-force methods. Think of it as one-way encryption where the key is destroyed. This doesn’t fully solve the problem, though.

“By definition, if the hashed secret is present in our database, it implies that its cleartext version was once publicly accessible, indicating that GitGuardian has or had knowledge of it,” the GitGuardian researchers explain . “In other words, the user would be leaking their secret to us. And that's not acceptable either.”

The solution is to send only a fragment of the hash — for example the first five characters. This would likely match multiple entries in the database that happen to have hashed versions that start with the same five characters. GitGuardan wouldn’t know which of those is the user’s secret or if it’s even any of them, so it would have to return all the entries that match from its database along with the location where they were seen. This generates another privacy problem: The user would receive the location of leaked secrets it doesn’t own.

So how to ensure the user can only read the part of the response that matches their own secret? Some clever use of encryption. The service encrypts each of the entries (secret + location) with the full-length hash of the leaked secret. GitGuardan knows the plaintext and therefore hash of every leaked secret so they can use the hash as an encryption key. The user only knows the full hash of their own secret they wanted to search to use as a decryption key.

Therefore, if one of the entries returned by GitGuardian in the “response bucket” based on the first five characters of the hash sent by the user happens to correspond to the user’s secret, the user will only be able to decrypt that entry from the response.

While this solves the privacy issue, it’s not very efficient because the client on the user side must use its hash to try to decrypt a full list of entries. That could take a long time. Imagine scaling this to a long list of secrets and hashes the user wants to search for. It would be easier if the service provides an indicator or hint of which entry from the response is likely to match and should be decrypted instead of trying them all. The answer? A hash of the hash.

In addition to using the hash of a secret itself as the encryption key for more information about the secret such as the source of the leak, the service hashes the hash of the secret, producing another hash and sends it alongside the entry as a hint. The client knows his own secret and its corresponding hash and can hash the secret’s hash with the same algorithm the service uses and then compare it to the hints in the response entries. If it matches any of the hints, the client knows that’s the entry it can decrypt.

Additional safeguards are in place to prevent abuse of the service by attackers who want to use the service to check for leaked secrets or find the location of secrets for which they somehow have a leaked hash that was generated using the same algorithm as the service. The service uses a “pepper,” a global unique value that gets combined with hashes to make the hashing algorithm more unique.

Also, rate-limiting for non-authenticated users limits them to five queries per day. Authenticated users have limits as well but less stringent. The service will also only return the first location where a secret was found, even though that secret might have been found in multiple locations inside a repository or in multiple repositories. This is to restrict the amount of information that can be misused in case an attacker abuses the system.

However, this information restriction also limits the investigation capabilities of organizations who might want to know not only if one of their secrets was leaked, but also how many times, where it happened, and who leaked it. The answers to those questions might be valuable to deciding incident response actions beyond just revoking that secret.

How to use HasMySecretLeaked and future plans

In addition to using the service through the web interface on the HasMySecretLeaked website, users can query the API by using GitGuardian’s command-line tool called ggshield . The tool requires an account with GitGuardian’s platform, which is free for organizations with teams of up to 25 developers.

GitGuardian tells CSO that in the future it hopes to expand the service and secrets scans beyond just GitHub repositories to things like packages in public registries like npm and PyPI, as well as public Docker images on DockerHub. They are also considering whether all the locations of a secret could be shared as part of the response.

Related content

Moveit carnage continues with over 2600 organizations and 77m people impacted so far, the alarming rise of quishing is a red flag for cisos, the sso tax is killing trust in the security industry, cloud identity security success: 3 critical factors, from our editors straight to your inbox.

Lucian Constantin writes about information security, privacy, and data protection for CSO.

More from this author

Ransomware gang files sec complaint against company that refused to negotiate, palestine-aligned cyberespionage actor shifts infection chain tactics, intel patches high-severity cpu privilege escalation flaw, misconfigured docker api endpoints allow attackers to deliver ddos botnet agent, most popular authors.

• Jon Gold Senior writer

Show me more

Identity security’s crucial role in safeguarding data privacy.

Why the DOD's Replicator should be a model for cybersecurity

Australian federal government announces cybersecurity support for SMBs

CSO Executive Sessions Australia with Robbie Whittome, CISO at Curtin University

CSO Executive Sessions / ASEAN: Cisco's Anthony Grieco on opportunities in Southeast Asia's cybersecurity landscape

CSO Executive Sessions Australia with Leron Zinatullin, CISO at Linkly

AI and Cybersecurity: Speed Bumps, Training, and Communication

CSO Executive Sessions Australia with Robbie Whittome

Sponsored Links

• Unified identity security is the new imperative. SailPoint has the roadmap for success - Learn more
• Tomorrow’s cybersecurity success starts with next-level innovation today. Join the discussion now to sharpen your focus on risk and resilience.

Research | Our Q3 report details what's new in the world of ransomware.

Access Keys Exposed: More Than 40% Are For Database Stores

• Threat Intelligence

By now, we’ve all heard news about AWS keys leaked by a developer on GitHub. While this can cause damaging headlines for the company, fortunately GitHub responded and can now automatically invalidate these API keys when they wind up on public repositories. 

This is great, but limited by two critical factors:

• GitHub does not monitor for all API key and secret types, and specifically it does not extend to a number of database stores
• This type of monitoring does not help if these keys are exposed across other code repositories or paste sites 

Over a 30-day period, we scanned more than 150 million entities from GitHub, GitLab, and Pastebin. During this time, our technology assessed and categorized almost 800,000 access keys and secrets. What did we find? More than 40% of these were for database stores – the majority not covered by GitHub.

What Leaked Access Keys Matter

Typically, when we think of credentials, the first thing that comes to mind is a username and password widely used by people and other systems to authenticate to systems. But in software, there’s an additional type of security credential – an access key. Access keys can be public or private and, depending on the type of services, provide system authentication to third party or internal systems. Access keys often have broader access than individuals and fewer checks/restrictions on their usage.

Unfortunately, these access keys can be exposed by internal software developers or contractors, who may have not noticed a repository’s settings have changed to public. Indeed, this was the case with recent headlines made by Starbucks and AWS . This is a relatively common occurrence; previous research from North Carolina State University discovered that over 100,000 GitHub repos have leaked API or cryptographic keys .

The misuse of these keys is not hypothetical. Last year, Imperva outlined how a breach resulted by stealing an AWS API key . More recently, in August, researchers discovered malware stealing AWS credentials for the purpose of crypto-mining. 

Hunting for Exposed Keys

Most analyses of exposed access keys have focused on GitHub, and for good reason. There’s an enormous amount of data and leaks we see. However, for this research, we also looked across GitLab and Pastebin to provide a more comprehensive idea of how often keys get exposed. Over a thirty-day period (9th August – 8th September 2020), we searched across approximately 150 million entities across GitHub, GitLab, and Pastebin.

Of the 800,000 access keys assessed in this time period (which included both our historical archive and new commits), we  broke down the 20 key types into four categories: databases, online services, cloud providers, and SSH keys. The breakdown of the number of keys is provided in Figure 1. 

Database Stores

The potential impact of exposed access keys is most obvious when we consider database stores. If exposed, these types of credentials could allow unauthorized access to company data (including PII) with the permission to expose, destroy or manipulate company data. 

For many years we have witnessed the targeting of MongoDB as part of ransomware campaigns (most recently there were 22,900 MongoDB instances held to ransom). 

But the impacts extend beyond this tactic. Depending on the nature of the data, such unauthorized access could have regulatory consequences, disrupt business critical systems, and damage the reputation of the organization.

We searched for 8 types of API credentials for the following databases: IBM DB2, Microsoft SQL Server, MongoDB, MySQL, Oracle DB, PostgreSQL, RabbitMQ, and Redis. In total, we discovered 129,550 credentials for these 8 database stores, with Redis (37.2%), MySQL (23.8%), and MongoDB (19.3%) the most common.

Cloud Providers

The second area outlined in this research focused an analysis of almost 300,000 keys across four types of cloud providers: AWS, Azure storage, Azure SAS, and Google Cloud. 

Successful authentication into these types of environments could allow access to the associated cloud infrastructure, with permission to expose, destroy and/or manipulate sensitive data. The data accessible depends on the services used and could include company information or and internal systems information. Theft of this type of information can be highly valuable for cybercriminals. Furthermore, as we have seen with the recent targeting of AWS keys for crypto mining purposes , there are many ways to monetize this type of access.

Online Services

The research focused on the following key types for online services: Google OAuth ID, Mailgun, Microsoft Nuget, Slack (Bot Token, User Token, and Webhook URL), and Stripe. Google OAuth was the clear majority, with 95% of the instances.This is somewhat concerning, given that this can be used to obtain permission from users to store files in their Google Drives.

Due to the high number of Google OAuth ID, these have been omitted from Figure 4 in order to better illustrate the other types of online services. 

Of more than 4,000 secret or API keys for online services, the majority (56.9%) were for Slack. These may either be used to trick users into clicking links or disrupt business operations. They include:

• A webhook URL , which could be used to post messages directly into a channel within the organization.
• Slack Bot token , which would give access to sensitive information on channels and conversations that the bot user is invited to. 
• Slack User token , which would give access to the users Slack workspace, e.g. the channels, conversations, users, and reactions.

It’s about more than Slack, of course. Access to a Microsoft Nuget API key, for example, could enable actors to upload malicious packages or delete existing packages from a code repository.

Even though a relatively small number of Stripe secrets were unearthed (274), it can have a high impact. In this case, access could result in the exposure of sensitive financial information, and allow an attacker to modify and delete information within the account. 

Alternatively, a Mailgun secret key could allow use of the API to send, receive and track emails – an incredibly useful type of access for phishing campaigns.

Finding Your Own Keys

For GitHub, there are a few options for gaining this visibility:

• Trufflehog . Searches through git repositories for secrets, digging deep into commit history and branches. This is effective at finding secrets accidentally committed.
• GitRob . Helps find potentially sensitive files pushed to public repositories on Github
• GitHub Secret Scanning . GitHub provides monitoring for many of the key types outlined in this blog. Although this doesn’t extend to many of the database stores (Redis, Oracle, MySQL, IBM DB2, and PostgreSQL), it’s a great start. 

Of course, this is about more than just GitHub, and so it’s worth referring to help provided by the technologies themselves. Google, for example, provides helpful guidance on steps to take if you unearth an exposed access key. 

Get in touch to learn more about how we help organizations to detect exposed access keys and other types of technical leakage! You can read more about our technical leakage capability here . 

Git Guardian

Full-featured free secret scanning for open source and small teams on GitHub and GitLab

Sign up and get API Key from your GG Dashboard

Get api key, other integrations, review bucket_s3.py scan result, open email “uncovered secrets”, try business collaboration pricing, edit $home/.env file, install and update using pip, use local git hooks to run on pre-commit, manual run steps, invoke scanning of public repo, configure your .gitguardian.yml, create a service account email and github service account, question: adding custom scanning rules, question: enterprise-wide trend analysis.

This is a hands-on introduction to Git Guardian (“GG”), a utility to detect (monitor) API keys and other credentials and secrets exposed in source code on public SaaS or private (internal/on-prem) GitHub.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

GG’s documentation is published at https://docs.gitguardian.com/internal-repositories-monitoring/home

PROTIP: Know the Glossary of technical terms before you dive into other docs, at: https://docs.gitguardian.com/internal-repositories-monitoring/glossary

“Why” and FAQ questions are answered in a blog article that’s also called “Learning Center” at https://www.gitguardian.com/secrets-detection .

All GG’s source is kept under Github account https://github.com/GitGuardian .

PROTIP: Specific advice is provided on GG’s https://github.com/GitGuardian/APISecurityBestPractices repository, which is in a zip file you get via email after you sign up to get their whitepaper “Implementing Automated Secrets Detection for Application Security” at https://www.gitguardian.com/secrets-detection-whitepaper

GG’s Leak Mitigation Checklist and Good development practices page is useful if you just leaked sensitive information in public source code. It lists what to do for each of the most common services: Alibaba, Algolia, AWS, DigitalOcean, Google, GitHub, GitLab, Heroku, HubSpot, Mailgun, SendGrid, Slack, Twilio, Twitter

GG’s Good development practices page (leak prevention practices)

[email protected]

At time of writing:

https://twitter.com/gitguardian?lang=en lists 814 followers

https://www.linkedin.com/company/gitguardian/ lists 36 employees:

CEO Jérémy Thomas ( oo-de-lally ) is based in Paris and San Francisco

Eric Fourier is co-founder.

https://twitter.com/PonicodeDev at #GitHubUniverse 2020.

Mackenzie Jackson (@advocatemack), GG’s Dev Advocate, wrote blog articles which are repeated in the website:

• https://blog.gitguardian.com/secret-sprawl 24 Jul 2020

In my blog article here, I wrote down the steps I took on my attempt at installing and making use of the GG web GUI and tools.

Use a browser to be at https://www.gitguardian.com

Click the blue “SIGN UP FOR FREE / Internal Monitoring”.

Click “GitHub”. ???

If you have a repository on GitHub account, specify it.

Signing up adds a browser cookie to your account’s dashboard at URL https://dashboard.gitguardian.com

Logic in the JavaScript would redirect your browser to your own account’s dashboard.

• Click the “API” icon on the left menu of your GG Dashboard.
• Scroll down to the bottom of the page for section title “Generate new API key”.
• In the Name field, type your name, such as “John Doe GUI test 1”.

Click “Crate new API key” for an API key such as (for example):

PROTIP: You don’t need to save that key if you’re only using the GUI page shown in the next section.

Alternately, if you want to run a scan anywhere else than the GUI you’re on, follow instructions below , then return here.

Click the “INTEGRATIONS” icon on the left menu of your GG Dashboard.

By default, you’re installed to use GitHub.com SaaS.

Scan sample code on GUI Dashboard

Get a taste of the response output by using the Dashboard to run a scan of GG’s sample Python and JavaScript programming code containing AWS and MongoDb secrets.

Click the “+” at the top of your Chrome browser window to open another browser window to the GG’-provided sample Python program files containing AWS and MongoDb secrets at:

https://github.com/GitGuardian/sample_secrets/blob/main/bucket_s3.py

Click “Raw”, click on any line of the file. Press control+A to select all. Press control+C to copy what has been selected.

• Switch back to the GG Dashboard.
• Click the “API” icon on the left menu of your Dashboard.
• Click within the blue box under title “Submit your text content” and press control+V to paste.

Click the blue “Scan”.

Click the blue icon to the right of the “API response” field to copy GG’s response (in JSON format).

In an editor, create empty file bucket_s3.py.json and paste the result file in it…

Notice the AWS Client Id and Client Secret in this line:

Also notice the database username (“testuser”) and password (“hub24aoeu”) in this line:

PROTIP: Pasting would remove line breaks:

CAUTION: Responses contains secrets you don’t want others to see (which is why we scanned GG’s sample code). So please don’t format to human-readable (indented) JSON using public web pages such as jsonformatter.curiousconcept.com .

Repeat the above procedure, except use the contents at

https://github.com/GitGuardian/sample_secrets/blob/main/postgres_model.js

Notice the secret in this line:

There is also an in-line secret in this line:

The result:

Click the “INCIDENTS” icon at the left menu on your GG Dashboard.

Note that text scanned in the GUI doesn’t count as incidents of secrets found.

Scan my own (vulnerable) repos

Click the “PERIMETER” icon at the left menu on your GG Dashboard.

Click “ADD SOURCES” if you want to specify more repositories to scan.

I specify to repos which were created to contain vulnerabilities, including secrets:

See my website at https://wilsonmar.github.io/webgoat

• PROTIP: Start a stopwatch to time how long it takes.
• Click the blue “Launch scan” button.

Watch and click your stopwatch when “100%” appears next to “REAL TIME MONITORING”.

NOTE: Sources are removed from the list after scanning.

• Below the “HEALTH” drop-down, click the arrow next to the list.

Check or uncheck the type of health determined for each source to be displayed.

NOTE: Icons under “PROTECTION” identify whether each source is being monitored.

Mouse over or click the time icon to see a pop-up of the date/time of last scan.

In your email client, open the email from “[email protected]” with subject

Click a repo listed under “repository is affected:” to open the repo on your default browser.

QUESTION: “17 secrets”???

View the comparison of plans at https://www.gitguardian.com/pricing .

PROTIP: GG scanning is free on SaaS for open source and up to 25 developer teams.

“STANDARD” “Business” accounts are $434 per month per developer , or $1667/month for 100 developers after a 30-day trial.

View the comparison of plans at SETTINGS > General .

PROTIP: The free plan has a limit of 1,000 (1K) calls per month. The “BUSINESS” plan has a limit of 10,000 (10K) calls per month.

Notice the difference between FREE and “BUSINESS” GG accounts is the “ Private collaboration repositories”.

“Collaborative” repos have “Teams” on the organization’s menu:

Within a Team is this menu with Members and Repositories associated with that Team:

Start your business trial at SETTINGS > General by clicking the blue “Start 30-day trial” button.

PROTIP: Git-guardian does not ask for your credit card up-front.

A Business plan enables you to more run options:

Invoke locally using CLI

Scan repos in on-prem GitHub Enterprise instance

Install and run on Github Actions

https://github.com/GitGuardian/gg-shield#circle-ci

Invoke from CLI scan of github.com repos

Highlight and copy these two lines into your Clipboard.

The API_URL stays the same.

In the editor, also open file .env in your laptop’s Home folder.

If the file doesn’t already exist, create a new file with that name.

In the .env file, position your cursor at the end of the file and paste from Clipboard.

Save the file.

NOTE: https://github.com/GitGuardian/gg-shield#installation mentions addition variables:

Save API to invoke from CLI scan of github.com repos

Follow the steps described in the section above .

With your editor open on file .env , highlight “ __ __FILL ME __ __” and replace it with your invisible Clipboard.

QUESTION: GG_API_KEY=”123456789…” ???

Alternately, if you are running on GitHub Actions, navigate to your project settings and paste the GITGUARDIAN_API_KEY secret there.

Open a Terminal window.

You can run my script that does all the following:

Install Python 3.6 and newer

In Terminal, navigate to the folder obtained from GitHub.

Check if pipenv

If that is valid, install gg-shield using pipenv

Alternately,

Check if using virtualenv:

The APIs called are defined at https://api.gitguardian.com/docs which provides an on-line API specification and Authentication description.

QUESTION: Swagger?

Now that ggshield is installed:

Install git pre-commit hooks in local (not global) mode:

Run the last build in CI, with defaults:

Run GitHub Actions

To invoke on a server from the GitHub Actions Marketplace at https://github.com/marketplace/actions/gitguardian-shield-action

In Terminal, create/navigate to a folder holding repositories created/cloned.

Obtain GG’s repo:

https://github.com/GitGuardian/py-gitguardian/blob/master/examples/content_scan.py

https://github.com/GitGuardian/py-gitguardian/blob/master/examples/directory_scan.py

[1] For running on a public Github.com repo, GG has a secrets scanning API library (written in Python 3.5+) at https://github.com/GitGuardian/py-gitguardian

The README says GG scans for “200 types of secrets”.

Source code for the GG scanning engine CLI code is at https://github.com/GitGuardian/gg-shield

It is invoked either locally on your laptop or invoked from GitHub Actions Marketplace at https://github.com/marketplace/actions/gitguardian-shield-action

Create an API key on the API Section of your GG dashboard .

To configure your .gitguardian.yml file:

View the contents based on the contents of the file at https://github.com/GitGuardian/gg-shield/blob/main/.gitguardian.example.yml

Click “Raw”, click on any part of the file, press control+A,

Add line(s) under paths-ignore to specify wild cards (*)

GG is run using GitHub Actions invoking https://github.com/GitGuardian/gg-shield-action

Run within on-prem GitHub Enterprise instance

GitGuardian Private Repository Monitoring is a Kubernetes application. You can install the software on an existing cluster or use our installer that has an embedded, production-ready Kubernetes distribution packaged with it. See https://docs.gitguardian.com/internal-repositories-monitoring/self_hosting/replicated_installation

Setup Single Sign On (or SSO) allows you to manage your workspace authentication and membership via a third-party identity provider.

Install GG app on your GitHub Enterprise on-prem. server.

GitGuardian integrates with GitHub Enterprise instance through a GitHub app that we need you to create. GG enables you to do so programmatically via GitHub manifest. This will ensure that your GitHub App is created with all the appropriate rights.

Provide GG with your GitHub Enterprise url.

WARNING: The GitHub app will be owned by the GitHub user who created it. We therefore recommend that you subsequently transfer ownership of the GitHub app to a bot user or a GitHub organization.

Once created, you will be prompted to install the GitHub app on the GitHub organization of your choice. The installation flow even allows you to individually select repositories that you would like to give GitGuardian access to!

You might also enjoy ( View all posts )

• Acronyms (and contractions)
• Security Certifications

Distribute Credentials Securely Using Secrets

This page shows how to securely inject sensitive data, such as passwords and encryption keys, into Pods.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

• Play with Kubernetes

Convert your secret data to a base-64 representation

Suppose you want to have two pieces of secret data: a username my-app and a password 39528$vdg7Jb . First, use a base64 encoding tool to convert your username and password to a base64 representation. Here's an example using the commonly available base64 program:

The output shows that the base-64 representation of your username is bXktYXBw , and the base-64 representation of your password is Mzk1MjgkdmRnN0pi .

Create a Secret

Here is a configuration file you can use to create a Secret that holds your username and password:

Create the Secret

View information about the Secret:

View more detailed information about the Secret:

Create a Secret directly with kubectl

If you want to skip the Base64 encoding step, you can create the same Secret using the kubectl create secret command. For example:

This is more convenient. The detailed approach shown earlier runs through each step explicitly to demonstrate what is happening.

Create a Pod that has access to the secret data through a Volume

Here is a configuration file you can use to create a Pod:

Create the Pod:

Verify that your Pod is running:

Get a shell into the Container that is running in your Pod:

The secret data is exposed to the Container through a Volume mounted under /etc/secret-volume .

In your shell, list the files in the /etc/secret-volume directory:

The output shows two files, one for each piece of secret data:

In your shell, display the contents of the username and password files:

The output is your username and password:

Modify your image or command line so that the program looks for files in the mountPath directory. Each key in the Secret data map becomes a file name in this directory.

Project Secret keys to specific file paths

You can also control the paths within the volume where Secret keys are projected. Use the .spec.volumes[].secret.items field to change the target path of each key:

When you deploy this Pod, the following happens:

• The username key from mysecret is available to the container at the path /etc/foo/my-group/my-username instead of at /etc/foo/username .
• The password key from that Secret object is not projected.

If you list keys explicitly using .spec.volumes[].secret.items , consider the following:

• Only keys specified in items are projected.
• To consume all keys from the Secret, all of them must be listed in the items field.
• All listed keys must exist in the corresponding Secret. Otherwise, the volume is not created.

Set POSIX permissions for Secret keys

You can set the POSIX file access permission bits for a single Secret key. If you don't specify any permissions, 0644 is used by default. You can also set a default POSIX file mode for the entire Secret volume, and you can override per key if needed.

For example, you can specify a default mode like this:

The Secret is mounted on /etc/foo ; all the files created by the secret volume mount have permission 0400 .

Define container environment variables using Secret data

You can consume the data in Secrets as environment variables in your containers.

If a container already consumes a Secret in an environment variable, a Secret update will not be seen by the container unless it is restarted. There are third party solutions for triggering restarts when secrets change.

Define a container environment variable with data from a single Secret

Define an environment variable as a key-value pair in a Secret:

Assign the backend-username value defined in the Secret to the SECRET_USERNAME environment variable in the Pod specification.

In your shell, display the content of SECRET_USERNAME container environment variable.

The output is similar to:

Define container environment variables with data from multiple Secrets

As with the previous example, create the Secrets first.

Define the environment variables in the Pod specification.

In your shell, display the container environment variables.

Configure all key-value pairs in a Secret as container environment variables

Create a Secret containing multiple key-value pairs

Use envFrom to define all of the Secret's data as container environment variables. The key from the Secret becomes the environment variable name in the Pod.

In your shell, display username and password container environment variables.

Example: Provide prod/test credentials to Pods using Secrets

This example illustrates a Pod which consumes a secret containing production credentials and another Pod which consumes a secret with test environment credentials.

Create a secret for prod environment credentials:

Create a secret for test environment credentials.

Special characters such as $ , \ , * , = , and ! will be interpreted by your shell and require escaping.

In most shells, the easiest way to escape the password is to surround it with single quotes ( ' ). For example, if your actual password is S!B\*d$zDsb= , you should execute the command as follows:

You do not need to escape special characters in passwords from files ( --from-file ).

Create the Pod manifests:

Apply all those objects on the API server by running:

Both containers will have the following files present on their filesystems with the values for each container's environment:

You could further simplify the base Pod specification by using two service accounts:

• prod-user with the prod-db-secret
• test-user with the test-db-secret

The Pod specification is shortened to:

What's next

• Learn more about Secrets .
• Learn about Volumes .

Was this page helpful?

Thanks for the feedback. If you have a specific, answerable question about how to use Kubernetes, ask it on Stack Overflow . Open an issue in the GitHub Repository if you want to report a problem or suggest an improvement .