juice-shop VS ZAP

Hi HN, I’m the founder of Corgea (https://corgea.com). We help companies fix their vulnerable source code using AI.

Originally, we started with a data security product that would detect data leaks at companies. Despite initial successes and customer acquisitions, we frequently heard that highlighting issues wasn't enough; customers wanted proactive fixes. They had hundreds (yes hundreds!) of security tools alerting them about vulnerabilities, but couldn’t afford a dedicated team to go through them all and fix them. One prospect we spoke to had tens of thousands of reported vulnerabilities in their SAST tool. With the rise of AI code generation, we saw an opportunity to give customers what they really wanted.

Having Corgea is like having a security engineer on staff focused on making your code more secure. We want security to be an enabler of engineering rather than a blocker to it, and the reverse to be true. To accomplish this, we built it on top of existing LLMs to issue code fixes.

To show Corgea’s capabilities, we took some popular vulnerable-by-design applications like Juice Shop (https://github.com/juice-shop/juice-shop), scanned them and issued fixes for their vulnerabilities. You can see some of them here: https://demo.corgea.com. Some examples of vulnerabilities it solves are like SQL injection, Path Traversal and XSS.

What makes this tough is that currently LLMs struggle at generalist coding tasks because it has to understand your whole code base, the domain you’re in, and the user’s request to do something. This can lead to a lot of unintended behavior where it codes things incorrectly because it’s giving a best guess at what you want. Adam, one of the founding engineers on the team coined it well: LLMs don’t reason, they fuzz.

We made several decisions that helped the LLM become more deterministic. First, what we’re doing is extremely domain specific: vulnerable code fixes in a limited number of programming languages. There are roughly 900 security vulnerabilities in code, called CWE’s (https://cwe.mitre.org/), that we’ve built into Corgea. An SQL injection vulnerability in a Javascript app is the same regardless if you’re a payments company or a travel booking website. Second, we have no user generated input going into the LLM, because SAST scanners everything needed to issue a fix. This makes it much more predictable and reproducible for us and customers. We can also create robust QA processes and checks.

To illustrate the point, let’s put some of this to the test using some napkin math. Assume you’re serving 5,000 enterprises that ship on average 300 domain specific features a year in 5 different programming languages that each require 30 lines of code changes across multiple files. You’ll have about 300m permutations the product needs to support. What a nightmare!

Using the same napkin math, Corgea needs to support the ~900 vulnerabilities (CWE’s). Most of them require 1 - 2 line changes. It doesn’t need to understand the whole codebase since the problem is usually isolated to a few lines. We want to support the 5 most popular programming languages. If we have 5,000 customers, we have to support ~4,500 permutations (900 issues x 5 different languages). This leads to a massive difference in accuracy. Obviously, this is an oversimplification of the whole thing but it illustrates the point.

What makes this different from Copilot and other code-gen tools is that they do not specialize in security and we’ve seen them inadvertently introduce security issues unbeknownst to the engineer. Additionally, they do not integrate into existing scanning tools that companies are using to resolve those issues. So unless a developer is working on every part of the product, they’re unable to clear security backlogs, which can be in the thousands of tickets.

As for security scanners, the current market is flooded with tools that report and overwhelm security teams and are not effective at fixing what they’re reporting. Most vulnerability scanners do not remediate issues, and if they do they’re mostly limited to upgrading packages from one version to another to reduce a CVSS. If they do offer CWE remediation capabilities their success rates are very low because they’re often based on traditional AI methodologies. Additionally, they do not integrate with each other because they want to only serve their own findings. Enterprises use multiple tools like Snyk, Semgrep, Checkmarx, but also have a penetration testing program, and a bug bounty program. They need a solution that consolidates across their existing tools. They also use Github, Gitlab and Bitbucket for their code repository.

We’re offering a free tier for smaller teams and priced tiers. We believe we can reduce 80% of the engineering effort for security fixes, which would equate to at least $10m a year for enterprises.

We’re really excited to share this with you all and we’d love any thoughts, feedback, and comments!

Just grab the source code and modify it. Read through the code. Make the shop sell new things. Why start from scratch on such a broad and complex topic?

If you want to explore vulnerabilities and bad practices, just look at OWASP Juice Shop the most insecure app out there

For a solid example of a vulnerable and insecure application see OWASP Juice Shop - this is an example of what NOT to do.

This is awesome! Convenient for folks who use the Express/Angular stack but conceptual stuff should be pretty universal regardless.

Wasn't aware of this project at all but found the following links useful for context:

The actual Juice Shop website can be found at https://juice-shop.herokuapp.com/#/

and the github link for viewing code is https://github.com/juice-shop/juice-shop/releases/

Right here. https://github.com/juice-shop/juice-shop/blob/master/Dockerfile

Exposed Prometheus metrics Endpoint

https://github.com/juice-shop/juice-shop#official-companion-...

If the versions installed in the book are outdated, you too can bump the version strings in the dependency specs in the git repo and send a PR Pull Request (which also updates the Screenshots and Menu > Sequences and Keyboard Shortcuts in the book&docs) and then manually test that everything works with the updated "deps" dependencies.

If it's an executablebooks/, a Computational Notebook (possibly in a Literate Computing style), you can "Restart & Run all" from the notebook UI button or a script, and then test that all automated test assertions pass, and then "diff" (visually compare), and then just manually read through the textual descriptions of commands to enter (because people who buy a Book presumably have a reasonable expectation that if they copy the commands from the book to a script by hand to learn them, the commands as written should run; it should work like the day you bought it for a projected term of many free word-of-mouth years.

From https://github.com/juice-shop/juice-shop#docker-container :

  docker pull bkimminich/juice-shop

Take a look at juice-shop as well.

I use ZAP [1] with the OAST add-on for this at the moment. I admit the UX isn't perfect, but it serves my purpose.

If I also want control over the responses (e.g. return a 401 status code for every fifth request), I have a custom extender script [2] for that.

[1]: https://www.zaproxy.org/

Implement tools like Burp Suite or OWASP ZAP for in-depth security scanning of your APIs.

OWASP ZAP

The use of capital punctuation implies a warning? an alert? Would this same response be warranted for Burp which is also a commercial, closed source product?

If this is an issue for some, then ZAP being open source[1] maybe favourable.

That said, Burp is the defacto tool for a reason - it's best in class. Every pentester I know, including myself, has a paid subscription. The fact that it's closed source hasn't been an issue.

[1] https://github.com/zaproxy/zaproxy

Briefly reviewed your product. Seems like OWASP ZAP is your competition: https://www.zaproxy.org/

It runs entirely in the browser so it uses the browser "native" frameworks.

Dynamic analysis involves testing your application while it's running. Tools like OWASP ZAP and Burp Suite can help identify vulnerabilities like SQL injection or Cross-Site Scripting by sending malicious requests to your application and analyzing the responses.

> Lying is not an embellishment or puffery, it's a lie. Engaging a company for a 3 day pen test that's totally insufficient, that would be an embellishment.

I agree, but if the RFP question was phrased "have you done penetration testing?" then that leaves a lot of room for embellishment. If the question is "do you have SOC2 certification?" and you answer "yes" untruthfully, then that is a lie. If they ask for the SOC2 or pentest report and you give them a falsified document, that's where you're (probably) committing fraud.

> One of the most important part of pen tests is that they are external.

AWS/Google/etc have internal security teams doing their pen tests, so no, this isn't true.

> Just doing your job as an engineer and looking for bugs is not a pen test.

What about an engineer spending an afternoon running ZAP[0]?

> It's like saying, "what is an audit really? We have accountants and they check our books for anomalies."

Yeah, which is why you don't just ask a company "do you keep track of your finances?" if you're investing in them, you request external auditors.

[0] https://www.zaproxy.org/

In addition to manual security reviews, you can also implement DevSecOps practices to automate security checks. For example, you can set up a CI/CD pipeline to run static code analysis tools like CodeQL and automatically run penetration tests using tools like OWASP ZAP.

OWASP ZAP (open source)

I would start by installing Burp Suite or OWASP Zap and seeing what the actual messages look like