Large Language Models (LLMs) such as ChatGPT, Claude, and Gemini are increasingly being used for more complex tasks. One particularly promising application is cybersecurity: LLMs could help identify security vulnerabilities in software more quickly and support developers and security professionals in protecting their systems. This is especially relevant for web applications, which form a major part of today’s digital infrastructure and often process sensitive information.  In our research, we investigate how reliably modern LLMs can detect security vulnerabilities in real-world web application code. We evaluated six current models, including commercial frontier models, as well as open-weight models that can be run locally. Our experiments used four different WordPress plugins containing known vulnerabilities in older plugin versions. Additionally, we evaluated five different prompts with different complexity.  Our results show that all tested models are capable of identifying security issues. However, their performance varies considerably depending on both the model and the chosen prompt. More focused and clearly defined prompts generally led to better results than open-ended instructions. Among the evaluated models, Claude Opus 4.6 achieved the highest vulnerability detection rate at ~63%. Notably, the open-weight model MiniMax M2.5 performed on par with Gemini 3.1 Pro and Codex GPT-5.4, achieving a detection rate of ~48% and demonstrating competitive performance compared with leading proprietary models. At the same time, the models did not always produce consistent results when the same analysis was repeated. We further noticed a high number of false-positive vulnerability reports.   The underlying study was published as a workshop paper at the international ARES 2026 conference. To support future research, we have also made the code and experimental data publicly available.