What to do when you find a malicious mail filter in Google Workspace
This guide details what to do when you find a malicious mail filter in Google Workspace. There are 4 steps:
Limit the damage: Take some quick steps to minimize damage.
Understand the root cause: How was this account compromised?
Check if other users are compromised: Is this account the only one affected?
Recovery: Once we understand how the attack happened, and how widespread it is, we can comprehensively rebuild and clean up.
Limit the damage
First, we want to take some quick actions to limit the impact of the compromise.
1. Delete the filter: Delete the malicious filter while you investigate. To do this, you have two options:
Use the Push platform: Select the malicious mail rule on the mail rules page in the Push admin console. In the top right, select Actions and then Delete this rule.
Delete in Gmail: View the mail rules in the victim's mailbox. Select Delete for the target filter.
2. Reset the user's password: although we don’t yet know if the user's password is compromised, it is a quick and safe precaution. Follow these instructions to reset the victim's password.
Find the root cause
Follow these to inspect relevant logs and data to pinpoint how the account was compromised:
Inspect login events
Look for phishing emails
Inspect OAuth apps
Analyze the victim's machine
Login events: Go to the Audit Log Login screen inside Google Workspace Admin and filter for the victim and the relevant time period.
Can you see a suspicious login? Specifically look at the location (you can use something like this to look up the approximate location of an IP address) and the login time. Take into account the victim's typical location and working hours and look for events that fall outside these norms.
Look at login attempts for the user prior to rule creation: Are there numerous failed logins, culminating in a successful login? This suggests a password brute force.
Alternatively, look more broadly at failed logins across all users: Are there numerous failed logins across accounts, culminating in a successful login for this victim? This suggests a password spray.
Email logs: If you haven’t found a suspicious login or app, the user may have received a phishing email that coerced them to disclose their credentials.
Inspect the email in the victim's mailbox leading up to the time of rule creation, looking back for at least a week. Although some attacks are completely automated, meaning credentials get used immediately after capturing them, often attackers have a credential harvesting phase before switching to a utilization phase. That means there could be a delay between credential compromise and usage.
Look for any email with links or buttons, particularly ones referencing Google Workspace, document sharing, or anything generally encouraging a login. Inspect the sender email address and the destination URLs of suspicious mails (don't click the links or visit the URLs, simply copy them to a notepad for manual inspection). Do they line up with what you’d expect if it was legitimate?
Compromised credentials: The victim's password may have been compromised elsewhere, rather than guessed. Since users commonly reuse passwords, a password compromise elsewhere can give an attacker direct access to many other accounts. haveibeenpwned.com is a site that tracks password breaches and will tell you which breaches your username has been part of. Don't panic if the target user is present in past breaches. Several very large sites have been compromised in the past, so this is actually quite common. However, it may indicate a possible root cause, especially if the breach was relatively recent.
OAuth apps: Select the user from the Google Admin console. Select Security, and then, at the bottom, select Connected Applications. Inspect the installed applications for anything unexpected, particularly with an authorization date around the time of rule creation.
If you see an application that looks suspicious, click its name, and inspect the permissions listed and their descriptions to determine if the app has requested suspicious permissions. For example, if the app created malicious mail rules, you might see the gmail.settings.basic permission.
If you find a suspicious app, this suggests consent phishing was used to compromise the victim.
Analyze the victim machine: It's possible that malware on the victim machine added the mail rule. There are several ways this could have happened:
Phishing mail with malicious attachment: Follow the “Phishing” steps above to investigate this path.
Malicious file download: The victim may have downloaded a file that was malicious. Inspect the victim’s downloads folder for anything suspicious around the time of compromise (remember not to execute or interact with the suspected file). Alternatively, if you have reliable network connection logs, you may be able to search for user web activity around the time of compromise to determine if this is the cause.
Out-of-date software: Software like browsers or PDF readers are common sources of bugs that are actively exploited. Inspect the software installed on the victim machine to see if it is significantly out of date. Unusual software crashes could be an indication of an exploit. It will be difficult to prove this is the cause, however, so be sure to exhaust all other possible root causes before drawing this conclusion.
Still not sure?
If you still haven't determined the cause of the account compromise, but you are confident a compromise has taken place, you should consider calling an incident response company. Here is a list of vendors approved by NCSC.
If that is not an option for you, jump to step 4 and complete all steps.
Check if other users are compromised
Once you’ve found the root cause, you need to determine if other users were affected by the incident. Although it is likely the attacker followed similar steps for each victim (e.g. creation of mail rules), this might not be the case. Choose the section relevant to your root cause for steps of how to investigate this.
Password guessing or spraying
If you have determined password guessing or spraying as the most likely cause, we need to try to understand how it happened, and if others in your organization may have been affected. To do this, we will take another look at sign-in logs.
Go back to the Audit Log Login screen inside Google Workspace Admin and look for failed attempts leading up to the one you previously identified as suspicious. Look at other failed logins around that time. Look at other attempts from the attacker IP. Is the attacker focusing on just the victim, or a range of users? Can you see if the attacker was successful for other accounts? If so, add those to your victim list.
If you've found the phishing mail that compromised the victim, you need to see if anyone else received the mail, and whether they also fell for the phish:
Run a message trace to look for all users who received mail from the suspicious address.
If you have reliable network logging for your users, search for any users who visited the domain of the link in the suspicious mail.
The safe and quick option here is to add all recipients to your victims list. Although you may want to avoid the disruption of password resets for users that potentially did not fall for the phish, it is safer to assume that all users were affected because users may be embarrassed to say they fell for a phishing email or are often unaware.
If you determined consent phishing is most likely, we need to check if other users installed the malicious app.
Navigate to the API Controls section of Google Workspace Admin. Select the malicious app and take note of the value under the Users column. If the count is more than the original victim count, we need to figure out who else installed the app.
Unfortunately, there is no prepared report for this in Google Workspace so we need to inspect the Token report in the Audit Log to see which users have authorized the app. Add a filter for the app name and add any other users to your victims list.
If you determined the user downloaded or opened a malicious file, inspect network logs, if available, for other users downloading a similar file.
Alternatively, inspect AV logs for other endpoints that have seen the same malicious file. Add any results to your victims list.
Now we have a clear picture of the root cause and wider impact, we can focus on recovery.
For each victim:
Delete the mail rule: View the mail filters in the victim's mailbox. Select Delete to remove the filter.
Reset their password: Follow these instructions to reset the victim's password
Remove the malicious OAuth app (if applicable): For each user, select the user from the Admin console, select Security, and then, at the bottom, select Connected Applications. Hover over the malicious application and click the trash icon to remove.
Rebuild the victim’s machine (if applicable): If you suspect the victim was compromised via malware, it is almost always quicker, easier and safer to just rebuild the machine.