feat: add Outlook HTTP API validation #1194

PsypherPunk · 2022-10-06T17:41:49Z

check the validity of Outlook/Office 365 email addresses via the method outlined here.
run only via the --outlook-use-api flags (defaulting to false.)

relates #937

core/src/smtp/mod.rs

PsypherPunk · 2022-10-06T17:45:11Z

core/src/smtp/hotmail.rs

@@ -193,4 +251,12 @@ mod tests {
 		let f = f1.try_join(f2).await;
 		assert!(f.is_ok(), "{:?}", f);
 	}
+
+	#[test]


No tests for the check_outlook_api() function—any idea if there's a [email protected] equivalent that could be used here?

amaury1093 · 2022-10-06T22:25:38Z

Hey @PsypherPunk, after some testing, I am a bit reluctant to merge this PR. I tested on some valid Outlook 365 emails, and the results are all incorrect: Reacher returns invalid when they are valid.

If you send me your email, I can send you a couple of addresses to test against. You can find mine in the README.

PsypherPunk · 2022-10-06T22:33:57Z

My email's in the commit message if you want to forward those addresses to test: will be happy to give them a whirl see if there's something I can tweak.

amaury1093 · 2022-10-06T22:59:35Z

Done. Yeah, let's play a bit to see if we can make this work 💪

PsypherPunk · 2022-10-07T13:15:11Z

I've tried this with those sample emails and yep, it fails miserably.

However, I've tried this with another couple of domains—presumably ones where OneDrive use is more ubiquitous—and it's 100% accurate. So, as the article suggests, positive responses are accurate but false-negatives are possible (and seemingly likely.)

Is there any value in doing something like this?

if input.outlook_use_api && host_lowercase.ends_with(".mail.protection.outlook.com.") {
	if let Some(smtp_details) = hotmail::check_outlook_api(to_email, input)
		.await
		.map_err(|err| err.into()) {
		return smtp_details;
	}
}

Where check_outlook_api() would return None in the event of 404, allowing further checks to be carried out…? That way, it's not taking a failed response as an absolute.

hckhanh · 2022-10-10T06:19:39Z

Hi @AmauryM and @PsypherPunk,
Is there any update for this PR? Is there anything that I can help with?

PsypherPunk · 2022-10-10T07:48:42Z

Have you given it a try, @hckhanh? I'd be curious to know how it's working for anyone else.

amaury1093 · 2022-10-10T11:40:56Z

positive responses are accurate but false-negatives are possible

and

Where check_outlook_api() would return None in the event of 404, allowing further checks to be carried out…?

I think this makes sense @PsypherPunk, let's do this. Can you please also add relevant comments detailing the drawbacks of this check_outlook_api approach?

- check the validity of Outlook/Office 365 email addresses via the method outlined [here](https://www.trustedsec.com/blog/achieving-passive-user-enumeration-with-onedrive/). - run only via the `--outlook-use-api` flags (defaulting to `false`.) relates reacherhq#937

use `.mail.protection.outlook.com.` for domains backed by Outlook/Office 365.

if using `--outlook-use-api`, only return immediately in the event of a positive response: negative responses are ambiguous and the process should fall back to subsequent checks.

hckhanh · 2022-10-10T13:49:23Z

Hi @PsypherPunk, I think the result is really good:

{
  "input": "hckhanh@[redacted].com",
  "is_reachable": "safe",
  "misc": {
    "is_disposable": false,
    "is_role_account": false,
    "gravatar_url": null
  },
  "mx": {
    "accepts_mail": true,
    "records": [
      "[redacted]-com.mail.protection.outlook.com."
    ]
  },
  "smtp": {
    "can_connect_smtp": true,
    "has_full_inbox": false,
    "is_catch_all": false,
    "is_deliverable": true,
    "is_disabled": false
  },
  "syntax": {
    "address": "hckhanh@[redacted].com",
    "domain": "[redacted].com",
    "is_valid_syntax": true,
    "username": "hckhanh"
  }
}

Even the result is much more better than the previous one (it was invalid)

hckhanh · 2022-10-10T14:13:07Z

I have another question, @PsypherPunk
Is there any support for iCloud through API?

PsypherPunk · 2022-10-10T14:16:57Z

Thanks for the output, @hckhanh.

Re. iCloud, I'm afraid I've no idea. The changes here were raised as part of #937 which are specific to Google and Outlook. There might be something similar for iCloud, but I'm not aware of it.

amaury1093

Code LGTM! I have one final comment about the public-facing API. Using outlook_ may seem like this method would work for @outlook.com emails too, which is incorrect.

Would like to here opinions about:

microsoft365_
office365_

I'm not too well-versed in the MS ecosystem to know what terms people usually use.

cli/src/main.rs

PsypherPunk · 2022-10-10T15:02:22Z

Good shout, @AmauryM: Wikipedia says Microsoft 365: "Microsoft 365, formerly Office 365…" so I'll go with that…?

update references to "Microsoft 365" to make is more explicit that this pertains to the underlying services, not Outlook addresses.

amaury1093 · 2022-10-10T20:45:27Z

core/src/smtp/hotmail.rs

+pub async fn check_microsoft365_api(
+	to_email: &EmailAddress,
+	input: &CheckEmailInput,
+) -> Result<Option<SmtpDetails>, HotmailError> {


I just tried this manually, and I believe there's a bug: if you run cargo run -- --microsoft365-use-api=true <email> with someone that hasn't opened their one drive, it returns a Err(ReqwestError). Whereas a Ok(None) is expected, right? I believe in #1194 (comment) we both meant that in that case the algorithm should just continue as normal, which is currently not the case.

I'm inclined to simply do this:

Suggested change

) -> Result<Option<SmtpDetails>, HotmailError> {

) -> Result<SmtpDetails, HotmailError> {

and below do:

if input.microsoft365_use_api && host_lowercase.ends_with(".mail.protection.outlook.com.") { // Continue in the event of an ambiguous result. if let Ok(smtp_details) = hotmail::check_microsoft365_api(to_email, input).await { return Ok(smtp_details); } }

I.e. continue the normal SMTP logic if any error occurs in this API. Do you have better ideas?

Ah, I think I know what's happening:

if the API returns a 403, it returns a positive response (Ok(Some(smtp_details)).)

if the API returns a 404, it returns a negative response (Ok(None).)

if the API fails because the OneDrive domain doesn't exist (i.e. the company in question just doesn't use OneDrive), it raises an error (Err(HotmailError(ReqwestError)).)

if the API fails for some other reason (e.g. a transient network issue), it raises an error (Err(HotmailError(ReqwestError)).)

The problem is that those last two are indistinguishable, although 3. should result in some indication that it's ambiguous, like 1. and 2.

However, I can't of a way to make that distinction so I think you're right: I'll aim to allow it continue in the event of any error (but might add an additional log which might allow things to be narrowed down in the future.)

@AmauryM, I've pushed up a change which resolves the issues (I think; I've tested the scenarios I'm aware of and they either pass or continue to SMTP checks) but leaves the Result<Option<SmtpDetails>, HotmailError> signature in place.

In order to have check_microsoft365_api() return a Result<SmtpDetails, HotmailError>, I'd have to change the "successful" 404 response into a HotmailError. That's easy enough but adding another HotmailError type (e.g. HotmailError::AmbiguousResponse) didn't seem quite right.

Basically:

if the HTTP request succeeds and we see a 403, we return the result.

if the HTTP request succeeds and we see a 404, we continue to SMTP checks.

if the HTTP request fails, we log the error and continue to SMTP checks.

allow both failures in the HTTP request and 404 responses to continue.

amaury1093

This LGTM, thanks @PsypherPunk!

hckhanh · 2022-10-12T01:32:32Z

Hi @AmauryM, @PsypherPunk
I try to switch to master branch to check this feature, but I get this bug when I build the project

error[E0433]: failed to resolve: use of undeclared crate or module `hotmail`
  --> C:\Users\hoang\.cargo\git\checkouts\check-if-email-exists-c4445d2e2cd9a1bc\5d3c49f\core\src\smtp\mod.rs:73:9
   |
73 |         match hotmail::check_microsoft365_api(to_email, input).await {
   |               ^^^^^^^ use of undeclared crate or module `hotmail`

For more information about this error, try `rustc --explain E0433`.
error: could not compile `check-if-email-exists` due to previous error
warning: build failed, waiting for other jobs to finish...
Internal Error: Command failed: cargo build

* feat: add Outlook HTTP API validation - check the validity of Outlook/Office 365 email addresses via the method outlined [here](https://www.trustedsec.com/blog/achieving-passive-user-enumeration-with-onedrive/). - run only via the `--outlook-use-api` flags (defaulting to `false`.) relates reacherhq#937 * fix: restrict Office 365 domain use `.mail.protection.outlook.com.` for domains backed by Outlook/Office 365. * fix: continue for non-definitive responses from Outlook API if using `--outlook-use-api`, only return immediately in the event of a positive response: negative responses are ambiguous and the process should fall back to subsequent checks. * fix: amend Outlook references update references to "Microsoft 365" to make is more explicit that this pertains to the underlying services, not Outlook addresses. * fix: continue in the event of a ReqwestError allow both failures in the HTTP request and 404 responses to continue. Co-authored-by: Amaury <[email protected]>

PsypherPunk commented Oct 6, 2022

View reviewed changes

core/src/smtp/mod.rs Outdated Show resolved Hide resolved

PsypherPunk commented Oct 6, 2022

View reviewed changes

PsypherPunk force-pushed the 937-outlook-validation branch 2 times, most recently from cebf38b to 0373295 Compare October 6, 2022 18:08

This was referenced Oct 6, 2022

Microsoft teams email address is classed as catch all, even though it isn't #1185

Open

Additional Validation methods - Gmail/Google Apps/Office 365 #937

Open

PsypherPunk added 3 commits October 10, 2022 14:12

fix: restrict Office 365 domain

1d28ad1

use `.mail.protection.outlook.com.` for domains backed by Outlook/Office 365.

fix: continue for non-definitive responses from Outlook API

16430dc

if using `--outlook-use-api`, only return immediately in the event of a positive response: negative responses are ambiguous and the process should fall back to subsequent checks.

PsypherPunk force-pushed the 937-outlook-validation branch from bb11e5e to 16430dc Compare October 10, 2022 13:50

amaury1093 reviewed Oct 10, 2022

View reviewed changes

cli/src/main.rs Outdated Show resolved Hide resolved

cli/src/main.rs Outdated Show resolved Hide resolved

fix: amend Outlook references

ccbb9a6

update references to "Microsoft 365" to make is more explicit that this pertains to the underlying services, not Outlook addresses.

PsypherPunk force-pushed the 937-outlook-validation branch from 661f867 to ccbb9a6 Compare October 10, 2022 15:12

amaury1093 reviewed Oct 10, 2022

View reviewed changes

fix: continue in the event of a ReqwestError

4f4570a

allow both failures in the HTTP request and 404 responses to continue.

PsypherPunk force-pushed the 937-outlook-validation branch from 31a2dff to 4f4570a Compare October 11, 2022 14:14

amaury1093 approved these changes Oct 11, 2022

View reviewed changes

Merge branch 'master' into 937-outlook-validation

f3d2868

amaury1093 enabled auto-merge (squash) October 11, 2022 15:58

amaury1093 merged commit 5d3c49f into reacherhq:master Oct 11, 2022

PsypherPunk deleted the 937-outlook-validation branch October 11, 2022 17:23

hckhanh mentioned this pull request Oct 12, 2022

Unused Headless feature for Microsoft 365 api #1203

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Outlook HTTP API validation #1194

feat: add Outlook HTTP API validation #1194

PsypherPunk commented Oct 6, 2022

PsypherPunk Oct 6, 2022

amaury1093 commented Oct 6, 2022 •

edited

Loading

PsypherPunk commented Oct 6, 2022

amaury1093 commented Oct 6, 2022

PsypherPunk commented Oct 7, 2022

hckhanh commented Oct 10, 2022 •

edited

Loading

PsypherPunk commented Oct 10, 2022

amaury1093 commented Oct 10, 2022

hckhanh commented Oct 10, 2022 •

edited

Loading

hckhanh commented Oct 10, 2022

PsypherPunk commented Oct 10, 2022

amaury1093 left a comment

PsypherPunk commented Oct 10, 2022

amaury1093 Oct 10, 2022 •

edited

Loading

amaury1093 Oct 10, 2022 •

edited

Loading

PsypherPunk Oct 11, 2022

PsypherPunk Oct 11, 2022 •

edited

Loading

amaury1093 left a comment

hckhanh commented Oct 12, 2022 •

edited

Loading

	) -> Result<Option<SmtpDetails>, HotmailError> {
	) -> Result<SmtpDetails, HotmailError> {

feat: add Outlook HTTP API validation #1194

feat: add Outlook HTTP API validation #1194

Conversation

PsypherPunk commented Oct 6, 2022

PsypherPunk Oct 6, 2022

Choose a reason for hiding this comment

amaury1093 commented Oct 6, 2022 • edited Loading

PsypherPunk commented Oct 6, 2022

amaury1093 commented Oct 6, 2022

PsypherPunk commented Oct 7, 2022

hckhanh commented Oct 10, 2022 • edited Loading

PsypherPunk commented Oct 10, 2022

amaury1093 commented Oct 10, 2022

hckhanh commented Oct 10, 2022 • edited Loading

hckhanh commented Oct 10, 2022

PsypherPunk commented Oct 10, 2022

amaury1093 left a comment

Choose a reason for hiding this comment

PsypherPunk commented Oct 10, 2022

amaury1093 Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

amaury1093 Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

PsypherPunk Oct 11, 2022

Choose a reason for hiding this comment

PsypherPunk Oct 11, 2022 • edited Loading

Choose a reason for hiding this comment

amaury1093 left a comment

Choose a reason for hiding this comment

hckhanh commented Oct 12, 2022 • edited Loading

amaury1093 commented Oct 6, 2022 •

edited

Loading

hckhanh commented Oct 10, 2022 •

edited

Loading

hckhanh commented Oct 10, 2022 •

edited

Loading

amaury1093 Oct 10, 2022 •

edited

Loading

amaury1093 Oct 10, 2022 •

edited

Loading

PsypherPunk Oct 11, 2022 •

edited

Loading

hckhanh commented Oct 12, 2022 •

edited

Loading