I have created an MCAS Activity policy to detect O365 downloads from external IPs and generate an alert. When applying policies filters I can see events from activity log, but alerts are NOT generated.
You may already know this, but let's clarify a few things for anyone reading this post.
First, activity policies only work on a go-forward basis. Past activities are not evaluated for policy match. Instead, use the activity log investigation blade, to view similar activities from the past.
Second, once you have your activity policy in place, use the activity log view with the 'matched policy' filter to see any activities that get stamped as matching your policy. That can help determine if it is a policy not getting applied. If no activities match the policy filter, you should also check to see if the download test actions have appeared in the activity log yet, at all. MDCA cannot evaluated any activities that are not in that log.
Finally, there have been occasional delays in activity policy processing with MDCA. If you think your policy should be triggering but just isn't despite you seeing, you may need to open a support request.