Cutting false alarms in memory-based malware detection without losing real threats.
Modern security teams rely on behavioural Endpoint Detection and Response to catch what signatures miss. It works and it drowns them. Industry surveys put false positives as the top detection challenge for over 70% of security teams, and when a Security Operations Centre spends its day dismissing benign administrative software flagged as malicious, two things happen: real incidents wait longer, and analysts stop trusting the automation. At the same time the threat landscape has moved into volatile memory. Fileless malware executes entirely in RAM and leaves nothing on disk, so the API and process logs these systems depend on are steadily losing their grip.
MemSift asks whether a fix proposed for the first problem still works when you move it into the second. A 2018 study cut behavioural false alarms by 97% using a deep-learning "second opinion" classifier but it ran on proprietary vendor telemetry nobody can reproduce, and on sparse binary API features where benign and malicious behaviour overlap heavily. The hypothesis here is that memory forensics changes the geometry of the problem: artefacts like injected memory regions and hidden processes are far more distinctive than API calls, so the feature space should be measurably more separable and if it is, the false-positive problem becomes a question of decision policy rather than model capacity.
The pipeline processes the CIC-MalMem-2022 benchmark (58,596 memory dumps, 55 numeric features extracted via VolMemLyzer) with scaling fitted strictly on the training split, and trains a feed-forward ANN in PyTorch against an XGBoost baseline on an identical held-out test set. Three interventions are then compared head-to-head on the same split: post-training threshold tuning, selected on validation and applied once to test; training-time reweighting via a focal-loss γ sweep and class weighting; and a nine-family feature ablation that removes each memory feature group in turn to isolate which artefacts actually drive false alarms. Every experiment is repeated across multiple random seeds and reported as mean ± standard deviation, so that effects can be separated from run-to-run noise.
The results argue for restraint. Threshold tuning alone removes false positives at no cost to detection, while heavier training-time interventions trade one error type for the other higher γ suppresses false alarms but misses more malware. XGBoost matches the neural network, which is not evidence the ANN is weak but confirmation that the memory feature space is close to linearly separable; on harder, less separable data the deep pipeline is expected to hold its advantage. Two methodology corrections a feature-scaling leakage fix and a validation-selected threshold protocol are documented as contributions rather than footnotes, because in a domain reporting 99%+ accuracy as standard, how a number was produced matters more than the number. The work forms the basis of my MSc dissertation; code and findings are to be open-sourced and submitted to a peer-reviewed venue.