Windows operating system, as the most widely used operating system of desktop computers, is still one of the main targets of malware writers. For this reason, many researches have recently been conducted to detect Windows malware. Due to the emergence and application of deep learning, although researchers have been able to use it to detect Windows malware, but there are still various challenges such as the detection of new and zero-day malwares and lack of evolution of processes of the feature engineering that increase the false positive rate. Currently, deep learning based malware detection approaches are either two class or multi classes, which fail to detect anomaly and zero-day malware. in this research, in addition to using a combination of various features of static and dynamic including file, registry, network, calls and PE import names, we also have increased the number and variety of normal datasets using the conditional tabular generative adversarial model for more accurate training, then we made it possible to detect anomalies and zero-day malware by presenting the deep approach of one-class generative adversarial network model. The result of the research includes a false alarm rate of approximately 1% with a high detection rate of 99% that compared to similar methods, indicates the success of the proposed method.
)