Today, malware developers use new methods to circumvent security solutions at the endpoint. Hooking system functions at the kernel level is difficult and impossible due to the presence of patchguard. Today, malware developers generally use four methods to avoid detection by hooks. Unhooking and patching of hooked functions have a similar approach in implementation. This research tries to identify these features by using periodic calculation or recursive function and corresponding to the initial value. If the memory byte value is changed and the first completeness is changed, it will generate a warning. In order to detect a direct call, first the syscall instruction must be specified in the address space of the processor. If the syscall instruction is called inside the process code space, the necessary warning should be generated. Detecting an indirect call attack requires call stack recovery. If there is a discrepancy between the call stack addresses and the return address from the kernel side, this action should be considered malicious. In order to evaluate, Sandbox or Matos is compared with several commercial products including Sophos endpoint solution, Crowdstrike, Ost, Trend Micro. In this research, the number of 40 executable files separated by 20 user-level rootkits and 20 healthy software are considered for testing. Finally, by examining the percentage of true positives and false negatives of the method implemented in this research, it is proved that the accuracy results of Sophos product are 0.63, Crowdstrike 0.70, Ost 0.58, Trend Micro 0.58. Matos was able to correctly identify 17 user-level rootkits out of 20 rootkits. In the continuation of this comparison, the accuracy of the sandbox in detecting malware is 0.93, which is improved by 0.32 with the highest detection accuracy, i.e. Crowd Strike.