I found that sAvailableMemory kernel global variable is used to track memory available for allocation, allocation failure is caused here. sAvailableMemory variable is available from userland by get_system_info function and system_info.free_memory field. sAvailableMemory initial value is set here and it has the same value as system_info.max_pages * B_PAGE_SIZE (it is set here).
system_info.used_pages value seems to be inaccurate, there are TODOs (1, 2) about it. Calculation based on system_info.used_pages and on system_info.free_memory may produce significantly different results. Most memory inspection tools (ProcessController, listarea, ActivityMonitor, Slayer) are based on system_info.used_pages and produce wrong results.
I fixed free memory calculation in my SystemManager and now it display correct values:
