ZFS Memory

ZFS frees up its cache in a way that does not cause a memory shortage. The system can operate with lower freemem without suffering a performance penalty. ZFS returns memory from the ARC only when there is a memory pressure. However, there are occasions when ZFS fails to evict memory from the ARC quickly which can lead to application startup failure due to a memory shortage or for example to less free memory for kernel-zones. Also, reaping memory from the ARC can trigger high system utilization at the expense of performance. You could limit the memory usage using "zfs_arc_max" or "user_reserve_hint_pct", please see MOS DOC-ID 1005367.1 for more details. But anyhow, limiting does not mean you have enough free like mentioned before. There is a small but nice hook you could use:

root@solaris:~# echo "::memstat" | mdb -k
Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                               10583129    80.7g  6.6%
  Regular Kernel                      8800037    67.1g        5.5%/83.1%
  Defdump prealloc                    1783092    13.6g        1.1%/16.8%
ZFS                                  25064230   191.2g 15.7%  <---- high usage
User/Anon                            87434765   667.0g 54.9%
  Regular User/Anon                  10150413    77.4g        6.3%/11.6%
  OSM                                77284352   589.6g       48.6%/88.3%
Exec and libs                          284069     2.1g  0.1%
Page Cache                            5677442    43.3g  3.5%
Free (cachelist)                       311034     2.3g  0.1%
Free                                 29635667   226.1g 18.6%
Total                               158990336     1.1t  100%
root@solaris:~# echo "needfree/Z 0x40000000"|mdb -kw ; sleep 1 ; echo "needfree/Z 0"|mdb -kw
needfree:       0                       =       0x40000000
needfree:       0x40000000              =       0x0
root@solaris:~# echo "::memstat" | mdb -k
Usage Type/Subtype                      Pages    Bytes  %Tot  %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
Kernel                               10585952    80.7g  6.6%
  Regular Kernel                      8802860    67.1g        5.5%/83.1%
  Defdump prealloc                    1783092    13.6g        1.1%/16.8%
ZFS                                   2976204    22.7g  1.8%  <---- it's gone
User/Anon                            87441852   667.1g 54.9%
  Regular User/Anon                  10157500    77.4g        6.3%/11.6%
  OSM                                77284352   589.6g       48.6%/88.3%
Exec and libs                          284067     2.1g  0.1%
Page Cache                            5676347    43.3g  3.5%
Free (cachelist)                       312849     2.3g  0.1%
Free                                 51713065   394.5g 32.5%  <---- free again
Total                               158990336     1.1t  100%
root@solaris:~#

BTW; it takes some seconds to shrink the ZFS usage... in my output above it took maybe 5 seconds on a M7-8 server running Solaris 11.4.28.82.3