Thank you very much for your code. Do you have the code for how to count the memory in Table 1? Thank you very much

[LiZhangMing]

[KaiLv69]

Hi, we modify the code here(https://gist.github.com/dojoteef/26cd46f7cc38b38e6f443c5f62411aa3). Other profile tools will also work well.

To get the memory usage of activation, we track the memory usage in the forward process, and sub the memory usage after before forward as the activation.