Skip to content

Conversation

@mooreland
Copy link
Contributor

currently pk circuit data quite large and >5G with k23
fixed data(256bit) occupy >80%
permutation data occupy ~15%

this pr just optimize the datas:
fixed data: only store valid bit
permutation data: only store permuted cell

after optimization, pk data shrink to ~200M

meanwhile, added parallel processing
store time reduce to 300ms from 1.3s
fetch time reduced to 2s from 2.3s

@mooreland mooreland requested review from junyu0312 and xgaozoyoe April 2, 2024 03:24
@lanbones
Copy link
Contributor

lanbones commented Apr 2, 2024

I believe the compression of sparse matrix is the most important. I have concern about the zero reduction progress. It involves mont and unmont computing in from_repr()/to_repr(), and lots of memory allocation on vec![]. So my options are:

  1. How much benefit without the zero reduction progress?
  2. Can we use zlib instead of zero reduction progress to avoid from_repr()/to_repr()?
  3. Is it possible to remove these small vec![]?

@mooreland
Copy link
Contributor Author

mooreland commented Apr 3, 2024

  1. How much benefit without the zero reduction progress?
    --actually, the fix zero reduction occupy >80% size in pk, the matrix only occupy ~15%
    we can calculate them, 13 permutation columns and 44 cell involved, for k23, 2^23 *13 *4=436M totally.
    for 17 fix columns, for k23, 2^23 *17 *32=4.5G, actually, most of fix value are 0 or 1(enable), there are quite much meaningless 0. if we keep fix unchanged, the pk size will have 4.5G more. and with zero reduction, it only have ~200M
  1. Can we use zlib instead of zero reduction progress to avoid from_repr()/to_repr()?
    --maybe not so efficient than zere reduction, we can try.
  2. Is it possible to remove these small vec![]?
    -- not so clear, what is means for small vec![]?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants