Polynomial Code for ISA-L erasure code - optional faster encode - decode unchanged

[jeffareid]

The intended ISA-L use case is as an erasure only code. Polynomial code could do a somewhat faster single thread full encode using LFSR, but would require separate software for LFSR based encoding. Multi-threading would be a simpler way to increase throughput, I don't have any plans to create a repository for LFSR based encoding unless there is some interest in this.

For erasure codes like ISA-L, error checking | correction would be normally be handled externally, such as the ZFS file system. If error checking | correction was to be added to erasure code, a polynomial code based matrix can be used for encoding.

[jeffareid]

Using the self-reciprocal polynomials represented by the matrices generated by gf_gen_rcp_matrix() in ec_rcp.c, I did a proof of concept and got the following results for a LFSR based encode. There are no inversion issues with the polynomial based matrices.

I ran the code on a Dell 5560 laptop - Intel Core i7-11850H - AVX512+GFNI, which isn't consistent, so I used the numbers from best runs. I choose m=20 since that is what BackBlaze uses. I tested number of parities p from 1 to 6.

Initially I tested 512·1024 columns, which fits in the common 24 MB L3 cache, but was running into a 51.2 GB/sec main memory bandwidth issue. Other stuff on Windows 11 uses L3 cache interfering with benchmarking. I then tested 48·1024 columns, which fits in each core's 1.25 MB cache. ISA and LFSR both keep pointers to destination rows in registers. For source rows, ISA loads an 8 byte pointer to load 64 bytes at a time, while LFSR loads 128 bytes at a time using a pair of ZMM registers, For p=1, LFSR operates on 256 bytes at a time and just uses XOR. For p=2 to 6, only p/2 unique multipliers are needed, so they are only loaded one time into ZMM registers. p=6, 3 multipliers. p=5|4, 2 multipliers, p=3|2 1 multiplier, p=1 XOR. These are the results for single thread 48·1024 columns per row:

               lfsr          isa        increase
k=14 p=6 m=20   93.6 GB/sec   56.1 GB/sec  66.8% 
k=15 p=5 m=20  101.3 GB/sec   62.0 GB/sec  63.4%
k=16 p=4 m=20  109.4 GB/sec   68.0 GB/sec  60.9%
k=17 p=3 m=20  140.4 GB/sec   81.6 GB/sec  72.0%
k=18 p=2 m=20  163.7 GB/sec   82.7 GB/sec  97.9%
k=19 p=1 m=20  182.0 GB/sec  112.8 GB/sec  61.3%

A simpler alternative to using LFSR encoding to increase performance would be multi-threading with the existing ISA-L code.

[jeffareid]

Update - includes links to old and proof of concept code. The proof of concept LFSR code is based on 1987 code used for QIC tape drives. That LFSR code was updated to use PSHUFB in 2006, on an Intel Core 2 X6800, Windows XP X64, Visual Studio 2005, using 5 byte strings for PSHUFB, since VS2005 didn't have it yet. The 2006 test code was verified using a debugger to compare against the 1987 version (there is no output). Similar LFSR code was created by programmers at other tape companies. I bought a laptop with AVX512+GFNI Aug 2025 and ported the LFSR code to use vgf2p8affineqb and vpternlogq. I modded that LFSR code to make it callable from the ISA-L benchmark code (4 parameters instead of 6, m is hard coded to 20: {k,p} = {19,1} {18,2} {17,3} {16,4} {15,5} {14,6}).

[sam-lexx]

@jeffareid - Since you mentioned Windows, I assume you are using Visual Studio 2022 or 2026. Can you make a zip of what is needed to build your affine LFSR code? I also did a PSHUFB version back in 2006, but with standard polynomial, not reciprocal polynomial. I tested ISA-L and @GrokDarkMatter LFSR code with the same parameters as you, m=20. Grok's code is C intrinsics, so it's slower than ISA-L, at least for m <= 20. Do you also have code for standard polynomial instead reciprocal polynomial, which will be slower, but I'd like to see how it compares to ISA-L. Single erasure correction can be done using XOR with reciprocal polynomial if p is odd, but a normal polynomial with one root = 1 is needed if p is even.

[jeffareid]

@sam-lexx

Links to the zip files:

https://rcgldr.net/erszip/ersxenc6.zip

Create a directory c:\ersxenc6 and unzip ersxenc6.zip into it.

https://rcgldr.net/erszip/nasm.zip

Create a directory c:\nasm and unzip nasm.zip into it.

I'm using Visual Studio 2022. The code only supports AVX512+GFNI. The solution | project name is ersxenc6. ersr6a.asm uses self-reciprocal polynomials, while ersp6a.asm uses standard polynomials and is slower. The project uses the file ersx6a.asm, copy ersr6a.asm or ersp6a.asm to ersx6a.asm to do the build. There are defines at the start of erasure_code_perf.c that I added that need to be edited depending on which encoding and how many parities to test. Define RCP to 1 for ersr6a.asm or define BCH to 1 for ersp6a.asm. ersx6a.asm is assembled using masm, while all of the gf_?vect...asm files are assembled using nasm. The total number of rows is defined by M and needs to be 20. The number of parities is defined by P and can be 1 to 6. Note that P=1 is just XOR.

I also tested Grok's code and it is slower than ISA-L since it uses C intrinsics as you noted. I looked at his repository and there have been no updates since 3 months ago, so I assume he lost interest.

No one else here at Intel has shown interest in any alternate encoding scheme, since it would not be compatible with the existing erasure code implementations (BackBlaze, Ceph, ZFS, ...). This was mostly an academic exercise for me, a derivative of my 1987 code which I updated to use PSHUFB in 2006, and recently updated again to use vgf2p8affineqb. The 1987 code was used for software based erasure | error correction on QIC-02 and QIC-40|80|... type tape drives. On later drives, erasure | error correction code was hardware based, with some software involved. After that, software based erasure + error correction code was only done for hardware development | verification. Intel announced PSHUFB in a 2004 publishing of their patent filed for in 2003. The first chips to include PSHUFB were released in June 2006, and my company bought systems that supported PSHUFB to use with LFSR based erasure + error correction software testing soon after. I bought a system that supported PSHUFB in Oct 2006 so I could enhance the LFSR stuff at home. I bought a laptop mid 2025 so I could test AVX512+GFNI based LFSR code at home. Throughout all of this, the code has been based on the 1987 code, using the same unfolded loop using macros logic as seen if you look at the old code from the links in my prior post.