aboutsummaryrefslogtreecommitdiffstats
path: root/module/icp/include
Commit message (Collapse)AuthorAgeFilesLines
* module: icp: remove provider statsнаб2022-02-151-50/+0
| | | | | | | | | | | These were all folded into a single kstat at /proc/spl/kstat/kcf/NONAME_provider_stats with no way to know which one it actually was, and only the AES and SHA (so not Skein) ones were ever updated Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: enforce KCF_{OPS_CLASSSIZE,MAXMECHTAB}наб2022-02-151-1/+7
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused pd_{remove_cv,hash_limit}наб2022-02-151-6/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove vestigia of crypto sessionsнаб2022-02-152-71/+15
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused (and mostly faked) ↵наб2022-02-152-11/+0
| | | | | | | | cm_{{min,max}_key_length,mech_flags} Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused crypto_provider_handle_tнаб2022-02-152-28/+11
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: rip out insane crypto_req_handle_t mechanism, inline KM_SLEEPнаб2022-02-154-87/+43
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused headers. Migrate {ops => sched}_implнаб2022-02-154-354/+5
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* include: crypto: clean out unused SYSCALL32 and flagsнаб2022-02-151-4/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove set-but-unused cd_miscdataнаб2022-02-151-4/+2
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: fold away all key formats except CRYPTO_KEY_RAWнаб2022-02-151-2/+0
| | | | | | | | It's the only one actually used Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused CRYPTO_* error codesнаб2022-02-152-13/+7
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: rip out modhash. Replace the one user with AVLнаб2022-02-153-256/+3
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused me_mutexнаб2022-02-151-1/+0
| | | | | | | | It only needs to be locked if dynamic changes can occur. They can't. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused me_thresholdнаб2022-02-151-4/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused struct crypto_ctx::cc_{session,flags,opstate}наб2022-02-152-22/+1
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused gswq, kcfpool, [as]req_cache, reqid_table, ↵наб2022-02-152-227/+0
| | | | | | | | obsolete kstat Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused notification frameworkнаб2022-02-152-70/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused kcf_op_{group,type}, req_params, ...наб2022-02-153-307/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused p[di]_flagsнаб2022-02-152-26/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused CRYPTO_{NOTIFY_OPDONE,SKIP_REQID,RESTRICTED}наб2022-02-151-5/+1
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove unused CRYPTO_ALWAYS_QUEUEнаб2022-02-151-5/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: drop software provider generation numbersнаб2022-02-151-5/+0
| | | | | | | | We register all providers at once, before anything happens Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: remove other provider typesнаб2022-02-153-147/+24
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: use original mechanismsнаб2022-02-151-2/+2
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: use original descriptionнаб2022-02-152-2/+2
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: spi: crypto_ops_t: remove unused op typesнаб2022-02-154-1404/+23
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: spi: flatten struct crypto_ops, crypto_provider_infoнаб2022-02-151-57/+2
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: spi: remove crypto_control_ops_tнаб2022-02-152-32/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* module: icp: spi: remove crypto_{provider,op}_notification()наб2022-02-151-2/+0
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12901
* icp: asm_linkage.h: clean out unused bits, CSTYLEDнаб2022-01-261-143/+3
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12993
* module/*.ko: prune .data, global .rodataнаб2022-01-143-31/+16
| | | | | | | | | | | | Evaluated every variable that lives in .data (and globals in .rodata) in the kernel modules, and constified/eliminated/localised them appropriately. This means that all read-only data is now actually read-only data, and, if possible, at file scope. A lot of previously- global-symbols became inlinable (and inlined!) constants. Probably not in a big Wowee Performance Moment, but hey. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12899
* Remove sha1 hashing from OpenZFS, it's not used anywhere.Tino Reichardt2022-01-063-199/+0
| | | | | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Attila Fülöp <[email protected]> Signed-off-by: Tino Reichardt <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12895 Closes #12902
* module: icp: remove solaris crypto and cryptoadm ioctl definitionsнаб2022-01-063-1620/+0
| | | | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Attila Fülöp <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12895 Closes #12902
* module: icp: rip out the Solaris loadable module architectureнаб2022-01-061-477/+0
| | | | | | | | | | | | | | | | | After progressively folding away null cases, it turns out there's /literally/ nothing there, even if some things are part of the Solaris SPARC DDI/DKI or the seventeen module types (some doubled for 32-bit userland), or the entire modctl syscall definition. Nothing. Initialisation is handled in illumos-crypto.c, which calls all the initialisers directly Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Attila Fülöp <[email protected]> Signed-off-by: Ahelenia Ziemiańska <[email protected]> Closes #12895 Closes #12902
* ICP: gcm: Allocate hash subkey table separatelyAttila Fülöp2020-10-301-4/+4
| | | | | | | | | | | | | | While evaluating other assembler implementations it turns out that the precomputed hash subkey tables vary in size, from 8*16 bytes (avx2/avx512) up to 48*16 bytes (avx512-vaes), depending on the implementation. To be able to handle the size differences later, allocate `gcm_Htable` dynamically rather then having a fixed size array, and adapt consumers. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Attila Fülöp <[email protected]> Closes #11102
* Make use of ZFS_DEBUG consistent within kmod sourcesMatthew Macy2020-07-251-1/+1
| | | | | | Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Ryan Moeller <[email protected]> Signed-off-by: Matt Macy <[email protected]> Closes #10623
* Add prototypesArvind Sankar2020-06-181-0/+10
| | | | | | | | | Add prototypes/move prototypes to header files. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10470
* Remove dead codeArvind Sankar2020-06-181-2/+2
| | | | | | | | | Delete unused functions. Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Arvind Sankar <[email protected]> Closes #10470
* Remove checks for null out value in encryption pathsDirkjan Bussink2020-03-261-3/+0
| | | | | | | | | | | These paths are never exercised, as the parameters given are always different cipher and plaintext `crypto_data_t` pointers. Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Attila Fueloep <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Closes #9661 Closes #10015
* ICP: gcm-avx: Support architectures lacking the MOVBE instructionAttila Fülöp2020-03-171-1/+2
| | | | | | | | | | | | | | | | | There are a couple of x86_64 architectures which support all needed features to make the accelerated GCM implementation work but the MOVBE instruction. Those are mainly Intel Sandy- and Ivy-Bridge and AMD Bulldozer, Piledriver, and Steamroller. By using MOVBE only if available and replacing it with a MOV followed by a BSWAP if not, those architectures now benefit from the new GCM routines and performance is considerably better compared to the original implementation. Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Adam D. Moss <[email protected]> Signed-off-by: Attila Fülöp <[email protected]> Followup #9749 Closes #10029
* ICP: Improve AES-GCM performanceAttila Fülöp2020-02-102-2/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently SIMD accelerated AES-GCM performance is limited by two factors: a. The need to disable preemption and interrupts and save the FPU state before using it and to do the reverse when done. Due to the way the code is organized (see (b) below) we have to pay this price twice for each 16 byte GCM block processed. b. Most processing is done in C, operating on single GCM blocks. The use of SIMD instructions is limited to the AES encryption of the counter block (AES-NI) and the Galois multiplication (PCLMULQDQ). This leads to the FPU not being fully utilized for crypto operations. To solve (a) we do crypto processing in larger chunks while owning the FPU. An `icp_gcm_avx_chunk_size` module parameter was introduced to make this chunk size tweakable. It defaults to 32 KiB. This step alone roughly doubles performance. (b) is tackled by porting and using the highly optimized openssl AES-GCM assembler routines, which do all the processing (CTR, AES, GMULT) in a single routine. Both steps together result in up to 32x reduction of the time spend in the en/decryption routines, leading up to approximately 12x throughput increase for large (128 KiB) blocks. Lastly, this commit changes the default encryption algorithm from AES-CCM to AES-GCM when setting the `encryption=on` property. Reviewed-By: Brian Behlendorf <[email protected]> Reviewed-By: Jason King <[email protected]> Reviewed-By: Tom Caputi <[email protected]> Reviewed-By: Richard Laager <[email protected]> Signed-off-by: Attila Fülöp <[email protected]> Closes #9749
* Fix icp build on FreeBSDMatthew Macy2019-11-011-1/+1
| | | | | | | | | | | | - ROTATE_LEFT is not used by amd64, move it down within the scope it's used to silence a clang warning. - __unused is an alias for the compiler annotation __attribute__((__unused__)) on FreeBSD. Rename the field to ____unused. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Matt Macy <[email protected]> Closes #9538
* Linux 4.14, 4.19, 5.0+ compat: SIMD save/restoreBrian Behlendorf2019-10-242-2/+2
| | | | | | | | | | | | | | | | | | | | Contrary to initial testing we cannot rely on these kernels to invalidate the per-cpu FPU state and restore the FPU registers. Nor can we guarantee that the kernel won't modify the FPU state which we saved in the task struck. Therefore, the kfpu_begin() and kfpu_end() functions have been updated to save and restore the FPU state using our own dedicated per-cpu FPU state variables. This has the additional advantage of allowing us to use the FPU again in user threads. So we remove the code which was added to use task queues to ensure some functions ran in kernel threads. Reviewed-by: Fabian Grünbichler <[email protected]> Reviewed-by: Tony Hutter <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Issue #9346 Closes #9403
* Fix typos in modules/icp/Andrea Gelmini2019-08-303-5/+5
| | | | | | | Reviewed-by: Ryan Moeller <[email protected]> Reviewed-by: Richard Laager <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Andrea Gelmini <[email protected]> Closes #9239
* Minor style cleanupBrian Behlendorf2019-07-162-3/+3
| | | | | | | | | | Resolve an assortment of style inconsistencies including use of white space, typos, capitalization, and line wrapping. There is no functional change. Reviewed-by: Tony Hutter <[email protected]> Reviewed-by: George Melikov <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #9030
* Linux 5.0 compat: SIMD compatibilityBrian Behlendorf2019-07-122-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Restore the SIMD optimization for 4.19.38 LTS, 4.14.120 LTS, and 5.0 and newer kernels. This is accomplished by leveraging the fact that by definition dedicated kernel threads never need to concern themselves with saving and restoring the user FPU state. Therefore, they may use the FPU as long as we can guarantee user tasks always restore their FPU state before context switching back to user space. For the 5.0 and 5.1 kernels disabling preemption and local interrupts is sufficient to allow the FPU to be used. All non-kernel threads will restore the preserved user FPU state. For 5.2 and latter kernels the user FPU state restoration will be skipped if the kernel determines the registers have not changed. Therefore, for these kernels we need to perform the additional step of saving and restoring the FPU registers. Invalidating the per-cpu global tracking the FPU state would force a restore but that functionality is private to the core x86 FPU implementation and unavailable. In practice, restricting SIMD to kernel threads is not a major restriction for ZFS. The vast majority of SIMD operations are already performed by the IO pipeline. The remaining cases are relatively infrequent and can be handled by the generic code without significant impact. The two most noteworthy cases are: 1) Decrypting the wrapping key for an encrypted dataset, i.e. `zfs load-key`. All other encryption and decryption operations will use the SIMD optimized implementations. 2) Generating the payload checksums for a `zfs send` stream. In order to avoid making any changes to the higher layers of ZFS all of the `*_get_ops()` functions were updated to take in to consideration the calling context. This allows for the fastest implementation to be used as appropriate (see kfpu_allowed()). The only other notable instance of SIMD operations being used outside a kernel thread was at module load time. This code was moved in to a taskq in order to accommodate the new kernel thread restriction. Finally, a few other modifications were made in order to further harden this code and facilitate testing. They include updating each implementations operations structure to be declared as a constant. And allowing "cycle" to be set when selecting the preferred ops in the kernel as well as user space. Reviewed-by: Tony Hutter <[email protected]> Signed-off-by: Brian Behlendorf <[email protected]> Closes #8754 Closes #8793 Closes #8965
* Add support for selecting encryption backendNathan Lewis2018-08-022-4/+121
| | | | | | | | | | | | | | | | | | - Add two new module parameters to icp (icp_aes_impl, icp_gcm_impl) that control the crypto implementation. At the moment there is a choice between generic and aesni (on platforms that support it). - This enables support for AES-NI and PCLMULQDQ-NI on AMD Family 15h (bulldozer) and newer CPUs (zen). - Modify aes_key_t to track what implementation it was generated with as key schedules generated with various implementations are not necessarily interchangable. Reviewed by: Gvozden Neskovic <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Reviewed-by: Tom Caputi <[email protected]> Reviewed-by: Richard Laager <[email protected]> Signed-off-by: Nathaniel R. Lewis <[email protected]> Closes #7102 Closes #7103
* [icp] fpu and asm cleanup for linuxGvozden Neskovic2017-03-071-0/+10
| | | | | | | | | | | | | | Properly annotate functions and data section so that objtool does not complain when CONFIG_STACK_VALIDATION and CONFIG_FRAME_POINTER are enabled. Pass KERNELCPPFLAGS to assembler. Use kfpu_begin()/kfpu_end() to protect SIMD regions in Linux kernel. Reviewed-by: Tom Caputi <[email protected]> Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Gvozden Neskovic <[email protected]> Closes #5872 Closes #5041
* Fix spellingka72017-01-031-1/+1
| | | | | | | | | Reviewed-by: Brian Behlendorf <[email protected] Reviewed-by: Giuseppe Di Natale <[email protected]>> Reviewed-by: George Melikov <[email protected]> Reviewed-by: Haakan T Johansson <[email protected]> Closes #5547 Closes #5543
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-04 22:34:51 +0000