summaryrefslogtreecommitdiffstats
path: root/src/util/disk_cache.c
Commit message (Collapse)AuthorAgeFilesLines
* util/u_queue: add an option to set the minimum thread priorityMarek Olšák2017-06-071-1/+1
| | | | Reviewed-by: Nicolai Hähnle <[email protected]>
* util/disk_cache: add new driver_flags param to cache keysTimothy Arceri2017-05-231-1/+7
| | | | | | | | | This will be used for things such as adding driver specific environment variables to the key. Allowing us to set environment vars that change the shader and not have the driver ignore them if it finds existing shaders in the cache. Reviewed-by: Eduardo Lima Mitev <[email protected]>
* util/disk_cache: remove percentage based max cache limitTimothy Arceri2017-04-281-6/+2
| | | | | | | | | | | | | | | | The more I think about it the more this seems like a bad idea. When we were deleting old cache dirs this wasn't so bad as it was unlikely we would ever hit the actual limit before things were cleaned up. Now that we only start cleaning up old cache items once the limit is reached the a percentage based max cache limit is more risky. For the inital release of shader cache I think its better to stick to a more conservative cache limit, at least until we have some way of cleaning up the cache more aggressively. Cc: "17.1" <[email protected]> Reviewed-by: Michel Dänzer <[email protected]>
* disk_cache: use block size rather than file sizeTimothy Arceri2017-04-271-5/+11
| | | | | | | | | | | | | | | | The majority of cache files are less than 1kb this resulted in us greatly miscalculating the amount of disk space used by the cache. Using the number of blocks allocated to the file is more conservative and less likely to cause issues. This change will result in cache sizes being miscalculated further until old items added with the previous calculation have all been removed. However I don't see anyway around that, the previous patch should help limit that problem. Cc: "17.1" <[email protected]> Reviewed-and-Tested-by: Michel Dänzer <[email protected]>
* disk_cache: reduce default cache size to 5% of filesystemTimothy Arceri2017-04-271-2/+2
| | | | | | | | | | | Modern disks are extremely large and are only going to get bigger. Usage has shown frequent Mesa upgrades can result in the cache growing very fast i.e. wasting a lot of disk space unnecessarily. 5% seems like a more reasonable default. Cc: "17.1" <[email protected]> Acked-by: Michel Dänzer <[email protected]>
* util/disk_cache: don't deadlock on premature EOFGrazvydas Ignotas2017-03-251-17/+26
| | | | | | | | | | | | | | If we get EOF earlier than expected, the current read loops will deadlock. This may easily happen if the disk cache gets corrupted. Fix it by using a helper function that handles EOF. Steps to reproduce (on a build with asserts disabled): $ glxgears $ find ~/.cache/mesa/ -type f -exec truncate -s 0 '{}' \; $ glxgears # deadlock Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: write cache entry keys to file headerTimothy Arceri2017-03-241-6/+44
| | | | | | | | | | | | This can be used to deal with key hash collisions from different versions (should we find that to actually happen) and to find which mesa version produced the cache entry. V2: use blob created at cache creation. v3: remove left over var from v1. Reviewed-by: Grazvydas Ignotas <[email protected]>
* util/disk_cache: hash pointer size and gpu name into cache keysGrazvydas Ignotas2017-03-241-26/+15
| | | | | | | | | | | | | | | | This allows to get rid of the arch and gpu name directories. v2: (Timothy Arceri) don't use an opaque data type to store pointer size and gpu name. v3: (Timothy Arceri) use blob to store driver keys just make sure to store null terminator for strings, and make sure blob is defined by disk_cache and not it's users. v4: (Timothy Arceri) fix typo, and make ptr_size a uint8_t. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Nicolai Hähnle <[email protected]>
* util/disk_cache: hash timestamps into the cache keysGrazvydas Ignotas2017-03-241-51/+26
| | | | | | | | | | | | | | | | | | | | | | | | Instead of using a directory, hash the timestamps into the cache keys themselves. Since there is no more timestamp directory, there is no more need for deleting the cache of other mesa versions and we rely on eviction to clean up the old cache entries. This solves the problem of using several incarnations of disk_cache at the same time, where one deletes a directory belonging to the other, like when both OpenGL and gallium nine are used simultaneously (or several different mesa installations). v2: using additional blob instead of trying to clone sha1 state v3: (Timothy Arceri) don't use an opaque data type to store timestamp. V4: (Timothy Arceri) use blob to store driver keys just make sure to store null terminator for strings, and make sure blob is defined by disk_cache and not it's users. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Nicolai Hähnle <[email protected]> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=100091
* util/disk_cache: use rand_xorshift128plus() to get our random intTimothy Arceri2017-03-231-8/+10
| | | | | | | | | | | Otherwise for apps that don't seed the regular rand() we will always remove old cache entries from the same dirs. V2: assume bits returned by rand are independent uniformly distributed bits and grab our hex value without taking the modulus of the whole value, this also fixes a bug where 'f' was always missing. Reviewed-by: Nicolai Hähnle <[email protected]>
* util/disk_cache: add a write helperGrazvydas Ignotas2017-03-211-14/+23
| | | | | | | | | | | | | Simplifies the write code a bit and handles EINTR. V2: (Timothy Arceri) Drop EINTR handling. To do it properly we would need a retry limit but it's probably best to just avoid trying to write if we hit EINTR and try again next time we see the program. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: use a helper to compute cache keysGrazvydas Ignotas2017-03-211-0/+7
| | | | | | | | | | This will allow to hash additional data into the cache keys or even change the hashing algorithm easily, should we decide to do so. v2: don't try to compute key (and crash) if cache is disabled Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: check rename resultGrazvydas Ignotas2017-03-201-2/+6
| | | | | | | | | I haven't seen this causing problems in practice, but for correctness we should also check if rename succeeded to avoid breaking accounting and leaving a .tmp file behind. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: delete .tmp if target existsGrazvydas Ignotas2017-03-201-1/+3
| | | | | | | | | | | At the time of target file check, .tmp file is already created and file lock is held, so we should remove the .tmp, like in other error paths. With this, piglit no longer leaves large amount of empty .tmp files behind, which waste directory entries and may interfere with eviction. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: fix stored_keys indexGrazvydas Ignotas2017-03-201-2/+2
| | | | | | | | | | | | | | It seems there is a bug because: - 20 bytes are compared, but only 1 byte stored_keys step is used - entries can overlap each other by 19 bytes - index_mmap is ~1.3M in size, but only first 64K is used With this fix for Deus Ex: - startup time (from launch to Feral logo): ~38s -> ~16s - disk_cache_has_key() hit rate: ~50% -> ~96% Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: pass predicate functions file stats directly (v4)Alan Swanson2017-03-181-34/+21
| | | | | | | | | | | | | | | Since switching to LRU eviction the only user of these predicate functions now resolves directory entry stats itself so pass them directly saving calling fstat and strlen twice (and the expensive strlen is skipped entirely if access time is newer). v2: Update for empty cache dir detection changes v3: Fix passing string length to predicate with the +1 for NULL termination and also pass sb as pointer v4: Missed ampersand for passing sb as pointer Reviewed-by: Grazvydas Ignotas <[email protected]> Acked-by: Timothy Arceri <[email protected]>
* util/disk_cache: do eviction before creating .tmpGrazvydas Ignotas2017-03-161-10/+8
| | | | | | | | | | | | | | cache_put() first creates a .tmp file and then tries to do eviction. The recently added LRU eviction code selects non-empty directory with the oldest access time, but that may easily be the one with just the new .tmp file, especially on Linux where atime is updated lazily (with "relatime" mount option, which is the default). So when cache is small, if random doesn't hit another dir LRU keeps selecting the same dir with just the .tmp and not deleting anything. To fix this (and the tests), do eviction earlier. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: scale cache according to filesystem sizeAlan Swanson2017-03-151-3/+8
| | | | | | | | Select higher of current 1G default or 10% of filesystem where cache is located. Acked-by: Timothy Arceri <[email protected]> Reviewed-by: Grazvydas Ignotas <[email protected]>
* util/disk_cache: actually enforce cache sizeAlan Swanson2017-03-151-1/+5
| | | | | | | | | | Currently only a one in one out eviction so if at max_size and cache files were to constantly increase in size then so would the cache. Restrict to limit of 8 evictions per new cache entry. V2: (Timothy Arceri) fix make check tests Reviewed-by: Grazvydas Ignotas <[email protected]>
* util/disk_cache: use LRU eviction rather than random evictionAlan Swanson2017-03-151-43/+34
| | | | | | | | | | | Still using fast random selection of two-character subdirectory in which to check cache files rather than scanning entire cache. v2: Factor out double strlen call v3: C99 declaration of variables where used Reviewed-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: don't fallback to an empty cache dir on evictTimothy Arceri2017-03-151-6/+27
| | | | | | | | | | If we fail to randomly select a two letter cache dir, don't select an empty dir on fallback. In real world use we should never hit the fallback path but it can be hit by tests when the cache is set to a very small max value. Reviewed-by: Grazvydas Ignotas <[email protected]>
* util/disk_cache: use a thread queue to write to shader cacheTimothy Arceri2017-03-151-13/+28
| | | | | | | | | | | | | | This should help reduce any overhead added by the shader cache when programs are not found in the cache. To avoid creating any special function just for the sake of the tests we add a one second delay whenever we call dick_cache_put() to give it time to finish. V2: poll for file when waiting for thread in test V3: fix poll delay to really be 100ms, and simplify the wait function Reviewed-by: Grazvydas Ignotas <[email protected]>
* util/disk_cache: add helpers for creating/destroying disk cache put jobsTimothy Arceri2017-03-151-0/+40
| | | | | | | V2: Make a copy of the data so we don't have to worry about it being freed before we are done compressing/writing. Reviewed-by: Grazvydas Ignotas <[email protected]>
* util/disk_cache: add thread queue to disk cacheTimothy Arceri2017-03-151-1/+15
| | | | | Reviewed-by: Marek Olšák <[email protected]> Reviewed-by: Grazvydas Ignotas <[email protected]>
* util/disk_cache: fix size subtraction on 32bitGrazvydas Ignotas2017-03-091-3/+3
| | | | | | | | | | Negating size_t on 32bit produces a 32bit result. This was effectively adding values close to UINT_MAX to the cache size (the files are usually small) instead of intended subtraction. Fixes 'make check' disk_cache failures on 32bit. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: fix compressed size calculationGrazvydas Ignotas2017-03-091-1/+1
| | | | | | | | | It incorrectly doubles the size on each iteration. Fixes: 85a9b1b5 "util/disk_cache: compress individual cache entries" Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: fix make checkTimothy Arceri2017-03-061-17/+0
| | | | | Fixes make check after 11f0efec2e615f5233d which caused disk cache to create an additional directory.
* Revert "glsl: Switch to disable-by-default for the GLSL shader cache"Timothy Arceri2017-03-061-7/+0
| | | | | | | | | This reverts commit 0f60c6616e93cba72bff4fbfedb72a753ef78e05. Piglit and all games tested so far seem to be working without issue. This change will allow wide user testing and we can decided before the next release if we need to turn it off again. Reviewed-by: Marek Olšák <[email protected]>
* util/disk_cache: support caches for multiple architecturesTimothy Arceri2017-03-061-0/+26
| | | | | | | | | | | | | | Previously we were deleting the entire cache if a user switched between 32 and 64 bit applications. V2: make the check more generic, it should now work with any platform we are likely to support. V3: Use suggestion from Emil to make even more generic/fix issue with __ILP32__ not being declared on gcc for regular 32-bit builds. Tested-by: Grazvydas Ignotas <[email protected]> Tested-by: Dieter Nützel <[email protected]>
* util/disk_cache: mark read-only arguments constGrazvydas Ignotas2017-03-061-17/+18
| | | | | | | No functional changes. Signed-off-by: Grazvydas Ignotas <[email protected]> Reviewed-by: Timothy Arceri <[email protected]>
* util/disk_cache: compress individual cache entriesTimothy Arceri2017-03-031-23/+150
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reduces the cache size for Deus Ex from ~160M to ~30M for radeonsi (these numbers differ from Grigori's results below probably due to different graphics quality settings). I'm also seeing the following improvements in minimum fps in the Shadow of Mordor benchmark on an i5-6400 [email protected], with a HDD: no-cache: ~10fps with-cache-no-compression: ~15fps with-cache-and-compression: ~20fps Note: The with cache results are from the second run after closing and opening the game to avoid the in-memory cache. Since we mainly care about decompression I went with Z_BEST_COMPRESSION as suggested on irc by Steinar H. Gunderson who has benchmarked decompression speeds. Grigori Goronzy provided the following stats for Deus Ex: Mankind Divided start-up times on a Athlon X4 860k with a SSD: No Cache 215 sec Cold Cache zlib BEST_COMPRESSION 285 sec Warm Cache zlib BEST_COMPRESSION 33 sec Cold Cache zlib BEST_SPEED 264 sec Warm Cache zlib BEST_SPEED 33 sec Cold Cache no compression 266 sec Warm Cache no compression 34 sec The total cache size for that game is 48 MiB with BEST_COMPRESSION, 56 MiB with BEST_SPEED and 170 MiB with no compression. These numbers suggest that it may be ok to go with Z_BEST_SPEED but we should gather some actual decompression times before doing so. Other options might be to do the compression in a separate thread, this might allow us to use a higher compression algorithim such as LZMA. Reviewed-by: Grigori Goronzy <[email protected]> Acked-by: Marek Olšák <[email protected]>
* util/disk_cache: add support for detecting corrupt cache entriesTimothy Arceri2017-03-031-3/+34
| | | | | | | V2: fix pointer increments for writing/reading crc Acked-by: Marek Olšák <[email protected]> Reviewed-by: Grigori Goronzy <[email protected]>
* util/disk_cache: fix bug with deleting old cache dirsTimothy Arceri2017-02-231-4/+6
| | | | | | | | | | If there was more than a single directory in the .cache/mesa dir then it would only remove one (or none) of the directories. Apparently Valgrind was also reporting: Conditional jump or move depends on uninitialised value Reviewed-by: Edward O'Callaghan <[email protected]>
* util/disk_cache: create timestamp and gpu_id dirs when MESA_GLSL_CACHE_DIR ↵Timothy Arceri2017-02-221-2/+8
| | | | | | | | is used The make check test is also updated to make sure these dirs are created. Reviewed-by: Nicolai Hähnle <[email protected]>
* glsl: disable on disk shader cache when running as another userTimothy Arceri2017-02-171-0/+4
| | | | Reviewed-by: Nicolai Hähnle <[email protected]>
* util/disk_cache: check cache exists before calling munmap()Timothy Arceri2017-02-171-1/+2
| | | | Reviewed-by: Nicolai Hähnle <[email protected]>
* util/disk_cache: add support for removing old versions of the cacheTimothy Arceri2017-02-171-0/+37
| | | | Reviewed-by: Nicolai Hähnle <[email protected]>
* util/disk_cache: allow drivers to pass a directory structureTimothy Arceri2017-02-171-4/+24
| | | | | | | | | | | | | | In order to avoid costly fallback recompiles when cache items are created with an old version of Mesa or for a different gpu on the same system we want to create directories that look like this: ./{TIMESTAMP}_{LLVM_TIMESTAMP}/{GPU_ID} Note: The disk cache util will take a single timestamp string, it is up to the backend to concatenate the llvm string with the mesa string if applicable. Reviewed-by: Nicolai Hähnle <[email protected]>
* util/disk_cache: do not allow space in MESA_GLSL_CACHE_MAX_SIZEEmil Velikov2017-02-161-2/+0
| | | | | | | No other env var used in mesa allows for space in the variable contents. Signed-off-by: Emil Velikov <[email protected]> Acked-by: Timothy Arceri <[email protected]>
* util/disk_cache: correctly use stat(3)Timothy Arceri2017-02-131-4/+4
| | | | | | | | | I forgot to error check stat() and also I wasn't using the subdir in is_two_character_sub_directory(). Fixes: d7b3707c612 "util/disk_cache: use stat() to check if entry is a directory" Reviewed-by: Plamena Manolova <[email protected]> Reviewed-by: Emil Velikov <[email protected]>
* util/disk_cache: use stat() to check if entry is a directoryTimothy Arceri2017-02-101-9/+24
| | | | | | | | d_type is not supported on all systems. Tested-by: Vinson Lee <[email protected]> Reviewed-by: Emil Velikov <[email protected]> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=97967
* util/disk_cache: error check asprintf()Timothy Arceri2017-02-101-5/+7
| | | | | | | Fixes: f3d911463e8 "util/disk_cache: stop using ralloc_asprintf() unnecessarily" Reviewed-by: Emil Velikov <[email protected]> Reviewed-by: Eric Engestrom <[email protected]>
* util/disk_cache: stop using ralloc_asprintf() unnecessarilyTimothy Arceri2017-02-091-13/+12
| | | | Reviewed-by: Anuj Phogat <[email protected]>
* util: add a disk_cache_remove() functionTimothy Arceri2017-02-091-0/+22
| | | | | | | | | | This will be used to remove cache items created with old versions of Mesa or other invalid cache items from the cache. V2: rename stub function (cache_* funtions were renamed disk_cache_*) in master. Reviewed-by: Anuj Phogat <[email protected]>
* glsl: Switch to disable-by-default for the GLSL shader cacheCarl Worth2017-01-311-0/+7
| | | | | | | | | | | | | | | | | The shader cache is expected to be developed incrementally over a fairly long series of commits. For that period of instability, we require users to opt into the shader cache by setting: MESA_GLSL_CACHE_ENABLE=1 In the future, when the shader cache is complete, we can revert this commit so that the cache will be on by default. The user can always disable the cache with MESA_GLSL_CACHE_DISABLE=1. That functionality is not affected by this commit, (nor will it be affected by the future revert). Reviewed-by: Eric Anholt <[email protected]>
* utils: build sha1/disk cache only with Android/AutoconfEmil Velikov2017-01-181-0/+4
| | | | | | | | | | Earlier commit imported a SHA1 implementation and relaxed the SHA1 and disk cache handling, broking the Windows builds. Restrict things for now until we get to a proper fix. Fixes: d1efa09d342 "util: import sha1 implementation from OpenBSD" Signed-off-by: Emil Velikov <[email protected]>
* util: import sha1 implementation from OpenBSD17.0-branchpointEmil Velikov2017-01-181-4/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | At the moment we support 5+ different implementations each with varying amount of bugs - from thread safely problems [1], to outright broken implementation(s) [2] In order to accommodate these we have 150+ lines of configure script and extra two configure toggles. Whist an actual implementation being ~200loc and our current compat wrapping ~250. Let's not forget that different people use different code paths, thus effectively makes it harder to test and debug since the default implementation is automatically detected. To minimise all these lovely experiences, import the "100% Public Domain" OpenBSD sha1 implementation. Clearly document any changes needed to get building correctly, since many/most of those can be upstreamed making future syncs easier. As an added bonus this will avoid all the 'fun' experiences trying to integrate it with the Android and SCons builds. v2: Manually expand __BEGIN_DECLS/__END_DECLS and document (Tapani). Furthermore it seems that some games (or surrounding runtime) static link against OpenSSL resulting in conflicts. For more information see the discussion thread [3] Bugzilla [1]: https://bugs.freedesktop.org/show_bug.cgi?id=94904 Bugzilla [2]: https://bugs.freedesktop.org/show_bug.cgi?id=97967 [3] https://lists.freedesktop.org/archives/mesa-dev/2017-January/140748.html Cc: Mark Janes <[email protected]> Cc: Vinson Lee <[email protected]> Cc: Tapani Pälli <[email protected]> Cc: Jonathan Gray <[email protected]> Tested-by: Jonathan Gray <[email protected]> Signed-off-by: Emil Velikov <[email protected]> Acked-by: Tapani Pälli <[email protected]> (v1) Acked-by: Jason Ekstrand <[email protected]> (v1)
* util/disk_cache: close a previously opened handle in disk_cache_put (v2)Gwan-gyeong Mun2016-11-221-6/+5
| | | | | | | | | | | | | | We're missing the close() to the matching open(). CID 1373407 v2: Fixes from Emil Velikov's review Update the teardown in reverse order of the setup/init. Cc: "13.0" <[email protected]> Signed-off-by: Mun Gwan-gyeong <[email protected]> Reviewed-by: Emil Velikov <[email protected]> Reviewed-by: Iago Toral Quiroga <[email protected]> (v1)
* util/disk_cache: use unambiguous namingMarek Olšák2016-11-151-15/+15
| | | | Reviewed-by: Emil Velikov <[email protected]>
* util: import cache.c/h from glslMarek Olšák2016-11-151-0/+712
It's not dependent on GLSL and it can be useful for shader caches that don't deal with GLSL. v2: address review comments v3: keep the other 3 lines in configure.ac Reviewed-by: Emil Velikov <[email protected]>
generated by cgit v1.2.3 (git 2.39.1) at 2024-12-28 08:40:00 +0000