Bug 260962

Summary: Network process crash in WebKit::CacheStorageDiskStore::readRecords
Product: WebKit Reporter: Michael Catanzaro <mcatanzaro>
Component: WebCore Misc.Assignee: Nobody <webkit-unassigned>
Status: NEW    
Severity: Normal CC: ddkilzer, mcatanzaro, sihui_liu, webkit-bug-importer
Priority: P2 Keywords: InRadar
Version: WebKit Nightly Build   
Hardware: PC   
OS: Linux   
Attachments:
Description Flags
Full backtrace none

Michael Catanzaro
Reported 2023-08-31 07:03:03 PDT
Created attachment 467506 [details] Full backtrace I've just seen this network process crash twice in the past hour. The short backtrace is pasted below. Notably, we tried to memcpy 18446744073709551615 (2^64-1) bytes, which is not good. Program terminated with signal SIGBUS, Bus error. #0 memcpy (__src=0x7f36cc554299, __len=8, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29 29 return __builtin___memcpy_chk (__dest, __src, __len, [Current thread is 1 (Thread 0x7f366bfff6c0 (LWP 31))] (gdb) bt #0 memcpy (__src=0x7f36cc554299, __len=8, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29 #1 WTF::Persistence::Decoder::decodeNumber<unsigned long>(std::optional<unsigned long>&) (this=0x7f366bffdcd0, optional=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/persistence/PersistentDecoder.cpp:85 #2 WTF::Persistence::Decoder::operator>>(std::optional<unsigned long>&) (this=0x7f366bffdcd0, result=std::optional<unsigned long> [no contained value]) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/persistence/PersistentDecoder.cpp:120 #3 0x00007f36cf806865 in WTF::Persistence::Decoder::operator>><WebCore::FetchOptionsDestination, (void*)0>(std::optional<WebCore::FetchOptionsDestination>&) (this=0x7f366bffdcd0, result=<optimized out>) at WTF/Headers/wtf/persistence/PersistentDecoder.h:75 #4 WebCore::FetchOptions::decodePersistent<WTF::Persistence::Decoder>(WTF::Persistence::Decoder&, WebCore::FetchOptions&) (decoder=..., options=...) at WebCore/PrivateHeaders/WebCore/FetchOptions.h:215 #5 0x00007f36cf854351 in WebKit::decodeRecordHeader(std::span<unsigned char const, 18446744073709551615ul>) (headerData=Python Exception <class 'gdb.error'>: value has been optimized out std::span of length 1967) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:234 #6 WebKit::readRecordInfoFromFileData(std::array<unsigned char, 8ul> const&, std::span<unsigned char const, 18446744073709551615ul>) (salt=..., fileData=Python Exception <class 'gdb.error'>: value has been optimized out ) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:294 #7 0x00007f36cf8536a7 in WebKit::CacheStorageDiskStore::readRecordFromFileData(std::span<unsigned char const, 18446744073709551615ul>, WTF::FileSystemImpl::MappedFileData&&) (this=<optimized out>, buffer=std::span of length 1528 = {...}, blobBuffer=...) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:305 #8 0x00007f36cf8627e6 in WebKit::CacheStorageDiskStore::readRecords(WTF::Vector<WebKit::CacheStorageRecordInformation, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> const&, WTF::CompletionHandler<void (WTF::Vector<std::optional<WebKit::CacheStorageRecord>, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_1::operator()<WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> >(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>) (this=0x7f36bd0b91a8, fileDatas=WTF::Vector of length 0, capacity 0, blobDatas=WTF::Vector of length 0, capacity 0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:412 #9 WTF::Detail::CallableWrapper<WebKit::CacheStorageDiskStore::readRecords(WTF::Vector<WebKit::CacheStorageRecordInformation, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> const&, WTF::CompletionHandler<void (WTF::Vector<std::optional<WebKit::CacheStorageRecord>, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_1, void, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&>::call(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) (this=0x7f36bd0b91a0, in=<optimized out>, in=<optimized out>) at WTF/Headers/wtf/Function.h:53 #10 0x00007f36cf862547 in WTF::Function<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>::operator()(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) const (in=..., in=..., this=<optimized out>) at WTF/Headers/wtf/Function.h:82 #11 WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>::operator()(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) (this=<optimized out>, in=..., in=...) at WTF/Headers/wtf/CompletionHandler.h:75 #12 WebKit::CacheStorageDiskStore::readRecordsInternal(WTF::Vector<WTF::String, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow--Type <RET> for more, q to quit, c to continue without paging--c , 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_0::operator()()::{lambda()#1}::operator()() (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:395 #13 WTF::Detail::CallableWrapper<WebKit::CacheStorageDiskStore::readRecordsInternal(WTF::Vector<WTF::String, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_0::operator()()::{lambda()#1}, void>::call() (this=<optimized out>) at WTF/Headers/wtf/Function.h:53 #14 0x00007f36ce2f834b in WTF::Function<void ()>::operator()() const (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:82 #15 WTF::RunLoop::performWork() (this=0x7f36bd03c180) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/RunLoop.cpp:147 #16 0x00007f36ce3520fd in WTF::RunLoop::RunLoop()::$_0::operator()(void*) const (userData=0x7f366bffdcd0, userData@entry=0x7f36bd03c180, this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:80 #17 WTF::RunLoop::RunLoop()::$_0::__invoke(void*) (userData=0x7f366bffdcd0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:79 #18 0x00007f36ce351501 in WTF::RunLoop::$_0::operator()(_GSource*, int (*)(void*), void*) const (source=0x7f3660000dc0, callback=0x7f36ce3520f0 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f36bd03c180, this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:53 #19 WTF::RunLoop::$_0::__invoke(_GSource*, int (*)(void*), void*) (source=0x7f3660000dc0, callback=0x7f36ce3520f0 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f36bd03c180) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:45 #20 0x00007f36ca8b6c97 in g_main_dispatch (context=context@entry=0x7f3660000b70) at ../glib/gmain.c:3476 #21 0x00007f36ca8b8da7 in g_main_context_dispatch_unlocked (context=0x7f3660000b70) at ../glib/gmain.c:4284 #22 g_main_context_iterate_unlocked (context=0x7f3660000b70, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4349 #23 0x00007f36ca8b9757 in g_main_loop_run (loop=0x7f3660000da0) at ../glib/gmain.c:4551 #24 0x00007f36ce351ad1 in WTF::RunLoop::run() () at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:108 #25 0x00007f36ce2fb9a7 in WTF::Function<void ()>::operator()() const (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:82 #26 WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*) (newThreadContext=0x7f36bd0340e0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Threading.cpp:250 #27 0x00007f36ce35582d in WTF::wtfThreadEntryPoint(void*) (context=0x7f366bffdcd0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/posix/ThreadingPOSIX.cpp:242 #28 0x00007f36cea8ee39 in start_thread (arg=<optimized out>) at pthread_create.c:444 #29 0x00007f36ceb16cc4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
Attachments
Full backtrace (40.67 KB, text/plain)
2023-08-31 07:03 PDT, Michael Catanzaro 		no flags
Details
View All Add attachment proposed patch, testcase, etc.
Michael Catanzaro
Comment 1 2023-08-31 07:08:09 PDT
For fun, 18446744073709551615 bytes is roughly 16,777,216 TiB. I don't have that much RAM. :(
Michael Catanzaro
Comment 2 2023-08-31 07:09:38 PDT
> Notably, we tried to memcpy 18446744073709551615 (2^64-1) bytes, which is not good. Well, actually, that's what WebKit thinks is the size of the record (see frame 5), but the memcpy is only __len=8. Oops, sorry.
Michael Catanzaro
Comment 3 2023-08-31 11:21:32 PDT
Hm, I just hit a third network process crash today, plus one more yesterday, which was the first time I ever hit this crash. But there has been no WebKitGTK update since August 19 (2.41.91). That is suspicious; I would normally expect a crash that occurs with such high frequency to be more clearly associated with a particular update. All three crashes today occurred when using https://discourse.gnome.org/
Michael Catanzaro
Comment 4 2023-09-01 09:49:55 PDT
I hit two more network process crashes since my previous comment, again both on discourse.gnome.org, so this is happening often enough that I would *expect* to eventually figure out how to reproduce it. The crash seems to occur when writing comments, but so far it only happens when I'm not trying to reproduce and never when I am. If I could reproduce, then I can add prints to verify where things are going wrong....
Radar WebKit Bug Importer
Comment 5 2023-09-07 07:03:14 PDT
<rdar://problem/115113736>
Michael Catanzaro
Comment 6 2023-12-04 13:28:27 PST
Hit again today. I somehow got a better backtrace this time and unfortunately it is a buffer overread. In frame 6, we have: (gdb) print fileDatas $2 = WTF::Vector of length 0, capacity 0 (gdb) print blobDatas $3 = WTF::Vector of length 0, capacity 0 So recordInfos is also required to be 0, but: (gdb) print recordInfos $1 = WTF::Vector of length 1, capacity 1 = {{key = {m_partition = "record", m_type = "c13dab6b-1248-4ed8-818a-66e7d5c489f4", m_identifier = "1e1f03fa-d804-45c7-b65f-ee68cd9a0421", m_range = "(null)", m_hash = {_M_elems = ",H\023\214~\362\230\027\216Έ\256LH}\347\357/\214D"}, m_partitionHash = {_M_elems = "\"%7H\237\262Y\343\"(&1\201\025\366\366\3776\\\230"}}, insertionTime = 23529902.664000001, identifier = 43, updateResponseCounter = 0, size = 90, url = { m_string = "https://globalnews.ca/wp-content/themes/shaw-globalnews/assets/dist/images/pattern-greystripe.png", m_isValid = 1, m_protocolIsInHTTPFamily = 1, m_hasOpaquePath = 0, m_portLength = 0, static maxPortLength = 7, static maxSchemeLength = 67108863, m_schemeEnd = 5, m_userStart = 8, m_userEnd = 8, m_passwordEnd = 8, m_hostEnd = 21, m_pathAfterLastSlash = 75, m_pathEnd = 97, m_queryEnd = 97}, hasVaryStar = false, varyHeaders = {m_impl = {static smallMaxLoadNumerator = 3, static smallMaxLoadDenominator = 4, static largeMaxLoadNumerator = 1, static largeMaxLoadDenominator = 2, static maxSmallTableCapacity = 1024, static minLoad = 6, static tableSizeOffset = -1, static tableSizeMaskOffset = -2, static keyCountOffset = -3, static deletedCountOffset = -4, static metadataSize = 16, {m_table = 0x0, m_tableForLLDB = 0x0}}}}} There are assertions at the top of the function that would catch this in debug builds: ASSERT(recordInfos.size() == fileDatas.size()); ASSERT(recordInfos.size() == blobDatas.size()); but they are unfortunately not enabled in release builds. I'm not sure how this happened. Looking in CacheStorageDiskStore::readRecordsInternal, it really looks like the sizes should always match.
Michael Catanzaro
Comment 7 2024-01-22 11:03:03 PST
I just hit this crash 10 times in a row with WebKitGTK 2.43.3. i.e. the network process would restart itself, then immediately crash again. This time it was caused by one of the product pages under https://www.corsair.com/us/en/c/psu though I'm not sure exactly which one. After 10 network process crashes, the UI process crashed, and after restarting the crashes no longer occur.
Note You need to log in before you can comment on or make changes to this bug.