Created attachment 467506 [details] Full backtrace I've just seen this network process crash twice in the past hour. The short backtrace is pasted below. Notably, we tried to memcpy 18446744073709551615 (2^64-1) bytes, which is not good. Program terminated with signal SIGBUS, Bus error. #0 memcpy (__src=0x7f36cc554299, __len=8, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29 29 return __builtin___memcpy_chk (__dest, __src, __len, [Current thread is 1 (Thread 0x7f366bfff6c0 (LWP 31))] (gdb) bt #0 memcpy (__src=0x7f36cc554299, __len=8, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29 #1 WTF::Persistence::Decoder::decodeNumber<unsigned long>(std::optional<unsigned long>&) (this=0x7f366bffdcd0, optional=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/persistence/PersistentDecoder.cpp:85 #2 WTF::Persistence::Decoder::operator>>(std::optional<unsigned long>&) (this=0x7f366bffdcd0, result=std::optional<unsigned long> [no contained value]) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/persistence/PersistentDecoder.cpp:120 #3 0x00007f36cf806865 in WTF::Persistence::Decoder::operator>><WebCore::FetchOptionsDestination, (void*)0>(std::optional<WebCore::FetchOptionsDestination>&) (this=0x7f366bffdcd0, result=<optimized out>) at WTF/Headers/wtf/persistence/PersistentDecoder.h:75 #4 WebCore::FetchOptions::decodePersistent<WTF::Persistence::Decoder>(WTF::Persistence::Decoder&, WebCore::FetchOptions&) (decoder=..., options=...) at WebCore/PrivateHeaders/WebCore/FetchOptions.h:215 #5 0x00007f36cf854351 in WebKit::decodeRecordHeader(std::span<unsigned char const, 18446744073709551615ul>) (headerData=Python Exception <class 'gdb.error'>: value has been optimized out std::span of length 1967) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:234 #6 WebKit::readRecordInfoFromFileData(std::array<unsigned char, 8ul> const&, std::span<unsigned char const, 18446744073709551615ul>) (salt=..., fileData=Python Exception <class 'gdb.error'>: value has been optimized out ) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:294 #7 0x00007f36cf8536a7 in WebKit::CacheStorageDiskStore::readRecordFromFileData(std::span<unsigned char const, 18446744073709551615ul>, WTF::FileSystemImpl::MappedFileData&&) (this=<optimized out>, buffer=std::span of length 1528 = {...}, blobBuffer=...) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:305 #8 0x00007f36cf8627e6 in WebKit::CacheStorageDiskStore::readRecords(WTF::Vector<WebKit::CacheStorageRecordInformation, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> const&, WTF::CompletionHandler<void (WTF::Vector<std::optional<WebKit::CacheStorageRecord>, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_1::operator()<WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> >(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>) (this=0x7f36bd0b91a8, fileDatas=WTF::Vector of length 0, capacity 0, blobDatas=WTF::Vector of length 0, capacity 0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:412 #9 WTF::Detail::CallableWrapper<WebKit::CacheStorageDiskStore::readRecords(WTF::Vector<WebKit::CacheStorageRecordInformation, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc> const&, WTF::CompletionHandler<void (WTF::Vector<std::optional<WebKit::CacheStorageRecord>, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_1, void, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&>::call(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) (this=0x7f36bd0b91a0, in=<optimized out>, in=<optimized out>) at WTF/Headers/wtf/Function.h:53 #10 0x00007f36cf862547 in WTF::Function<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>::operator()(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) const (in=..., in=..., this=<optimized out>) at WTF/Headers/wtf/Function.h:82 #11 WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>::operator()(WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&) (this=<optimized out>, in=..., in=...) at WTF/Headers/wtf/CompletionHandler.h:75 #12 WebKit::CacheStorageDiskStore::readRecordsInternal(WTF::Vector<WTF::String, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow--Type <RET> for more, q to quit, c to continue without paging--c , 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_0::operator()()::{lambda()#1}::operator()() (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WebKit/NetworkProcess/storage/CacheStorageDiskStore.cpp:395 #13 WTF::Detail::CallableWrapper<WebKit::CacheStorageDiskStore::readRecordsInternal(WTF::Vector<WTF::String, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::CompletionHandler<void (WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&, WTF::Vector<WTF::FileSystemImpl::MappedFileData, 0ul, WTF::CrashOnOverflow, 16ul, WTF::FastMalloc>&&)>&&)::$_0::operator()()::{lambda()#1}, void>::call() (this=<optimized out>) at WTF/Headers/wtf/Function.h:53 #14 0x00007f36ce2f834b in WTF::Function<void ()>::operator()() const (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:82 #15 WTF::RunLoop::performWork() (this=0x7f36bd03c180) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/RunLoop.cpp:147 #16 0x00007f36ce3520fd in WTF::RunLoop::RunLoop()::$_0::operator()(void*) const (userData=0x7f366bffdcd0, userData@entry=0x7f36bd03c180, this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:80 #17 WTF::RunLoop::RunLoop()::$_0::__invoke(void*) (userData=0x7f366bffdcd0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:79 #18 0x00007f36ce351501 in WTF::RunLoop::$_0::operator()(_GSource*, int (*)(void*), void*) const (source=0x7f3660000dc0, callback=0x7f36ce3520f0 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f36bd03c180, this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:53 #19 WTF::RunLoop::$_0::__invoke(_GSource*, int (*)(void*), void*) (source=0x7f3660000dc0, callback=0x7f36ce3520f0 <WTF::RunLoop::RunLoop()::$_0::__invoke(void*)>, userData=0x7f36bd03c180) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:45 #20 0x00007f36ca8b6c97 in g_main_dispatch (context=context@entry=0x7f3660000b70) at ../glib/gmain.c:3476 #21 0x00007f36ca8b8da7 in g_main_context_dispatch_unlocked (context=0x7f3660000b70) at ../glib/gmain.c:4284 #22 g_main_context_iterate_unlocked (context=0x7f3660000b70, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib/gmain.c:4349 #23 0x00007f36ca8b9757 in g_main_loop_run (loop=0x7f3660000da0) at ../glib/gmain.c:4551 #24 0x00007f36ce351ad1 in WTF::RunLoop::run() () at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/glib/RunLoopGLib.cpp:108 #25 0x00007f36ce2fb9a7 in WTF::Function<void ()>::operator()() const (this=<optimized out>) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Function.h:82 #26 WTF::Thread::entryPoint(WTF::Thread::NewThreadContext*) (newThreadContext=0x7f36bd0340e0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/Threading.cpp:250 #27 0x00007f36ce35582d in WTF::wtfThreadEntryPoint(void*) (context=0x7f366bffdcd0) at /buildstream/gnome/sdk/webkitgtk-6.0.bst/Source/WTF/wtf/posix/ThreadingPOSIX.cpp:242 #28 0x00007f36cea8ee39 in start_thread (arg=<optimized out>) at pthread_create.c:444 #29 0x00007f36ceb16cc4 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100
For fun, 18446744073709551615 bytes is roughly 16,777,216 TiB. I don't have that much RAM. :(
> Notably, we tried to memcpy 18446744073709551615 (2^64-1) bytes, which is not good. Well, actually, that's what WebKit thinks is the size of the record (see frame 5), but the memcpy is only __len=8. Oops, sorry.
Hm, I just hit a third network process crash today, plus one more yesterday, which was the first time I ever hit this crash. But there has been no WebKitGTK update since August 19 (2.41.91). That is suspicious; I would normally expect a crash that occurs with such high frequency to be more clearly associated with a particular update. All three crashes today occurred when using https://discourse.gnome.org/
I hit two more network process crashes since my previous comment, again both on discourse.gnome.org, so this is happening often enough that I would *expect* to eventually figure out how to reproduce it. The crash seems to occur when writing comments, but so far it only happens when I'm not trying to reproduce and never when I am. If I could reproduce, then I can add prints to verify where things are going wrong....
<rdar://problem/115113736>
Hit again today. I somehow got a better backtrace this time and unfortunately it is a buffer overread. In frame 6, we have: (gdb) print fileDatas $2 = WTF::Vector of length 0, capacity 0 (gdb) print blobDatas $3 = WTF::Vector of length 0, capacity 0 So recordInfos is also required to be 0, but: (gdb) print recordInfos $1 = WTF::Vector of length 1, capacity 1 = {{key = {m_partition = "record", m_type = "c13dab6b-1248-4ed8-818a-66e7d5c489f4", m_identifier = "1e1f03fa-d804-45c7-b65f-ee68cd9a0421", m_range = "(null)", m_hash = {_M_elems = ",H\023\214~\362\230\027\216Έ\256LH}\347\357/\214D"}, m_partitionHash = {_M_elems = "\"%7H\237\262Y\343\"(&1\201\025\366\366\3776\\\230"}}, insertionTime = 23529902.664000001, identifier = 43, updateResponseCounter = 0, size = 90, url = { m_string = "https://globalnews.ca/wp-content/themes/shaw-globalnews/assets/dist/images/pattern-greystripe.png", m_isValid = 1, m_protocolIsInHTTPFamily = 1, m_hasOpaquePath = 0, m_portLength = 0, static maxPortLength = 7, static maxSchemeLength = 67108863, m_schemeEnd = 5, m_userStart = 8, m_userEnd = 8, m_passwordEnd = 8, m_hostEnd = 21, m_pathAfterLastSlash = 75, m_pathEnd = 97, m_queryEnd = 97}, hasVaryStar = false, varyHeaders = {m_impl = {static smallMaxLoadNumerator = 3, static smallMaxLoadDenominator = 4, static largeMaxLoadNumerator = 1, static largeMaxLoadDenominator = 2, static maxSmallTableCapacity = 1024, static minLoad = 6, static tableSizeOffset = -1, static tableSizeMaskOffset = -2, static keyCountOffset = -3, static deletedCountOffset = -4, static metadataSize = 16, {m_table = 0x0, m_tableForLLDB = 0x0}}}}} There are assertions at the top of the function that would catch this in debug builds: ASSERT(recordInfos.size() == fileDatas.size()); ASSERT(recordInfos.size() == blobDatas.size()); but they are unfortunately not enabled in release builds. I'm not sure how this happened. Looking in CacheStorageDiskStore::readRecordsInternal, it really looks like the sizes should always match.
I just hit this crash 10 times in a row with WebKitGTK 2.43.3. i.e. the network process would restart itself, then immediately crash again. This time it was caused by one of the product pages under https://www.corsair.com/us/en/c/psu though I'm not sure exactly which one. After 10 network process crashes, the UI process crashed, and after restarting the crashes no longer occur.