perfparser raises SIGABRT in PerfUnwind::unwindStack

[GitMensch]

Describe the bug
SIGABRT in https://github.com/KDAB/perfparser/blob/b455d4092aef0861b52a75bb818e34af6cb6ee32/app/perfunwind.cpp#L548-L574

To Reproduce
Steps to reproduce the behavior:

1. Open a huge uncompressed perf data (1,25 GB / 2,3 GB) created by sudo perf record -o perf.data.6-50 --call-graph dwarf,50 -a --uid somuser -- sleep 30m, which was aborted by CTRL+C because of size issues (no max size option on this older perf)
2. See error after 23% / 34% parsing.

Additional info

• files with same and bigger size open well
• files with same and bigger frequency recorded open well
• all of the files chrashing hotspot-perfparser open fine with perf report -i
• of course: I can't share that perf.data file :-(

Expected behavior

1. perf.data which can be opened by perf report should be able to be opened by hotspot-perfparser

2. Check if it is possible to circumvent this specific error.

3. Improved error message in the GUI, when possible

   The hotsport-perfparser binary exited with code 6 (invalid perf data file).

   -->The question: "What was invalid" would need more communication from perfparser to hotspot than a signal.

GDB info
The stacktrace is the same in all cases, so I did not check the objects on any abort but the first).

#0  0x00007fcdd4aea387 in raise () from /usr/lib64/libc.so.6
#1  0x00007fcdd4aeba78 in abort () from /usr/lib64/libc.so.6
#2  0x00007fcdd4ae31a6 in __assert_fail_base () from /usr/lib64/libc.so.6
#3  0x00007fcdd4ae3252 in __assert_fail () from /usr/lib64/libc.so.6
#4  0x000000000045ccee in new_unwound ()
#5  0x000000000045e018 in handle_cfi ()
#6  0x000000000045e469 in __libdwfl_frame_unwind ()
#7  0x000000000045c770 in dwfl_thread_getframes ()
#8  0x000000000045c53a in get_one_thread_cb ()
#9  0x000000000045c5d3 in dwfl_getthreads ()
#10 0x000000000045cafb in dwfl_getthread_frames ()
#11 0x0000000000421747 in PerfUnwind::unwindStack (this=0x7ffcdd0706e0) at /__w/hotspot/hotspot/3rdparty/perfparser/app/perfunwind.cpp:554
#12 0x0000000000425bcc in PerfUnwind::analyze (this=0x7ffcdd0706e0, sample=...) at /__w/hotspot/hotspot/3rdparty/perfparser/app/perfunwind.cpp:743
#13 0x00000000004269d8 in PerfUnwind::flushEventBuffer (this=0x7ffcdd0706e0, desiredBufferSize=324384) at /usr/include/QtCore/qlist.h:151
#14 0x000000000041c07b in PerfData::processEvents (this=0x7ffcdd070550, stream=...) at /__w/hotspot/hotspot/3rdparty/perfparser/app/perfdata.cpp:242
#15 0x000000000041ced3 in PerfData::doRead (this=0x7ffcdd070550) at /__w/hotspot/hotspot/3rdparty/perfparser/app/perfdata.cpp:413
#16 0x000000000041d212 in PerfData::read (this=0x7ffcdd070550) at /__w/hotspot/hotspot/3rdparty/perfparser/app/perfdata.cpp:429
#17 0x00007fcdd66af16d in ?? () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#18 0x0000000000415128 in PerfHeader::read (this=0x7ffcdd0704d0) at /__w/hotspot/hotspot/3rdparty/perfparser/app/perfheader.cpp:122
#19 0x00007fcdd66a579e in QObject::event(QEvent*) () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#20 0x00007fcdd667aa67 in QCoreApplication::notifyInternal2(QObject*, QEvent*) () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#21 0x00007fcdd667d837 in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*) () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#22 0x00007fcdd66cf3d3 in ?? () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#23 0x00007fcdd236e119 in g_main_context_dispatch () from /tmp/.mount_hotspoomGEza//usr/lib/libglib-2.0.so.0
#24 0x00007fcdd236e478 in g_main_context_iterate.isra () from /tmp/.mount_hotspoomGEza//usr/lib/libglib-2.0.so.0
#25 0x00007fcdd236e52c in g_main_context_iteration () from /tmp/.mount_hotspoomGEza//usr/lib/libglib-2.0.so.0
#26 0x00007fcdd66ceaa4 in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#27 0x00007fcdd667961b in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#28 0x00007fcdd668126c in QCoreApplication::exec() () from /tmp/.mount_hotspoomGEza//usr/lib/libQt5Core.so.5
#29 0x000000000040fb58 in main (argc=<optimized out>, argv=<optimized out>) at /__w/hotspot/hotspot/3rdparty/perfparser/app/main.cpp:432
(gdb) f 11
#11 0x0000000000421747 in PerfUnwind::unwindStack (this=0x7ffcdd0706e0) at /__w/hotspot/hotspot/3rdparty/perfparser/app/perfunwind.cpp:554
554     /__w/hotspot/hotspot/3rdparty/perfparser/app/perfunwind.cpp: No such file or directory.
(gdb) p this
$1 = (PerfUnwind * const) 0x7ffcdd0706e0
(gdb) p *this
$2 = {<QObject> = {<No data fields>}, static staticMetaObject = {d = {superdata = {direct = 0x7fcdd6964e20 <QObject::staticMetaObject>}, stringdata = 0x488d00 <qt_meta_stringdata_PerfUnwind>,
      data = 0x488c80 <qt_meta_data_PerfUnwind>, static_metacall = 0x412640 <PerfUnwind::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)>, relatedMetaObjects = 0x0,
      extradata = 0x0}}, static s_kernelPid = -1, m_currentUnwind = {stackValues = {{d = 0x4551650, e = 0x4551650}}, frames = {d = 0x6dd88d0}, unwind = 0x7ffcdd0706e0, sample = 0xca10a30,
    maxFrames = 1024, firstGuessedFrame = -1, isInterworking = false}, m_output = 0xcec0e0, m_offlineCallbacks = {find_elf = 0x45b1f0 <dwfl_build_id_find_elf>,
    find_debuginfo = 0x41e040 <find_debuginfo(Dwfl_Module*, void**, char const*, Dwarf_Addr, char const*, char const*, GElf_Word, char**)>,
    section_address = 0x4586f0 <dwfl_offline_section_address>, debuginfo_path = 0x7ffcdd070748}, m_debugInfoPath = 0xcec180 "://usr/lib/debug:/root/.debug:.debug:::/",
  m_architecture = PerfRegisterInfo::ARCH_X86, m_systemRoot = {static null = {<No data fields>}, d = 0xcea250}, m_extraLibsPath = {static null = {<No data fields>},
    d = 0x7fcdd6760a40 <QArrayData::shared_null>}, m_appPath = {static null = {<No data fields>}, d = 0x7fcdd6760a40 <QArrayData::shared_null>}, m_debugPath = {
    static null = {<No data fields>}, d = 0xcec6b0}, m_kallsymsPath = {static null = {<No data fields>}, d = 0xcec7f0}, m_ignoreKallsymsBuildId = false,
  m_sampleBuffer = {<QListSpecialMethods<PerfRecordSample>> = {<No data fields>}, {p = {static shared_null = {ref = {atomic = {_q_value = {<std::__atomic_base<int>> = {
                  static _S_alignment = 4, _M_i = -1}, <No data fields>}}}, alloc = 0, begin = 0, end = 0, array = {0x0}}, d = 0x6d50920}, d = 0x6d50920}},
  m_mmapBuffer = {<QListSpecialMethods<PerfRecordMmap>> = {<No data fields>}, {p = {static shared_null = {ref = {atomic = {_q_value = {<std::__atomic_base<int>> = {static _S_alignment = 4,
                  _M_i = -1}, <No data fields>}}}, alloc = 0, begin = 0, end = 0, array = {0x0}}, d = 0x7fcdca7f7010}, d = 0x7fcdca7f7010}},
  m_taskEventsBuffer = {<QListSpecialMethods<PerfUnwind::TaskEvent>> = {<No data fields>}, {p = {static shared_null = {ref = {atomic = {_q_value = {<std::__atomic_base<int>> = {
                  static _S_alignment = 4, _M_i = -1}, <No data fields>}}}, alloc = 0, begin = 0, end = 0, array = {0x0}}, d = 0x25434b0}, d = 0x25434b0}}, m_symbolTables = {{d = 0x33361b0,
      e = 0x33361b0}}, m_kallsyms = {m_entries = {d = 0x51f77b0}, m_errorString = {static null = {<No data fields>}, d = 0x7fcdd6760a40 <QArrayData::shared_null>}}, m_addressCache = {
    m_cache = {{d = 0x45aadf0, e = 0x45aadf0}}, m_symbolCache = {{d = 0x45c7270, e = 0x45c7270}}}, m_tracingData = {m_size = 0, m_version = {d = 0x7fcdd6760a40 <QArrayData::shared_null>},
    m_bigEndian = false, m_fileLongSize = false, m_filePageSize = 0, m_eventFormats = {{d = 0x7fcdd67614c0 <QHashData::shared_null>, e = 0x7fcdd67614c0 <QHashData::shared_null>}},
    m_headerFields = {d = 0x7fcdd6760a40 <QArrayData::shared_null>}, m_ftracePrintk = {{d = 0x7fcdd67614c0 <QHashData::shared_null>, e = 0x7fcdd67614c0 <QHashData::shared_null>}},
    m_savedCmdlines = {d = 0x7fcdd6760a40 <QArrayData::shared_null>}}, m_strings = {{d = 0xee32d0, e = 0xee32d0}}, m_locations = {{d = 0x457f4b0, e = 0x457f4b0}}, m_symbols = {{d = 0x457f3e0,
      e = 0x457f3e0}}, m_attributeIds = {{d = 0xceca80, e = 0xceca80}}, m_attributes = {d = 0xee31d0}, m_buildIds = {{d = 0xfbb250, e = 0xfbb250}}, m_lastEventBufferSize = 33554432,
  m_maxEventBufferSize = 1073741824, m_targetEventBufferSize = 0, m_eventBufferSize = 351976, m_timeOrderViolations = 0, m_lastFlushMaxTime = 3772865635007259,
  m_byteOrder = QSysInfo::LittleEndian, m_stats = {numSamples = 0, numMmaps = 0, numRounds = 0, numBufferFlushes = 0, numTimeViolatingSamples = 0, numTimeViolatingMmaps = 0,
    numSamplesInRound = 0, numMmapsInRound = 0, numTaskEventsInRound = 0, maxSamplesPerRound = 0, maxMmapsPerRound = 0, maxTaskEventsPerRound = 0, maxSamplesPerFlush = 0,
    maxMmapsPerFlush = 0, maxTaskEventsPerFlush = 0, maxBufferSize = 0, maxTotalEventSizePerRound = 0, maxTime = 0, maxTimeBetweenRounds = 0, maxReorderTime = 0, lastRoundTime = 0,
    totalEventSizePerRound = 0, enabled = false}}
(gdb)

(gdb) p dwfl
$3 = (Dwfl *) 0xc945f90
(gdb) p *dwfl
$4 = <incomplete type>
(gdb) p m_currentUnwind
$4 = {stackValues = {{d = 0x4551650, e = 0x4551650}}, frames = {d = 0x6dd88d0}, unwind = 0x7ffcdd0706e0, sample = 0xca10a30, maxFrames = 1024, firstGuessedFrame = -1, isInterworking = false}
(gdb) p m_currentUnwind->sample
$5 = (const PerfRecordSample *) 0xca10a30
(gdb) p *m_currentUnwind->sample
$6 = {<PerfRecord> = {m_header = {type = 9, misc = 1, size = 344}, m_sampleId = {m_pid = 1578, m_tid = 1578, m_time = 3772865635007259, m_id = 0, m_streamId = 0, m_cpu = 0, m_res = 0, {
        m_ignoredDuplicateId = 45359, m_sampleType = 45359}}}, m_readFormat = 4, m_registerMask = 16715775, m_ip = 18446744072601946048, m_addr = 0, m_period = 250000, m_timeEnabled = 0,
  m_timeRunning = 0, m_registerAbi = 2, m_weight = 0, m_dataSrc = 84410401, m_transaction = 13415346,
  m_readFormats = {<QListSpecialMethods<PerfRecordSample::ReadFormat>> = {<No data fields>}, {p = {static shared_null = {ref = {atomic = {_q_value = {<std::__atomic_base<int>> = {
                  static _S_alignment = 4, _M_i = -1}, <No data fields>}}}, alloc = 0, begin = 0, end = 0, array = {0x0}}, d = 0x7fcdd6761550 <QListData::shared_null>},
      d = 0x7fcdd6761550 <QListData::shared_null>}}, m_callchain = {<QListSpecialMethods<unsigned long long>> = {<No data fields>}, {p = {static shared_null = {ref = {atomic = {
              _q_value = {<std::__atomic_base<int>> = {static _S_alignment = 4, _M_i = -1}, <No data fields>}}}, alloc = 0, begin = 0, end = 0, array = {0x0}}, d = 0x1048afc0},
      d = 0x1048afc0}}, m_rawData = {d = 0x7fcdd6760a40 <QArrayData::shared_null>}, m_branchStack = {<QListSpecialMethods<PerfRecordSample::BranchEntry>> = {<No data fields>}, {p = {
        static shared_null = {ref = {atomic = {_q_value = {<std::__atomic_base<int>> = {static _S_alignment = 4, _M_i = -1}, <No data fields>}}}, alloc = 0, begin = 0, end = 0, array = {
            0x0}}, d = 0x7fcdd6761550 <QListData::shared_null>}, d = 0x7fcdd6761550 <QListData::shared_null>}}, m_registers = {<QListSpecialMethods<unsigned long long>> = {<No data fields>}, {
      p = {static shared_null = {ref = {atomic = {_q_value = {<std::__atomic_base<int>> = {static _S_alignment = 4, _M_i = -1}, <No data fields>}}}, alloc = 0, begin = 0, end = 0, array = {
            0x0}}, d = 0xed288d0}, d = 0xed288d0}}, m_userStack = {d = 0xa7e8110}}
(gdb) p frameCallback
$7 = {int (Dwfl_Frame *, void *)} 0x420fb0 <frameCallback(Dwfl_Frame*, void*)>

Version Info (please complete the following information):

• Linux Kernel version: 3.10.0-1160.31.1.el7.x86_64
• perf version: 3.10.0-1160.el7.x86_64.debug
• hotspot version: hotspot-v1.3.0-364-g322803e-x86_64.AppImage

[milianw]

this report is not actionable without a reliable way to reproduce this. Furthermore, the crash happens within elfutils. What assertion do you see? With that you might be able to report this upstream to the elfutils developers.

[GitMensch]

Furthermore, the crash happens within elfutils.

Not sure if this is an elfutils problem or an issue with how elfutils is used ;-)

What assertion do you see?

Can you please add a non-stripped elfutils to the AppImage and specify the version used? This would allow me to check what actually is happening.

[GitMensch]

ping @milianw on the AppImage - could you create that with non-stripped elfutils?

[milianw]

I don't have time for that in the next 1-2 weeks I'm afraid @GitMensch - but it should be pretty easy for you to hack the docker file. Ideally we would extract the debug symbols for all things we put into the appimage and then upload that separately as an artifact. That way we can then manually debug issues.

Patches welcome!

[GitMensch]

docker and GH actions are both not in my skill stack, will wait for 2-8 weeks ;-)

[milianw]

I got some time for this after all today and set everything up: https://github.com/KDAB/hotspot#debugging-the-appimage

Note that for elfutils debug symbols itself, you won't need most of this - just use the official centos debuginfod, that should be enough to get you debug symbols for that.

[GitMensch]

Hm, shouldn't the docker package be available via https://github.com/orgs/KDAB/packages?tab=packages&q=kdesrc, too? I don't have ghcr.io setup...

... tested without that - just to find that the GDB instances are either too old or:

(gdb) set debuginfod enabled on
Support for debuginfod is not compiled into GDB.

Installation of an updated GDB likely will take a while :-/

I guess I could download the necessary debuginfo manually, but no clue how to do that.

Side note: You may want to add https://debuginfod.elfutils.org in the noted URLs, it seems it would be reasonable to use it as it federates to centos and other servers.

From inspecting the current source (which elfutils is contained in the appimage?) there are only two asserts... https://github.com/evverx/elfutils/blob/0bf61bc17ebf002dbf58fc5540bfb22343e7b354/libdwfl/frame_unwind.c#L499-L520

[GitMensch]

Note that for elfutils debug symbols itself, you won't need most of this - just use the official centos debuginfod, that should be enough to get you debug symbols for that.

Are you sure? The appimage uses centos7 (I think only centos8+ has debuginfod) and also elfutils seems to be statically linked in the appimage, not sure if debuginfod can help here for self-built binaries.

Also see #560.

[milianw]

please create a new issue instead of commenting on old ones

[GitMensch]

This isn't a new issue and the "old one" is still open.

This issue is about a SIGABRT in PerfUnwind, as seen in the AppImage.
To debug AppImage issues you've added debug infos - but did not include the ones from elfutils, where that crash happens, so I've commented here.

Also you've told me to use the debuginfod option to get the missing symols - but I question if there is a debuginfod for the elfutils used for the appimage - as that is CentOS7 - and also that it would be used because elfutils are statically linked in (both could be me confusing things).

Don't you agree that this issue is the right to comment about those to issues? What would a new issue about that look like ?!?