Page error in layer when dumping Windows 10 hashes

[arepi-nemui]

Describe the bug
After commit e5a5b89, volatility is no longer able to run various modules on a memory dump from a Windows 10 system, such as windows.hashdump.Hashdump and windows.info.Info.

Context
Volatility Version: Commit e5a5b89 and later
Operating System: Linux
Python Version: 3.12.3
Suspected Operating System: Windows 10
Command: python vol.py -f memory.dmp -vvv windows.hashdump.Hashdump

To Reproduce
Steps to reproduce the behavior:

git bisect start
git bisect bad develop
git bisect good 55dd39f2
git bisect run python vol.py -f memory.dmp windows.hashdump.Hashdump

Example output

python vol.py -f memory.dmp -vvv windows.hashdump.Hashdump
Volatility 3 Framework 2.7.1
INFO     volatility3.cli: Volatility plugins path: ['/home/ctf/volatility3/volatility3/plugins', '/home/ctf/volatility3/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/home/ctf/volatility3/volatility3/symbols', '/home/ctf/volatility3/volatility3/framework/symbols']
INFO     volatility3.framework.automagic: Detected a windows category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.windows: Detecting Self-referential pointer for recent windows
DEBUG    volatility3.framework.automagic.windows: DtbSelfRef64bit test succeeded at 0x1ad000
DEBUG    volatility3.framework.automagic.windows: DTB was found at: 0x1ad000
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name.memory_layer
DETAIL 1 volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Hashdump.kernel.layer_name.memory_layer.base_layer
DETAIL 1 volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
DETAIL 1 volatility3.framework.interfaces.configuration: TypeError - kernel_virtual_offset requirements only accept int type: None
DETAIL 1 volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
DETAIL 1 volatility3.framework.interfaces.configuration: TypeError - kernel_banner requirements only accept str type: None
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump.kernel
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Hashdump
DEBUG    volatility3.framework.automagic.stacker: physical_layer maximum_address: 1073750015
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['IntelLayer', 'WindowsCrashDump64Layer', 'FileLayer']
INFO     volatility3.framework.automagic: Running automagic: WinSwapLayers
INFO     volatility3.framework.automagic: Running automagic: KernelPDBScanner
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DETAIL 1 volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Hashdump.kernel.symbol_table_name
DEBUG    volatility3.framework.automagic.pdbscan: Kernel base determination - searching layer module list structure
DEBUG    volatility3.framework.automagic.pdbscan: Setting kernel_virtual_offset to 0xf8043b209000
DEBUG    volatility3.framework.symbols.windows.pdbutil: Using symbol library: ntkrnlmp.pdb/68A17FAF3012B7846079AEECDBE0A583-1
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder
INFO     volatility3.framework.automagic: Running automagic: KernelModule
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_ACCESS_STATE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_CPU_RATE_CONTROL
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_NET_RATE_CONTROL
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_JOB_NOTIFICATION_INFORMATION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_PSP_STORAGE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_KTMNOTIFICATION_PACKET
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_EXP_LICENSE_STATE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_DBGKP_ERROR_PORT
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_CI_NGEN_PATHS
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_EX_WNF_SUBSCRIPTION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_PO_PROCESS_ENERGY_CONTEXT
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_EPROCESS_QUOTA_BLOCK
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_PAGEFAULT_HISTORY
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_DEVICE_NODE_IOMMU_EXTENSION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_EVENT_CALLBACK_CONTEXT
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_SOFT_RESTART_CONTEXT
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_STACK_CACHE
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ACTIVATION_CONTEXT_DATA
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ASSEMBLY_STORAGE_MAP
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_ETW_PERFECT_HASH_FUNCTION
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_EX_TIMER
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_HAL_PMC_COUNTERS
DEBUG    volatility3.framework.symbols: Unresolved reference: symbol_table_name1!_SCSI_REQUEST_BLOCK

User    rid     lmhash  nthash


DEBUG    volatility3.cli: Traceback (most recent call last):
  File "/home/ctf/volatility3/volatility3/cli/__init__.py", line 469, in run
    renderer.render(grid)
  File "/home/ctf/volatility3/volatility3/cli/text_renderer.py", line 198, in render
    grid.populate(visitor, outfd)
  File "/home/ctf/volatility3/volatility3/framework/renderers/__init__.py", line 245, in populate
    for level, item in self._generator:
  File "/home/ctf/volatility3/volatility3/framework/plugins/windows/hashdump.py", line 570, in _generator
    bootkey = self.get_bootkey(syshive)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/plugins/windows/hashdump.py", line 359, in get_bootkey
    lsa = cls.get_hive_key(syshive, lsa_base)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/plugins/windows/hashdump.py", line 334, in get_hive_key
    result = hive.get_key(key)
             ^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/registry.py", line 174, in get_key
    root_node = self.get_node(self.root_cell_offset)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/registry.py", line 143, in get_node
    signature = cell.cast("string", max_length=2, encoding="latin-1")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/interfaces/objects.py", line 189, in cast
    return object_template(context=self._context, object_info=object_info)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/objects/templates.py", line 96, in __call__
    return self.vol.object_class(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/objects/__init__.py", line 352, in __new__
    cls._unmarshall(
  File "/home/ctf/volatility3/volatility3/framework/objects/__init__.py", line 202, in _unmarshall
    data = context.layers.read(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/interfaces/layers.py", line 638, in read
    return self[layer].read(offset, length, pad)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/linear.py", line 63, in read
    self._context.layers.read(layer, mapped_offset, mapped_length, pad)
  File "/home/ctf/volatility3/volatility3/framework/interfaces/layers.py", line 638, in read
    return self[layer].read(offset, length, pad)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/linear.py", line 45, in read
    for offset, _, mapped_offset, mapped_length, layer in self.mapping(
  File "/home/ctf/volatility3/volatility3/framework/layers/intel.py", line 295, in mapping
    for offset, size, mapped_offset, mapped_size, map_layer in self._mapping(
  File "/home/ctf/volatility3/volatility3/framework/layers/intel.py", line 351, in _mapping
    chunk_offset, page_size, layer_name = self._translate(offset)
                                          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/intel.py", line 503, in _translate
    return self._translate_swap(self, offset, self._bits_per_register // 2)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/intel.py", line 450, in _translate_swap
    return super()._translate(offset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/intel.py", line 155, in _translate
    entry, position = self._translate_entry(offset)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ctf/volatility3/volatility3/framework/layers/intel.py", line 184, in _translate_entry
    raise exceptions.PagedInvalidAddressException(
volatility3.framework.exceptions.PagedInvalidAddressException: Entry outside virtual address range: 0x55fe001

Volatility was unable to read a requested page:
Page error 0xffffc60795389024 in layer layer_name_Process92_1 (Entry outside virtual address range: 0x55fe001)

        * Memory smear during acquisition (try re-acquiring if possible)
        * An intentionally invalid page lookup (operating system protection)
        * A bug in the plugin/volatility3 (re-run with -vvv and file a bug)

No further results will be produced

Additional information
memory.dmp: https://www.mediafire.com/file/c0v7xmu0f6kq2lr/memory.dmp.gz/file

[ikelos]

This was a commit made by @gcmoreira , hopefully he can diagnose what's now happening here. I've reverted it for now, pending review to see what problem it was causing. It might be it was too restrictive, or there may be more off-by-one issues in the codebase.

Thanks for providing the memory image that caused the problems, that should help a lot in discovering the issue...

[gcmoreira]

Hm that's weird, now I'm even more curious why the test cases didn't fail. @ikelos Any idea?

@ikelos and @arepi-nemui
Anyway, this is why I thought that if is dead code, not matter what the values are.

>>> 1 > 5 > 10
False
>>> 1 > 100 > 10
False
>>> 1 > -1 > 10
False

And this is what we want to detect to raise the exception:

>>> not (1 <= 5 <= 10)
False
>>> not (1 <= 100 <= 10)
True
>>> not (1 <= -1 <= 10)
True

So, I think here there is bug somewhere else that was never detected because of that funny if.

Setting a breakpoint in that line I got this:

>> hex(self.minimum_address), hex(offset), hex(self.maximum_address)
('0x0', '0xffffc60795389024', '0xffffffffffff')

So, @ikelos the fix I proposed was actually correct.

The problem is that offset is a canonical address. If instead we use the line below it works as expected:

>> not (self.minimum_address <= self.decanonicalize(offset) <= self.maximum_address)
False

Unfortunately, it seems that _translate_entry is called with both canonical and not canonical addresses.

It looks like this will work for all cases, but it needs to be tested properly.

>> not (self.minimum_address <= offset <= self.canonicalize(self.maximum_address))
False

I will need more time to figure out how to properly fix this. Feel free to continue investigating this if you'd like.

[atcuno]

if translate_entry is the broken code path for all cases then it should either normalize offsets sent in relative t self.maximum_address or the form Gus sent with self.canonicalize(self.maximum_address) should be used. It feels like offset should be normalized by translate_entry before its used versus trying to enforce a long term meaning to self.maximum_address.