Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Misc issues - RC testing #218

Closed
28 of 29 tasks
joachimmetz opened this issue Jun 2, 2015 · 28 comments
Closed
28 of 29 tasks

Misc issues - RC testing #218

joachimmetz opened this issue Jun 2, 2015 · 28 comments
Assignees
Labels
bug enhancement New or improved functionality

Comments

@joachimmetz
Copy link
Member

This is a place holder issue for issues uncovered while pre RC testing see below for detailed information.

  • test for psort time slice

Completed

Won't fix (Unlikely to be fixed before release)

@joachimmetz joachimmetz self-assigned this Jun 2, 2015
@joachimmetz
Copy link
Member Author

Bug fixed, some after care needed

[WARNING] [firefox_old_cache] did not explicitly close file-object for path specification:

Fix and clean up in: https://codereview.appspot.com/234700044/

More clean up needed reminder added to: #160

@joachimmetz
Copy link
Member Author

Open - WIP

Traceback (most recent call last):
  File "plaso/engine/worker.py", line 169, in _ParseFileEntryWithParser
    parser_object.UpdateChainAndParse(self._parser_mediator)
  File "plaso/parsers/interface.py", line 72, in UpdateChainAndParse
    self.Parse(parser_mediator, **kwargs)
  File "plaso/parsers/interface.py", line 256, in Parse
    self.ParseFileObject(parser_mediator, file_object, **kwargs)
  File "plaso/parsers/custom_destinations.py", line 142, in ParseFileObject
    lnk_file_object = resolver.Resolver.OpenFileObject(path_spec)
  File "dfvfs/resolver/resolver.py", line 113, in OpenFileObject
    file_object.open(path_spec=path_spec)
  File "dfvfs/file_io/file_io.py", line 75, in open
    self._Open(path_spec=path_spec, mode=mode)
  File "dfvfs/file_io/data_range_io.py", line 90, in _Open
    path_spec.parent, resolver_context=self._resolver_context)
  File "dfvfs/resolver/resolver.py", line 113, in OpenFileObject
    file_object.open(path_spec=path_spec)
  File "dfvfs/file_io/file_io.py", line 75, in open
    self._Open(path_spec=path_spec, mode=mode)
  File "dfvfs/file_io/tsk_file_io.py", line 62, in _Open
    self._tsk_file = file_entry.GetTSKFile()
AttributeError: 'NoneType' object has no attribute 'GetTSKFile'

So file entry is None here.

    file_entry = self._file_system.GetFileEntryByPathSpec(path_spec)
    self._tsk_file = file_entry.GetTSKFile()

Ddetermine which path spec is causing this

@joachimmetz joachimmetz changed the title Misc issues - pre RC testing Misc issues - RC testing Jun 8, 2015
@joachimmetz
Copy link
Member Author

Fixed

[winreg] did not explicitly close file-object for file: TSK:/Users/$USERNAME/AppData/Local/Microsoft/Windows/UsrClass.dat

@kiddinn
Copy link
Member

kiddinn commented Jun 8, 2015

Test again with dfvfs changes
https://codereview.appspot.com/246090043/

Still some issues with file handles not properly closed. This is taken from one of the Mac test images:

    2015-06-08 19:59:03,495 [WARNING] (Worker_07 ) PID:20048 <worker> File-object not explicitly closed for file: TSK:/.DocumentRevisions-V100/PerUID/501/5/com.apple.documentVersions/A66C960A-4399-4DBA-89FF-4BE6FF9E5A74.pdf

These errors (both the Mac OS X ones) appear quite frequently in all Mac OS X parsing, need to go over the Mac OS X parsers and make sure the explicitly close file handles.

All Mac enabled parsers still seem to have an explicit close operation (most inherit from SingleFileBaseParser).

@kiddinn
Copy link
Member

kiddinn commented Jun 8, 2015

Test again with timelib changes
https://codereview.appspot.com/245190043/

Another random error in logs (408 challenge image):

<timelib> Unable to copy .google.com /search/ to a datetime object with error: unknown string format

Unknown origin, need to track that one down.

Seeing few of these:

2015-06-10 18:02:26,827 ERROR] (Worker_12 ) PID:23453 <mediator> [mcafee_protection] unable to parse file: TSK:/Users/Susan Storm Richards/AppData/LocalLow/Google/FastSearch/dictionaries/dictionaries.txt with error: Unable to parse time string: .google.com /search/

@joachimmetz
Copy link
Member Author

[winreg/userassist] unable to parse file: TSK:/Users/$USERNAME/NTUSER.DAT with error: Unsupported value data size: 1612

@kiddinn
Copy link
Member

kiddinn commented Jun 9, 2015

Fixed
https://codereview.appspot.com/246830043/

Using --partition 2 gets me the traceback:

Traceback (most recent call last):
  File "/usr/bin/log2timeline.py", line 611, in <module>
    if not Main():
  File "/usr/bin/log2timeline.py", line 597, in Main
    tool.ProcessSources()
  File "/usr/bin/log2timeline.py", line 531, in ProcessSources
    self.ScanSource(self._front_end)
  File "/usr/lib/python2.7/dist-packages/plaso/cli/storage_media_tool.py", line 724, in ScanSource
    location = u'/{0:s}'.format(partition_identifier)
ValueError: Unknown format code 's' for object of type 'int'

Changing this to a decimal value now reveals:

2015-06-09 15:46:46,665 [WARNING] (MainProcess) PID:15279 <log2timeline> Invalid or missing volume scan node.

@kiddinn
Copy link
Member

kiddinn commented Jun 9, 2015

Fixed

Status view = window on Windows has minor issues with extra characters added:
Version:
←[2J←[Hplaso - log2timeline version 1.3.0_20150607
Column:
←[1mIdentifier PID Status Events File←[0m

@kiddinn
Copy link
Member

kiddinn commented Jun 9, 2015

Fixed
https://codereview.appspot.com/246830043/

If you run the tool against a disk image that has VSS stores on them and you supply the --no-vss parameter nothing will get parsed:

 <log2timeline> No supported file system found in source.

the _ScanVolume inside the storage_media_tool is not handling this properly.

If there is a VSS scan node the TSK current volume scan node is not moved to the same level as the VSS one is in the case where VSS is not processed.

@kiddinn
Copy link
Member

kiddinn commented Jun 10, 2015

Open

Running against nfury image I see a lot of:

2015-06-09 20:49:12,072 **ERROR** (Worker_03 ) PID:44042 <mediator> **chrome_cache** unable to parse file: TSK:/Users/nfury/AppData/Local/Google/Chrome/User Data/Default/Cache/index with error: Missing data block file: data_1


2015-06-09 20:49:12,076 **ERROR** (Worker_03 ) PID:44042 <mediator> **chrome_cache** unable to parse file: TSK:/Users/nfury/AppData/Local/Google/Chrome/User Data/Default/Cache/index with error: Cache address: 0xa1010dde missing data file.

@kiddinn
Copy link
Member

kiddinn commented Jun 11, 2015

Test again with dfvfs fixes
https://codereview.appspot.com/246090043/

One of the Mac test images:

2015-06-11 20:34:20,397 [WARNING] (Worker_14 ) PID:18236 <worker> Unhandled exception while processing path spec: type: OS, location: /<PATH>.E01
type: EWF
type: TSK_PARTITION, location: /p2, part index: 5, start offset: 0x0c805000
type: TSK, inode: 542723, location: /System/Library/Caches/com.apple.kext.caches/Startup/KextPropertyValues_OSBundleHelper_x86_64.plist.gz
type: GZIP
.
2015-06-11 20:34:20,397 [ERROR] (Worker_14 ) PID:18236 <worker> Maximum number of cached values reached.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/plaso/engine/worker.py", line 417, in _ProcessPathSpec
    path_spec, resolver_context=self._resolver_context)
  File "/usr/lib/python2.7/dist-packages/dfvfs/resolver/resolver.py", line 53, in OpenFileEntry
    path_spec, resolver_context=resolver_context)
  File "/usr/lib/python2.7/dist-packages/dfvfs/resolver/resolver.py", line 163, in OpenFileSystem
    file_system.Open(path_spec=path_spec)
  File "/usr/lib/python2.7/dist-packages/dfvfs/vfs/file_system.py", line 234, in Open
    self._resolver_context.CacheFileSystem(path_spec, self)
  File "/usr/lib/python2.7/dist-packages/dfvfs/resolver/context.py", line 62, in CacheFileSystem
    self._file_system_cache.CacheObject(identifier, file_system)
  File "/usr/lib/python2.7/dist-packages/dfvfs/resolver/cache.py", line 83, in CacheObject
    raise errors.CacheFullError(u'Maximum number of cached values reached.')
CacheFullError: Maximum number of cached values reached.

@kiddinn
Copy link
Member

kiddinn commented Jun 11, 2015

Won't fix for now

This message is repeated quite frequently, sometimes as:
2015-06-11 16:38:08,563 [WARNING] (Worker_14 ) PID:18236 <worker> File-object not explicitly closed for file: GZIP:/.fseventsd/000000000006ed2b

Addition: So this file contains gzip compressed data but not complete. The issue here is greedy parsing approach and hard to detect certain file types. A fix could be to excluded certain files from parsing.

@kiddinn
Copy link
Member

kiddinn commented Jun 11, 2015

These appear to be TSK HFS bugs

with error: pysigscan_scanner_scan_file_object: unable to scan file.
pysigscan_file_object_read_buffer: unable to read from file object with error: 'File_read_random:  (tsk3.c:494) Read error: Attribute not found in file (tsk_fs_attrlist_get: Attribute 4352 not found)'.
pysigscan_file_object_io_handle_read: unable to read from file object.
libbfio_handle_read_buffer: unable to read from handle.
libsigscan_scanner_scan_file_io_handle: unable to read buffer.
2015-06-11 16:38:08,096 [WARNING] (Worker_08 ) PID:18223 <worker> File-object not explicitly closed for file: TSK:/.DocumentRevisions-V100/PerUID/501/2/com.apple.documentVersions/3C43F7DD-4BC7-4B22-99FD-192CE21E21A8.rtf
2015-06-11 16:38:08,097 [WARNING] (Worker_08 ) PID:18223 <worker> Unable to process path spec: type: OS, location: <PATH>.E01
type: EWF
type: TSK_PARTITION, location: /p2, part index: 5, start offset: 0x0c805000
type: TSK, inode: 672541, location: /.DocumentRevisions-V100/PerUID/501/2/com.apple.documentVersions/3C43F7DD-4BC7-4B22-99FD-192CE21E21A8.rtf
 with error:
pysigscan_scanner_scan_file_object: unable to scan file.
pysigscan_file_object_read_buffer: unable to read from file object with error: 'File_read_random: (tsk3.c:494) Read error: Attribute not found in file (tsk_fs_attrlist_get: Attribute 4352 not found)'.
pysigscan_file_object_io_handle_read: unable to read from file object.
libbfio_handle_read_buffer: unable to read from handle.
libsigscan_scanner_scan_file_io_handle: unable to read buffer.

Addition: If you do a manual icat of the file TSK throws the same error, so this seems to be a bug in TSK HFS support Already filed: sleuthkit/sleuthkit#401

2015-06-11 08:43:38,546 [WARNING] (Worker_08 ) PID:8858 <worker> Unable to process path spec: type: OS, location: <PATH>.E01
type: EWF
type: TSK_PARTITION, location: /p2, part index: 5, start offset: 0x0c805000
type: TSK, inode: 450919, location: /Applications/iTunes.app/Contents/Resources/Japanese.lproj/Localizable.strings
 with error: File_read_random: (tsk3.c:494) Read error: Error reading image file ( zlib_inflate: zlib returned error -5 ((null))) ( hfs_attr_walk_special: zlib inflation (uncompression) failed)

This zlib error comes up quite a few times in the logs

Addition: This looks like a TSK HFS bug, filed sleuthkit/sleuthkit#471

@kiddinn
Copy link
Member

kiddinn commented Jun 12, 2015

Fixed
Minor CL for handling the ASL exception better: http://codereview.appspot.com/248800043

2015-06-11 10:03:06,913 [WARNING] (Worker_05 ) PID:8851 <worker> Unhandled exception while processing path spec: type: OS, location: <PATH>E01
type: EWF
type: TSK_PARTITION, location: /p2, part index: 5, start offset: 0x0c805000
type: TSK, inode: 676714, location: /private/var/log/asl/2013.12.17.U0.G80.asl
.
2015-06-11 10:03:06,914 [ERROR] (Worker_05 ) PID:8851 <worker> list index out of range
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/plaso/engine/worker.py", line 425, in _ProcessPathSpec
    self._ProcessFileEntry(file_entry)
  File "/usr/lib/python2.7/dist-packages/plaso/engine/worker.py", line 382, in _ProcessFileEntry
    self._ParseFileEntryWithParser(parser_object, file_entry)
  File "/usr/lib/python2.7/dist-packages/plaso/engine/worker.py", line 200, in _ParseFileEntryWithParser
    parser_object.UpdateChainAndParse(self._parser_mediator)
  File "/usr/lib/python2.7/dist-packages/plaso/parsers/interface.py", line 72, in UpdateChainAndParse
    self.Parse(parser_mediator, **kwargs)
  File "/usr/lib/python2.7/dist-packages/plaso/parsers/interface.py", line 256, in Parse
    self.ParseFileObject(parser_mediator, file_object, **kwargs)
  File "/usr/lib/python2.7/dist-packages/plaso/parsers/asl.py", line 205, in ParseFileObject
    event_object, offset = self.ReadAslEvent(file_object, offset)
  File "/usr/lib/python2.7/dist-packages/plaso/parsers/asl.py", line 347, in ReadAslEvent
    message = values[3]
IndexError: list index out of range

@kiddinn
Copy link
Member

kiddinn commented Jun 12, 2015

Work in progress
https://codereview.appspot.com/243260043/

2015-06-11 08:36:59,200 [WARNING] (Worker_12 ) PID:8866 <worker> Unable to process path spec: type: OS, location: /<PATH>.E01
type: EWF
type: TSK_PARTITION, location: /p2, part index: 5, start offset: 0x0c805000
type: TSK, inode: 531208, location: /Applications/iPhoto.app/Contents/Resources/Themes/Shared/Card/ModernOrnaments/Orn-Red.png
 with error: Invalid offset value less than zero.
    2015-06-08 22:42:36,682 [WARNING] (Worker_06 ) PID:20045 <worker> Unable to process path spec: type: OS, location: /<imagepath.e01>
    type: EWF
    type: TSK, inode: 290287, location: /System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/test/test_zipimport.pyc
     with error: Invalid offset value less than zero.

The invalid offset function is when a seek operation is attempted with a <0 offset value (after calculations based on the whence value)

type: TSK, inode: 655591, location: /Applications/iPhoto.app/Contents/Resources/Themes/Shared/Card/ModernOrnaments/Orn-Red.png
with error: Invalid offset value less than zero.

offset = -1520440957
self._current_offset = 37936
whence = 0 (SEEK_SET)

printf "0x%x\n" $(( -1520440957 - 37936 ))
0xffffffffa55f5553

Something is doing a negative seek set, seems to be openxml

2015-06-12 13:32:24,346 [WARNING] (Worker_01 ) PID:10797 <worker> IO error in openxml while parsing file (TSK:/Applications/iPhoto.app/Contents/Resources/Themes/Shared/Card/ModernOrnaments/Orn-Red.png) - Invalid offset value less than zero.

Looking at the code of that parser my best guess is that zipfile might be the culprit.

@rodgermoore
Copy link

Plaso 1.3.0 RC2 20150701
Running VT analysis plugin without providing VT api key yields a 403 client error from VT website. It is unclear to the user why this happens. If a user does not know he needs to provide a api key this can be very confusing as the error message is not clear. Imho a "missing api key error" should be raised first before even contacting VT servers.

@Onager
Copy link
Contributor

Onager commented Jul 2, 2015

Hmm, yes, that was the intention.

I'll look into while the error isn't being raised.
#249

-Daniel

On Thu, 2 Jul 2015 at 17:22 rodgermoore [email protected] wrote:

Plaso 1.3.0 RC2 20150701
Running VT analysis plugin without providing VT api key yields a 403
client error from VT website. It is unclear to the user why this happens.
If a user does not know he needs to provide a api key this can be very
confusing as the error message is not clear. Imho a "missing api key error"
should be raised first before even contacting VT servers.


Reply to this email directly or view it on GitHub
#218 (comment).

@joachimmetz
Copy link
Member Author

Moved remaining issues to separate issue trackers, closing this one.

kiddinn added a commit that referenced this issue Dec 31, 2015
@xhlika
Copy link

xhlika commented Oct 12, 2021

Was this " Add parser information to the "Unable to process path spec" bug if possible to make debugging easier" fixed? I am getting:

*********************************** Error: 0 ***********************************
           Message : Worker failed to process path specification
      Parser chain : None
                     C:\Users\xxxx\Desktop\xxx\xxx.E01
                   : type: EWF
                   : type: TSK, location: /
--------------------------------------------------------------------------------

*********************************** Error: 1 ***********************************
           Message : Worker failed to process path specification
      Parser chain : None
Path specification : type: OS, location:
                     C:\Users\xxx\xxx\xxx\xxx.E01
                   : type: EWF
                   : type: VSHADOW, location: /vss1, store index: 0
                   : type: TSK, location: /
-------------------------------------------------------------------------------- 

@joachimmetz
Copy link
Member Author

@xhlika why are you commenting on a closed issue that is not related to your issue? It also looks like you're using an old version of Plaso, so what version are you using?

@xhlika
Copy link

xhlika commented Oct 13, 2021

@joachimmetz sorry about that. This GitHub issue was the only google search result that was linked to my issue. I think the version I am is relatively new version (seems 2019): plaso - log2timeline version 20190331. Any idea where the issue lies, I am simply running plaso with this command:

log2timeline -z UTC --status_view window --parsers "win7,-filestat" C:\Users............\plaso_full.dump C:\Users...........\x.E01

@xhlika
Copy link

xhlika commented Oct 13, 2021

https://prnt.sc/1vyeksy it appears that after the "Main" worker, the next worker failed. The File column is saying "TSK:/".

Update: I just tried without VSS and it is working now. I really need the VSS to be there as well, any hint how to bypass the issue. There is only 1 vss (vss1) and when i am asked from log2timeline "please specify vss to consider" I choose "......... :1". Any hint what I am doing wrong?

@joachimmetz
Copy link
Member Author

https://prnt.sc/1vyeksy it appears that after the "Main" worker, the next worker failed. The File column is saying "TSK:/".

What does pinfo tell you more about the error? (as the message suggests)

See here for tips on troubleshooting https://plaso.readthedocs.io/en/latest/sources/Troubleshooting.html

plaso - log2timeline version 20190331

This version (from March 2019) is older than 6 months so we consider it out of date and strongly recommend to update to a more recent version, including dependencies. Note that issues get addressed continously

@xhlika
Copy link

xhlika commented Oct 13, 2021

What does pinfo tell you more about the error? (as the message suggests)

The first comment above was the output from pinfo (see above). The issue seems to be VSS. When I execute log2timeline and ignore the VSS, the parsing works (still ongoing).

If you have no idea why VSS is causing this, then I can maybe try the update but I wonder if it will help. The issue should have been fixed in 2015 (at least based on this issue).

@joachimmetz
Copy link
Member Author

Worker failed to process path specification is a generic catch all, typically debug logs contain more info.

The issue should have been fixed in 2015 (at least based on this issue).

what issue should be fixed in 2015? "Unable to process path spec" is not a bug its a warning that things could not be processed, there are many different reasons for this.

If you have no idea why VSS is causing this

I have insufficient information. You'll need to be more specific and provide the information outlined in https://plaso.readthedocs.io/en/latest/sources/Troubleshooting.html

@joachimmetz
Copy link
Member Author

The "bug" that this issue is referring to is that parser information was missing from the debug logs, but in your case an issue seems to happen outside of a parser. The 2 things are not related.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug enhancement New or improved functionality
Projects
None yet
Development

No branches or pull requests

5 participants