Detect files with incorrect/unexpected permissions and user/group own…

…er (#716)

Note that this artifact depends on Velocidex/velociraptor#3038. Here is
a verbatim copy of the artifact description:

This artifacts checks a number of files and directories to verify
whether
they have the expected owner, group owner and mode. A file with an
incorrect
owner may allow attackers to modify important system files. Similarly,
incorrect mode, like word-writable configuration or passwd/shadow files
may
also be signs of serious misconfiguration or signs of malicious
activity.

The parameter FilesToInspect contains lines of globs to search for. Each
line may specify expected user, expected group, expected file mode and
expected directory mode. It is very important that the order of the
lines
is in order of increasted specificity. For example:

```csv
/etc/*,root,root,644,755
/etc/?shadow*,,shadow,640,
```

Here, every file in the directory /etc is expected to be owned by
root:root
and have file permissions set to 644, and group permissions set to 755.
The files under /etc matching "?shadow*" are still expected to be owned
by
root, since the override for user is empty, but the group is no longer
expected to be "root", but "shadow". File permissions should be "640"
instead of "644". Note that this is an example and will most likely
return
hits, since a number of files in /etc have different owners and modes.

"User" may either be an integer (UID) or a string (username). "Group"
may be either an integer (GID) or a string (group name). The names for
all
UIDs and GIDs are looked up and displayed along with their IDs in the
result.

Modes may be specified in either octal numbers or strings.

Modes specified in octal numbers, e.g. 755, 640, 1777, are matched
using a regular expression, so that both "0640", "640" and "100640"
matches
"100640". An implicit anchor, '$', is used to match against the end of
the
octal mode string.

When using mode strings (NumericMode unchecked), modes take the format
"-rwx-r-x-r-x". Regex comparison is used, and an implicit '$' anchor
is inserted at the end of the string. String modes allows for verifying
only
certain bits of permissions, like ensuring that only the owner has write
access, no one has permission to execute, but read access is not
important:
"r.-.--.--". Or ensuring that SUID/GUID is not set. For finding files
specifically with SUID set, look at Linux.Sys.SUID.

Mixing both formats is not supported and will result in unexpected
results.

This artifact can also be used to look for all files owned by root with
world-writable permissions, for instance. Uncheck NumericMode, add a
glob,
select "root" as owner and enter any invalid permission string in
UserMode.
This will return every file owned by root. In the notebook, add
something
like "WHERE Mode=~'w.$'" to the query. The User field may also be empty,
essentially returning every file in the glob as long as the UserMode
field
contains an invalid value. This turns this artifact into a file
finder-like
tool with metadata like username and group names for further processing.

The following columns are returned:

  - OSPath
  - IsDir
  - UID
  - User
  - EUser (expected user from FilesToInspect)
  - GID
  - Group
  - EGroup (expected group from FilesToInspect)
  - Mode (file/directory mode/permissions)
  - EMode (expected file/directory mode from FilesToInspect)
- Mismatch (a comma-separated string of one or several of "uid", "gid"
and "mode")
  - Mtime
  - Ctime
  - Atime

Note that the artifacts used to look up usernames and group names use
the
files /etc/passwd and /etc/group. You will have to modify this artifact
to
use `getent passwd`/`getent group` to use NSS and get users and groups
from
Active Directory etc.

The provided default values in FilesToInspect is an example only.

content/exchange/artifacts/Linux.Detection.IncorrectPermissions.yaml

@@ -0,0 +1,241 @@
+name: Linux.Detection.IncorrectPermissions
+author: Andreas Misje – @misje
+description: |
+    NOTE: Requires velociraptor 0.7.1 or higher. – Alternatively, import the
+    artifact dependency Linux.Sys.Groups manually into your installation.
+
+    This artifacts checks a number of files and directories to verify whether
+    they have the expected owner, group owner and mode. A file with an incorrect
+    owner may allow attackers to modify important system files. Similarly,
+    incorrect mode, like word-writable configuration or passwd/shadow files may
+    also be signs of serious misconfiguration or signs of malicious activity.
+
+    The parameter FilesToInspect contains lines of globs to search for. Each
+    line may specify expected user, expected group, expected file mode and
+    expected directory mode. It is very important that the order of the lines
+    is in order of increasing specificity. For example:
+
+    ```csv
+    /etc/*,root,root,644,755
+    /etc/?shadow*,,shadow,640,
+    ```
+
+    Here, every file in the directory /etc is expected to be owned by root:root
+    and have file permissions set to 644, and group permissions set to 755.
+    The files under /etc matching "?shadow*" are still expected to be owned by
+    root, since the override for user is empty, but the group is no longer
+    expected to be "root", but "shadow". File permissions should be "640"
+    instead of "644". Note that this is an example and will most likely return
+    hits, since a number of files in /etc have different owners and modes.
+
+    "User" may either be an integer (UID) or a string (username). "Group"
+    may be either an integer (GID) or a string (group name). The names for all
+    UIDs and GIDs are looked up and displayed along with their IDs in the result.
+
+    Modes may be specified in either octal numbers or strings.
+
+    Modes specified in octal numbers, e.g. 755, 640, 1777, are matched
+    using a regular expression, so that both "0640", "640" and "100640" matches
+    "100640". An implicit anchor, '$', is used to match against the end of the
+    octal mode string.
+
+    When using mode strings (NumericMode unchecked), modes take the format
+    "-rwx-r-x-r-x". Regex comparison is used, and an implicit '$' anchor
+    is inserted at the end of the string. String modes allows for verifying only
+    certain bits of permissions, like ensuring that only the owner has write 
+    access, no one has permission to execute, but read access is not important:
+    "r.-.--.--". Or ensuring that SUID/GUID is not set. For finding files
+    specifically with SUID set, look at Linux.Sys.SUID.
+
+    Mixing both formats is not supported and will result in unexpected results.
+
+    This artifact can also be used to look for all files owned by root with
+    world-writable permissions, for instance. Uncheck NumericMode, add a glob,
+    select "root" as owner and enter any invalid permission string in UserMode.
+    This will return every file owned by root. In the notebook, add something
+    like "WHERE Mode=~'w.$'" to the query. The User field may also be empty,
+    essentially returning every file in the glob as long as the UserMode field
+    contains an invalid value. This turns this artifact into a file finder-like
+    tool with metadata like username and group names for further processing.
+
+    The following columns are returned:
+
+      - OSPath
+      - IsDir
+      - UID
+      - User
+      - EUser (expected user from FilesToInspect)
+      - GID
+      - Group
+      - EGroup (expected group from FilesToInspect)
+      - Mode (file/directory mode/permissions)
+      - EMode (expected file/directory mode from FilesToInspect)
+      - Mismatch (a comma-separated string of one or several of "uid", "gid" and "mode")
+      - Mtime
+      - Ctime
+      - Atime
+
+    Note that the artifacts used to look up usernames and group names use the
+    files /etc/passwd and /etc/group. You will have to modify this artifact to
+    use `getent passwd`/`getent group` to use NSS and get users and groups from
+    Active Directory etc.
+
+    The provided default values in FilesToInspect is an example only.
+
+parameters:
+  - name: FilesToInspect
+    type: csv
+    default: |
+        Globs,User,Group,FileMode,DirMode
+        /etc/passwd?,root,root,644,
+        /etc/?shadow?,root,shadow,640,
+        /etc/group?,root,root,,
+    description: The files to investigate. The default is just an example.
+  - name: NumericMode
+    type: bool
+    default: true
+    description: |
+        Whether modes should be interpreted, compared and presented as octal
+        numbers (e.g. 640) rather than strings (e.g. -rw-rw-r--)
+  - name: IncludeDirs
+    type: bool
+    description: Include directories
+  - name: FollowLinks
+    type: bool
+    description: Inlcude all symlinks, even though they may interfere with the results
+
+precondition: |
+    SELECT OS From info() WHERE OS = 'linux'
+
+sources:
+    - name: Discrepancies
+      query: |
+        /* Passwd/group will be looked up a lot. Make it efficient: */
+        LET Users <= memoize(query={
+            SELECT int(int=Uid) AS UID, User FROM Artifact.Linux.Sys.Users()
+        }, key='UID')
+        LET Groups <= memoize(query={
+            SELECT GID, Group FROM Artifact.Linux.Sys.Groups()
+        }, key='GID')
+
+        LET FindUser(uid) = get(item=Users, field=uid).User
+        LET FindGroup(gid) = get(item=Groups, field=gid).Group
+        LET StrIf(str) = if(condition=str, then=str(str=str))
+
+        LET Files = SELECT OSPath, IsDir, ModeString,
+            /* If the globs are specified in the correct order, picking the
+               last item will get a correct override behaviour: */
+            /* Filter out all empty strings and keep integers: */
+            filter(list=enumerate(items=User), regex='.', condition='x=>true')[-1] AS EUser,
+            filter(list=enumerate(items=Group), regex='.', condition='x=>true')[-1] AS EGroup,
+            filter(list=enumerate(items=FileMode), regex='.', condition='x=>true')[-1] AS FMode,
+            filter(list=enumerate(items=DirMode), regex='.', condition='x=>true')[-1] AS DMode
+            /* globs() can of course take a list of globs, like FilesToInspect.Globs,
+               but by using that approach, we would no longer be able to tie
+               User, Group etc. to the individual globs: */
+            FROM foreach(row={
+                SELECT Globs AS _Globs, * FROM FilesToInspect
+            }, query={
+                SELECT OSPath, IsDir, User, Group, FileMode, DirMode,
+                    Mode.String AS ModeString
+                    FROM glob(globs=_Globs)
+                    WHERE (IncludeDirs OR NOT IsDir) AND (NOT IsLink OR FollowLinks)
+            })
+            GROUP BY OSPath
+
+        LET FilesInfo = SELECT * FROM foreach(row=Files, async=true, query={
+            SELECT OSPath, Sys.Uid AS UID, Sys.Gid AS GID, if(condition=NumericMode,
+                    then=format(format='%o', args=[Sys.Mode]), else=ModeString)
+                AS Mode, Mtime, Atime, Ctime, EUser, EGroup, log(message='%v, %v, %v', args=[Sys.Mode, ModeString, FMode]), StrIf(str=FMode) AS FMode,
+                StrIf(str=DMode) AS DMode, IsDir
+                FROM stat(filename=OSPath)
+            })
+            GROUP BY OSPath
+
+        LET _UIDFiltered = SELECT OSPath, IsDir,
+            UID,
+            EUser,
+            GID, FindGroup(gid=GID) AS Group,
+            null AS EGroup,
+            'uid' AS Mismatch,
+            Mode,
+            null AS EMode,
+            Mtime, Atime, Ctime,
+            get(item=Users, field=UID) AS _u
+            FROM FilesInfo
+            WHERE EUser AND _u.UID=UID AND
+                /* User can be either an ID or a string: */
+                if(condition=EUser=~'^\\d+$', then=EUser!=UID,
+                            else=EUser!=_u.User)
+
+        LET _GIDFiltered = SELECT OSPath, IsDir,
+            UID, FindUser(uid=UID) AS User,
+            EUser,
+            GID,
+            EGroup,
+            'gid' AS Mismatch,
+            Mode,
+            null AS EMode,
+            Mtime, Atime, Ctime,
+            get(item=Groups, field=GID) AS _g
+            FROM FilesInfo
+            WHERE EGroup AND _g.GID=GID AND
+                if(condition=EGroup=~'^\\d+$', then=EGroup!=GID,
+                            else=EGroup!=_g.Group)
+
+        LET _FileModeFiltered = SELECT OSPath, IsDir,
+            UID, FindUser(uid=UID) AS User,
+            EUser,
+            GID, FindGroup(gid=GID) AS Group,
+            EGroup,
+            'mode' AS Mismatch,
+            Mode,
+            FMode AS EMode,
+            Mtime, Atime, Ctime
+            FROM FilesInfo
+            WHERE NOT IsDir AND FMode AND NOT Mode=~FMode+'$'
+
+        LET _DirModeFiltered = SELECT OSPath, IsDir,
+            UID, FindUser(uid=UID) AS User,
+            EUser,
+            GID, FindGroup(gid=GID) AS Group,
+            EGroup,
+            'mode' AS Mismatch,
+            Mode,
+            DMode AS EMode,
+            Mtime, Atime, Ctime
+            FROM FilesInfo
+            WHERE IsDir AND DMode AND NOT Mode=~DMode+'$'
+
+        SELECT OSPath, IsDir, UID, User, EUser, GID, Group, EGroup, Mode, EMode,
+            join(array=enumerate(items=Mismatch), sep=',') AS Mismatch,
+            Mtime, Atime, Ctime
+        FROM chain(a={
+            SELECT OSPath, IsDir, UID, _u.User AS User, EUser,
+            GID, Group, EGroup, Mode, EMode,
+            join(array=enumerate(items=Mismatch), sep=',') AS Mismatch,
+            Mtime, Atime, Ctime
+            FROM _UIDFiltered
+            GROUP BY OSPath
+        }, b={
+            SELECT OSPath, IsDir, UID, User, EUser, GID, _g.Group AS Group,
+            EGroup, Mode, EMode,
+            join(array=enumerate(items=Mismatch), sep=',') AS Mismatch,
+            Mtime, Atime, Ctime
+            FROM _GIDFiltered
+            GROUP BY OSPath
+        }, c={
+            SELECT OSPath, IsDir, UID, User, EUser, GID, Group, EGroup, Mode, EMode,
+            join(array=enumerate(items=Mismatch), sep=',') AS Mismatch,
+            Mtime, Atime, Ctime
+            FROM _FileModeFiltered
+            GROUP BY OSPath
+        }, d={
+            SELECT OSPath, IsDir, UID, User, EUser, GID, Group, EGroup, Mode, EMode,
+            join(array=enumerate(items=Mismatch), sep=',') AS Mismatch,
+            Mtime, Atime, Ctime
+            FROM _DirModeFiltered
+            GROUP BY OSPath
+        })
+        GROUP BY OSPath
+        ORDER BY OSPath