使用osqueryd监控系统

说明

osquery初识主要是借由osqueryi的方式对osquery进行了一个基本的介绍。可以看到osqueryi是一个交互式的shell，我们可以很方便使用它进行测试，但是如果我们要将osquery投入实际使用，明显是osqueryd更加合适。本篇文章将详细地介绍osqueryd的使用。

osqueryd配置

如果使用osqueryi，我们可以通过osqueryi -audit_allow_config=true --audit_allow_sockets=true --audit_persist=true这样的方式传入设置。如果是osqueryd呢？其实我们安装好osquery之后，会以service的方式存在于系统中，同时可以利用systemctl的方式进行控制，其文件位于/usr/lib/systemd/system/osqueryd.service。

[Unit]
Description=The osquery Daemon
After=network.service syslog.service

[Service]
TimeoutStartSec=0
EnvironmentFile=/etc/sysconfig/osqueryd
ExecStartPre=/bin/sh -c "if [ ! -f $FLAG_FILE ]; then touch $FLAG_FILE; fi"
ExecStartPre=/bin/sh -c "if [ -f $LOCAL_PIDFILE ]; then mv $LOCAL_PIDFILE $PIDFILE; fi"
ExecStart=/usr/bin/osqueryd \
  --flagfile $FLAG_FILE \
  --config_path $CONFIG_FILE
Restart=on-failure
KillMode=process
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

启动方式就是ExecStart=/usr/bin/osqueryd --flagfile $FLAG_FILE --config_path $CONFIG_FILE，通过--flagfile和--config_path的方式指定配置文件的路径。$FLAG_FILE和$CONFIG_FILE是在/etc/sysconfig/osqueryd中定义。

FLAG_FILE="/etc/osquery/osquery.flags"
CONFIG_FILE="/etc/osquery/osquery.conf"
LOCAL_PIDFILE="/var/osquery/osqueryd.pidfile"
PIDFILE="/var/run/osqueryd.pidfile"

默认的配置文件就是位于/etc/osquery/osquery.flags和/etc/osquery/osquery.conf。当启动osqueryd时，如果不存在osquery.flags和osquery.conf会创建两个空文件，否则直接读取此文件的内容。其实osquery.conf可以认为是osquery.flags的超集，因为osquery.flags仅仅只是设置一些配置，而这些配置也同样可以在osquery.conf中实现，同时在osquery.conf中还可以配置osqueryd需要执行的SQL。所以接下来本文将仅仅只介绍osquery.conf的使用。

osquery.conf

osquery本身提供了一个osquery.conf的例子，其写法是一个JSON格式的文件，在这里我们将其简化一下。

{
  // Configure the daemon below:
  "options": {
    // Select the osquery config plugin.
    "config_plugin": "filesystem",

    // Select the osquery logging plugin.
    "logger_plugin": "filesystem",

    // The log directory stores info, warning, and errors.
    // If the daemon uses the 'filesystem' logging retriever then the log_dir
    // will also contain the query results.
    //"logger_path": "/var/log/osquery",

    // Set 'disable_logging' to true to prevent writing any info, warning, error
    // logs. If a logging plugin is selected it will still write query results.
    //"disable_logging": "false",

    // Splay the scheduled interval for queries.
    // This is very helpful to prevent system performance impact when scheduling
    // large numbers of queries that run a smaller or similar intervals.
    //"schedule_splay_percent": "10",

    // A filesystem path for disk-based backing storage used for events and
    // query results differentials. See also 'use_in_memory_database'.
    //"database_path": "/var/osquery/osquery.db",

    // Comma-delimited list of table names to be disabled.
    // This allows osquery to be launched without certain tables.
    //"disable_tables": "foo_bar,time",

    "utc": "true"
  },

  // Define a schedule of queries:
  "schedule": {
    // This is a simple example query that outputs basic system information.
    "system_info": {
      // The exact query to run.
      "query": "SELECT hostname, cpu_brand, physical_memory FROM system_info;",
      // The interval in seconds to run this query, not an exact interval.
      "interval": 3600
    }
  },

  // Decorators are normal queries that append data to every query.
  "decorators": {
    "load": [
      "SELECT uuid AS host_uuid FROM system_info;",
      "SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;"
    ]
  },
  "packs": {
    // "osquery-monitoring": "/usr/share/osquery/packs/osquery-monitoring.conf",
    ....
  }, 
}

osquery.conf文件大致可以分为4个部分。

options，配置选项,Command Line Flags基本上对所有的配置选项都进行了说明。其实osquery.flags所配置也是这个部分。这也是之前说的osquery.conf可以认为是osquery.flags的超集的原因；
schedule，配置SQL语句。因为osqueryd是以daemon的方式运行，所以需要通过在schedule中定义SQL语句使其定期执行返回结果；
decorators，中文意思是“装饰”。在decorators中也是定义了一系列的SQL语句，执行得到的结果会附加在是在执行schedule中的结果的后面；所以我们看到在decorators我们取的是uuid和登录的username；
packs,就是一系列SQL语句的合集；

配置说明

上一节中对osquery.conf中的配置进了一个简单的说明，在本节中将详细说明。

options

options就是配置。Command Line Flags基本上对所有的配置选项都进行了说明。我们可以进行多种配置，有兴趣的可以自行研究。本节仅仅说明几个常用的配置；

config_plugin,配置选项是filesystem。如果是通过osquery.conf管理osquery就是采用filesystem，还有一种选项是tls(这一种主要是通过API的方式来配置osquery)。
logger_plugin,配置选项是filesystem，这也是osquery的默认值。根据Logger plugins，还可以配置tls,syslog (for POSIX,windows_event_log (for Windows),kinesis,firehose,kafka_producer。
database_path,默认值是/var/osquery/osquery.db。因为osquery内部会使用到数据，所以配置此目录是osquery的数据库文件位置。
disable_logging,是配置设置osquery的结果是否需要保存到本地，这个配置其实和logger_plugin:filesystem有点重复。
hostIdentifier,相当于表示每个主机的标识，比如可以采用hostname作为标识。

schedule

schedule是osqeuryd用于写SQL语句的标签。其中的一个示例如下所示：

"system_info": {
    // The exact query to run.
    "query": "SELECT hostname, cpu_brand, physical_memory FROM system_info;",
    // The interval in seconds to run this query, not an exact interval.
    "interval": 3600
}

其中system_info是定义的一个SQL任务的名字，也是一个JSON格式。在其中可以进行多项设置，包括：

query，定义需要执行的SQL语句；
interval，定时执行的时间，示例中是3600，表示每隔3600秒执行一次；
snapshot，可选选项，可以配置为snapshot:true。osquery默认执行的是增量模式，使用了snapshot则是快照模式。比如执行select * from processes;，osqeury每次产生的结果是相比上一次变化的结果;如果采用的是snapshot,则会显示所有的进程的，不会与之前的结果进行对比；
removed，可选选项，默认值是true,用来设置是否记录action为remove的日志。

当然还有一些其他的不常用选项，如platform、version、shard、description等等。

更多关于schedule的介绍可以参考schedule

decorators

正如其注释Decorators are normal queries that append data to every query所说，Decorators会把他的执行结果添加到schedule中的sql语句执行结果中。所以根据其作用Decorators也不是必须存在的。。在本例中Decorators存在两条记录:

1 2	SELECT uuid AS host_uuid FROM system_info; SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;

SELECT uuid AS host_uuid FROM system_info，从system_info获取uuid作为标识符1；
SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;,从logged_in_users选择user(其实查询的是用户名)的第一项作为标识符2；

当然可以在Decorators写多条语句作为标识符，但是感觉没有必要；

packs

packs就是打包的SQL语句的合集，本示例中使用的/usr/share/osquery/packs/osquery-monitoring.conf，这是官方提供的一个监控系统信息的SQL语句的集合；

{
  "queries": {
    "schedule": {
      "query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory, last_executed from osquery_schedule;",
      "interval": 7200,
      "removed": false,
      "blacklist": false,
      "version": "1.6.0",
      "description": "Report performance for every query within packs and the general schedule."
    },
    "events": {
      "query": "select name, publisher, type, subscriptions, events, active from osquery_events;",
      "interval": 86400,
      "removed": false,
      "blacklist": false,
      "version": "1.5.3",
      "description": "Report event publisher health and track event counters."
    },
    "osquery_info": {
      "query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;",
      "interval": 600,
      "removed": false,
      "blacklist": false,
      "version": "1.2.2",
      "description": "A heartbeat counter that reports general performance (CPU, memory) and version."
    }
  }
}

packs中的配置和schedule的配置方法并没有什么区别。我们在packs中查询到的信息包括：

从osquery_schedule拿到osqueryd设置的schedule的配置信息；
从osquery_events中拿到osqueryd所支持的所有的event；
从processes和osquery_info中拿到进程相关的信息；

使用packs的好处是可以将一系列相同功能的SQL语句放置在同一个文件中；

运行osqueryd

当以上配置完毕之后，我们就可以通过sudo osqueryd的方式启动；如果我们设置logger_plugin:filesystem，那么日志就会落在本地/var/log/osquery下。此目录下包含了多个文件，每个文件分别记录不同的信息。

osqueryd.results.log，osqueryd的增量日志的信息都会写入到此文件中；保存结果的形式是JSON形式。示例如下：

1
2

{"name":"auditd_process_info","hostIdentifier":"localhost.localdomain","calendarTime":"Wed Oct 24 13:07:12 2018 UTC","unixTime":1540386432,"epoch":0,"counter":0,"decorations":{"host_uuid":"99264D56-9A4E-E593-0B4E-872FBF3CD064","username":"username"},"columns":{"atime":"1540380461","auid":"4294967295","btime":"0","cmdline":"awk { sum += $1 }; END { print 0+sum }","ctime":"1538239175","cwd":"\"/\"","egid":"0","euid":"0","gid":"0","mode":"0100755","mtime":"1498686768","owner_gid":"0","owner_uid":"0","parent":"4086","path":"/usr/bin/gawk","pid":"4090","time":"1540386418","uid":"0","uptime":"1630"},"action":"added"}
{"name":"auditd_process_info","hostIdentifier":"localhost.localdomain","calendarTime":"Wed Oct 24 13:07:12 2018 UTC","unixTime":1540386432,"epoch":0,"counter":0,"decorations":{"host_uuid":"99264D56-9A4E-E593-0B4E-872FBF3CD064","username":"username"},"columns":{"atime":"1540380461","auid":"4294967295","btime":"0","cmdline":"sleep 60","ctime":"1538240835","cwd":"\"/\"","egid":"0","euid":"0","gid":"0","mode":"0100755","mtime":"1523421302","owner_gid":"0","owner_uid":"0","parent":"741","path":"/usr/bin/sleep","pid":"4091","time":"1540386418","uid":"0","uptime":"1630"},"action":"added"}

其中的added表示的就是相当于上一次增加的进程信息；每一次执行的结果都是一条JSON记录；

squeryd.snapshots.log,记录的是osqueryd中使用snapshot:true标记的SQL语句执行结果；

1
2

{"snapshot":[{"header":"Defaults","rule_details":"!visiblepw"},{"header":"Defaults","rule_details":"always_set_home"},{"header":"Defaults","rule_details":"match_group_by_gid"},{"header":"Defaults","rule_details":"env_reset"},{"header":"Defaults","rule_details":"env_keep = \"COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS\""},{"header":"Defaults","rule_details":"env_keep += \"MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE\""},{"header":"Defaults","rule_details":"env_keep += \"LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES\""},{"header":"Defaults","rule_details":"env_keep += \"LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE\""},{"header":"Defaults","rule_details":"env_keep += \"LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY\""},{"header":"Defaults","rule_details":"secure_path = /sbin:/bin:/usr/sbin:/usr/bin"},{"header":"root","rule_details":"ALL=(ALL) ALL"},{"header":"%wheel","rule_details":"ALL=(ALL) ALL"}],"action":"snapshot","name":"sudoers","hostIdentifier":"localhost.localdomain","calendarTime":"Tue Oct  9 11:54:00 2018 UTC","unixTime":1539086040,"epoch":0,"counter":0,"decorations":{"host_uuid":"99264D56-9A4E-E593-0B4E-872FBF3CD064","username":"username"}}
{"snapshot":[{"header":"Defaults","rule_details":"!visiblepw"},{"header":"Defaults","rule_details":"always_set_home"},{"header":"Defaults","rule_details":"match_group_by_gid"},{"header":"Defaults","rule_details":"env_reset"},{"header":"Defaults","rule_details":"env_keep = \"COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS\""},{"header":"Defaults","rule_details":"env_keep += \"MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE\""},{"header":"Defaults","rule_details":"env_keep += \"LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES\""},{"header":"Defaults","rule_details":"env_keep += \"LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE\""},{"header":"Defaults","rule_details":"env_keep += \"LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY\""},{"header":"Defaults","rule_details":"secure_path = /sbin:/bin:/usr/sbin:/usr/bin"},{"header":"root","rule_details":"ALL=(ALL) ALL"},{"header":"%wheel","rule_details":"ALL=(ALL) ALL"}],"action":"snapshot","name":"sudoers","hostIdentifier":"localhost.localdomain","calendarTime":"Tue Oct  9 11:54:30 2018 UTC","unixTime":1539086070,"epoch":0,"counter":0,"decorations":{"host_uuid":"99264D56-9A4E-E593-0B4E-872FBF3CD064","username":"username"}}

由于snapshot是快照模式，所以即使两次结果相同也会全部显示出来；

osqueryd.INFO,记录osqueryd中正在运行的情况。示例如下：

Log file created at: 2018/11/22 17:06:06
Running on machine: osquery.origin
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1122 17:06:06.729902 22686 events.cpp:862] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
I1122 17:06:06.730651 22686 events.cpp:862] Event publisher not enabled: syslog: Publisher disabled via configuration

osqueryd.WARNING，记录osquery的警告。示例如下：

Log file created at: 2018/10/09 19:53:45
Running on machine: localhost.localdomain
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1009 19:53:45.471046 104258 events.cpp:987] Requested unknown/failed event publisher: auditeventpublisher
E1009 19:53:45.471606 104259 events.cpp:987] Requested unknown/failed event publisher: inotify
E1009 19:53:45.471634 104260 events.cpp:987] Requested unknown/failed event publisher: syslog
E1009 19:53:45.471658 104261 events.cpp:987] Requested unknown/failed event publisher: udev

osqueryd.ERROR，记录的是osquery的错误信息。示例如下：

Log file created at: 2018/10/09 19:53:45
Running on machine: localhost.localdomain
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1009 19:53:45.471046 104258 events.cpp:987] Requested unknown/failed event publisher: auditeventpublisher
E1009 19:53:45.471606 104259 events.cpp:987] Requested unknown/failed event publisher: inotify
E1009 19:53:45.471634 104260 events.cpp:987] Requested unknown/failed event publisher: syslog
E1009 19:53:45.471658 104261 events.cpp:987] Requested unknown/failed event publisher: udev

在本例中错误信息和警告信息完全相同。在实际情况下，可能很多时候均不相同；

总结

本文主要是对osqueryd的常用配置进行了简要的说法。通过本文能够快速地利用上手osquery，由于篇幅的原因，有关osquery的很多东西没有介绍或者说明得很详细。官方的文档[]对osqueryd的配置已经说明得很是详尽了，如果对本文有任何的不解，可以去查阅相关的文档，也欢迎大家就相关问题与我讨论。

以上