文档文档

使用 Flux 对数据进行窗口和聚合

对时间序列数据进行的一个常见操作是将数据分组到时间窗口中,或“窗口化”数据,然后将窗口化的值聚合到一个新的值中。本指南将指导您如何使用 Flux 对数据进行窗口化和聚合,并演示在这个过程中数据是如何被塑形的。

如果您刚开始使用 Flux 查询,请查看以下内容

以下示例深入介绍了进行窗口化和聚合数据所需的步骤。此 aggregateWindow() 函数 会为您执行这些操作,但了解数据在过程中的形状有助于成功创建所需输出。

数据集

在本指南中,定义一个表示基本数据集的变量。以下示例查询宿主机的内存使用情况。

dataSet = from(bucket: "example-bucket")
    |> range(start: -5m)
    |> filter(fn: (r) => r._measurement == "mem" and r._field == "used_percent")
    |> drop(columns: ["host"])

此示例从返回的数据中删除了 host 列,因为内存数据只为单个宿主机跟踪,这简化了输出表。删除 host 列是可选的,如果监控多个宿主机的内存,则不推荐这样做。

dataSet 现在可以用来表示基本数据,其外观类似于以下内容

Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:50:00.000000000Z             71.11611366271973
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:50:10.000000000Z             67.39630699157715
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:50:20.000000000Z             64.16666507720947
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:50:30.000000000Z             64.19951915740967
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:50:40.000000000Z              64.2122745513916
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:50:50.000000000Z             64.22209739685059
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:51:00.000000000Z              64.6336555480957
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:51:10.000000000Z             64.16516304016113
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:51:20.000000000Z             64.18349742889404
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:51:30.000000000Z             64.20474052429199
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:51:40.000000000Z             68.65062713623047
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:51:50.000000000Z             67.20139980316162
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:52:00.000000000Z              70.9143877029419
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:52:10.000000000Z             64.14549350738525
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:52:20.000000000Z             64.15379047393799
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:52:30.000000000Z              64.1592264175415
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:52:40.000000000Z             64.18190002441406
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:52:50.000000000Z             64.28837776184082
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:53:00.000000000Z             64.29731845855713
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:53:10.000000000Z             64.36963081359863
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:53:20.000000000Z             64.37397003173828
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:53:30.000000000Z             64.44413661956787
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:53:40.000000000Z             64.42906856536865
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:53:50.000000000Z             64.44573402404785
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:00.000000000Z             64.48912620544434
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:10.000000000Z             64.49522972106934
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:20.000000000Z             64.48652744293213
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:30.000000000Z             64.49949741363525
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:40.000000000Z              64.4949197769165
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:50.000000000Z             64.49787616729736
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:55:00.000000000Z             64.49816226959229

数据窗口化

使用 window() 函数 根据时间范围对数据进行分组。与 window() 一起传递的最常见参数是 every,它定义了窗口之间的时间长度。其他参数也可以使用,但在这个例子中,将基本数据集窗口化为一分钟窗口。

dataSet
    |> window(every: 1m)

every 参数支持所有 有效的持续时间单位,包括 日历月份(1mo年份(1y

每个时间窗口输出在其自己的表中,包含所有在窗口内的记录。

窗口() 输出表
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem  2018-11-03T17:50:00.000000000Z             71.11611366271973
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem  2018-11-03T17:50:10.000000000Z             67.39630699157715
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem  2018-11-03T17:50:20.000000000Z             64.16666507720947
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem  2018-11-03T17:50:30.000000000Z             64.19951915740967
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem  2018-11-03T17:50:40.000000000Z              64.2122745513916
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem  2018-11-03T17:50:50.000000000Z             64.22209739685059


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem  2018-11-03T17:51:00.000000000Z              64.6336555480957
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem  2018-11-03T17:51:10.000000000Z             64.16516304016113
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem  2018-11-03T17:51:20.000000000Z             64.18349742889404
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem  2018-11-03T17:51:30.000000000Z             64.20474052429199
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem  2018-11-03T17:51:40.000000000Z             68.65062713623047
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem  2018-11-03T17:51:50.000000000Z             67.20139980316162


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem  2018-11-03T17:52:00.000000000Z              70.9143877029419
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem  2018-11-03T17:52:10.000000000Z             64.14549350738525
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem  2018-11-03T17:52:20.000000000Z             64.15379047393799
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem  2018-11-03T17:52:30.000000000Z              64.1592264175415
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem  2018-11-03T17:52:40.000000000Z             64.18190002441406
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem  2018-11-03T17:52:50.000000000Z             64.28837776184082


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem  2018-11-03T17:53:00.000000000Z             64.29731845855713
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem  2018-11-03T17:53:10.000000000Z             64.36963081359863
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem  2018-11-03T17:53:20.000000000Z             64.37397003173828
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem  2018-11-03T17:53:30.000000000Z             64.44413661956787
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem  2018-11-03T17:53:40.000000000Z             64.42906856536865
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem  2018-11-03T17:53:50.000000000Z             64.44573402404785


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:00.000000000Z             64.48912620544434
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:10.000000000Z             64.49522972106934
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:20.000000000Z             64.48652744293213
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:30.000000000Z             64.49949741363525
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:40.000000000Z              64.4949197769165
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:50.000000000Z             64.49787616729736


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:55:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:55:00.000000000Z             64.49816226959229

在 InfluxDB UI 中可视化时,每个窗口表以不同的颜色显示。

Windowed data

汇总数据

汇总函数 取得表中所有行的值,并使用它们执行汇总操作。结果以单一行的表中的新值输出。

由于窗口化数据被分割成单独的表,汇总操作对每个表单独运行,并输出只包含汇总值的新表。

对于这个例子,使用 mean() 函数 输出每个窗口的平均值

dataSet
    |> window(every: 1m)
    |> mean()
mean() 输出表
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------------
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem             65.88549613952637


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------------
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem             65.50651391347249


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------------
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem             65.30719598134358


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------------
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem             64.39330975214641


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------------
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem             64.49386278788249


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ----------------------------
2018-11-03T17:55:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem             64.49816226959229

由于每个数据点都包含在其自己的表中,当可视化时,它们看起来像单个、未连接的点。

Aggregated windowed data

重新创建时间列

注意,_time 列不在 汇总输出表中 因为每个表中的记录都被汇总在一起,它们的时间戳不再适用,因此该列从分组键和表中删除。

同时请注意,_start_stop 列仍然存在。这些表示时间窗口的下限和上限。

许多 Flux 函数依赖于 _time 列。要在执行汇总函数后进一步处理数据,您需要重新添加 _time。使用 duplicate() 函数_start_stop 列复制为新的 _time 列。

dataSet
    |> window(every: 1m)
    |> mean()
    |> duplicate(column: "_stop", as: "_time")
duplicate() 输出表
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:50:00.000000000Z  2018-11-03T17:51:00.000000000Z            used_percent                     mem  2018-11-03T17:51:00.000000000Z             65.88549613952637


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:51:00.000000000Z  2018-11-03T17:52:00.000000000Z            used_percent                     mem  2018-11-03T17:52:00.000000000Z             65.50651391347249


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:52:00.000000000Z  2018-11-03T17:53:00.000000000Z            used_percent                     mem  2018-11-03T17:53:00.000000000Z             65.30719598134358


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:53:00.000000000Z  2018-11-03T17:54:00.000000000Z            used_percent                     mem  2018-11-03T17:54:00.000000000Z             64.39330975214641


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:54:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:55:00.000000000Z             64.49386278788249


Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:55:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:55:00.000000000Z             64.49816226959229

“取消窗口”汇总表

通常,将汇总值保留在单独的表中不是您想要的数据格式。使用 window() 函数将数据“取消窗口”到一个单一的无限(inf)窗口。

dataSet
    |> window(every: 1m)
    |> mean()
    |> duplicate(column: "_stop", as: "_time")
    |> window(every: inf)

窗口化需要 _time 列,这就是为什么在汇总后必须 重新创建 _time 的原因。

取消窗口化输出表
Table: keys: [_start, _stop, _field, _measurement]
                   _start:time                      _stop:time           _field:string     _measurement:string                      _time:time                  _value:float
------------------------------  ------------------------------  ----------------------  ----------------------  ------------------------------  ----------------------------
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:51:00.000000000Z             65.88549613952637
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:52:00.000000000Z             65.50651391347249
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:53:00.000000000Z             65.30719598134358
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:54:00.000000000Z             64.39330975214641
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:55:00.000000000Z             64.49386278788249
2018-11-03T17:50:00.000000000Z  2018-11-03T17:55:00.000000000Z            used_percent                     mem  2018-11-03T17:55:00.000000000Z             64.49816226959229

将汇总值放在单一表中,可视化中的数据点连接在一起。

Unwindowed aggregate data

总结

您现在已经创建了一个 Flux 查询,它对数据进行窗口化和汇总。本指南中概述的数据转换过程应适用于所有汇总操作。

Flux 还提供了 aggregateWindow() 函数,该函数为您执行所有这些单独的功能。

以下 Flux 查询将返回相同的结果

aggregateWindow 函数
dataSet
    |> aggregateWindow(every: 1m, fn: mean)

这个页面有帮助吗?

感谢您的反馈!


Flux 的未来

Flux 将进入维护模式。您可以继续按照当前方式使用它,而无需对您的代码进行任何更改。

阅读更多

InfluxDB v3 增强功能和 InfluxDB 集群版现已上市

新功能,包括更快的查询性能和管理工具,推动了 InfluxDB v3 产品线的进步。InfluxDB 集群版现已上市。

InfluxDB v3 性能和功能

InfluxDB v3 产品线在查询性能方面进行了重大改进,并提供了新的管理工具。这些改进包括用于监控 InfluxDB 集群健康状况的操作仪表板、InfluxDB 云专享中的单点登录(SSO)支持,以及用于令牌和数据库的新管理 API。

了解新的 v3 增强功能


InfluxDB 集群版上市

InfluxDB 集群版现已上市,并为您在自行管理的堆栈中提供 InfluxDB v3 的功能。

与我们谈谈 InfluxDB 集群版