Skip to content

About

Tools for the processing and manipulation of time-series data in Clojure.

Topics

Resources

Stars

19 stars

Watchers

8 watching

Forks

Repository files navigation

tablecloth.time

Clojars Project build

Description

This library offers tools for manipulating and processing time-series data. It complements and extends the API provided by tablecloth for working with the highly performant columnar datasets of tech.ml.dataset.

Status: Under active development. The API may change.

Design Philosophy

This library is experimental and has evolved since its inception. The current approach favors composability over high-level abstraction.

In practice, this means we do not store metadata on the dataset indicating which column contains temporal data. Rather than providing functions like adjust-frequency that implicitly operate on a designated time index, we provide primitives like add-time-columns that make it easy to derive temporal fields explicitly. Users then compose these with standard tablecloth operations (group-by, aggregate, etc.).

A Note on Indexing

Previous versions of tablecloth.time implemented index-aware functions built on an indexing mechanism in tech.ml.dataset. That indexing mechanism was removed as part of TMD v7's simplification effort (discussion on Zulip).

Key points:

  1. This doesn't preclude future indexing. A future version of tablecloth.time or tech.ml.dataset may reintroduce an optional mechanism for designating a temporal index.

  2. The SciCloj community is experimenting with index-free approaches. The current thinking is that explicit column arguments + binary search performs well enough for most time-series workloads, without the complexity that indexing adds. As Chris Nuernberger (dtype-next author) noted: "Just sorting the dataset and using binary search will outperform most/all tree structures in this scenario."

  3. Pandas-style indexing is complex. Harold observed that "there is about as much code in Pandas doing indexing stuff as there is all of the code in TMD." The removal was a deliberate choice to keep the core library focused.

For now, we prioritize explicit column arguments and composable primitives. See doc/zulip-indexing-discussion-summary.md for the full context.

Usage

(require '[tablecloth.api :as tc]
         '[tablecloth.time.api :as tct]
         '[tablecloth.time.column.api :as tct-col])

;; Add temporal fields to a dataset
(-> my-dataset
    (tct/add-time-columns :timestamp {:year "Year" 
                                      :month "Month"
                                      :day-of-week "DayOfWeek"}))

;; Slice a time range
(-> my-dataset
    (tc/order-by :timestamp)
    (tct/slice :timestamp #time/date "2024-01-01" #time/date "2024-03-31"))

;; Add lag columns for time series analysis
(-> my-dataset
    (tct/add-lags :price [1 2 3 4]))

;; Column-level operations
(tct-col/year (my-dataset :timestamp))        ; extract year
(tct-col/floor-to-month (my-dataset :timestamp))  ; truncate to month

Resampling Example

Resample half-hourly electricity data to daily averages — no magic, just composable primitives:

;; Load half-hourly Victorian electricity data
(def vic-elec (tc/dataset "data/fpp3/vic_elec.csv" {:key-fn keyword}))

;; Resample to daily averages:
;; 1. Extract date from datetime using add-time-columns
;; 2. Group and aggregate with standard tablecloth
(-> vic-elec
    (tct/add-time-columns :Time {:date-string "Day"})
    (tc/group-by ["Day"])
    (tc/mean :Demand))

The philosophy: add-time-columns extracts temporal components you need, then standard tablecloth does the rest. Explicit columns throughout — no implicit index, no magic.

Victorian electricity demand (daily average)

See notebooks/chapter_02_plotje.clj (rendered) for more examples based on Forecasting: Principles and Practice.

Lag and Lead

add-lag / add-lead (and the column-level lag / lead) shift by row position, not by timestamp: sort first, and note that gaps in the time axis are not accounted for.

Rows with no source value (the first k for a lag, the last k for a lead) are missing by default. :fill puts a value there instead; values that were already missing in the source stay missing.

(def ds (tc/dataset {:x [1.0 2.0 nil 4.0 5.0]}))

(tct/add-lag ds :x 2 :x_lag2)            ; :x_lag2 => [nil nil 1.0 2.0 nil]
(tct/add-lag ds :x 2 :x_lag2 {:fill 0})  ; :x_lag2 => [0.0 0.0 1.0 2.0 nil]
(tct/add-lead ds :x 1 :x_lead1 {:fill 0}) ; :x_lead1 => [2.0 nil 4.0 5.0 0.0]

;; :fill also applies to every column of add-lags / add-leads. Filled rows are
;; not missing, so :drop-missing (default true) then drops only rows whose
;; shifted value was missing in the source.
(tct/add-lags ds :x [1 2] {:fill 0 :drop-missing false})

k = 0 returns the column unchanged; k at or beyond the row count gives an all-boundary column of the same length and type; negative k throws.

To benchmark lag implementations: clj -M:dev bench/lag_lead_bench.clj.

Development

We use a "hybrid" Leinigen/tools.deps setup. You can use either lein or clj/clojure. We chose this hybrid setup so that we get the benefits of Leinigen's suite of build-related tools, while also benefitting from the simplicity of tools.deps' approach to dependencies. For more context on the differences and trade-offs between the the two, see this post.

Linting

We use both cljfmt and clj-kondo to lint our code. To run the linters, do:

lein lint

Tests

We run tests using the midje test runner, which will run both any midje tests and any standard clojure tests:

lein midje

Releasing to Clojars

Publishing is currently manual.

  1. Update the version in project.clj (release versions should not include -SNAPSHOT).
  2. Run local checks:
lein lint
lein midje
  1. Deploy:
lein deploy clojars
  1. After deploy, bump project.clj to the next -SNAPSHOT version for ongoing development.

Contributing

Development for this project happens in the SciCloj fundamentals study group, a group focused on improving the Cojure datascience ecosystem. We tend to hang out in the #sci-fu stream on the Clojurians Zulip, and we meet regularly to coordinate, learn, and solve problems.

We eagerly invite your participation in this project. The project is currently in an experimental phase and we are approaching its development interactively as a group in a way that is driven by openness and learning. If you are interested, please reach out.

Please peruse this project's issues to get a sense of work that is ongoing for this project.

License

MIT for now, but this is basically a placeholder. Open to suggestions.

About

Tools for the processing and manipulation of time-series data in Clojure.

Topics

Resources

Stars

19 stars

Watchers

8 watching

Forks

Releases

Packages

Used by

Contributors

Languages

⌂