Skip to content

About

Dataset manipulation library built on the top of tech.ml.dataset

Topics

Resources

Stars

370 stars

Watchers

10 watching

Forks

Repository files navigation

Tablecloth

Tablecloth logo

Dataset (data frame) manipulation API for the tech.ml.dataset library

Versions

tech.ml.dataset 7.x (master branch)

tech.ml.dataset 4.x (4.0 branch)

[scicloj/tablecloth "4.04"]

Introduction

tech.ml.dataset is a great and fast library which brings columnar dataset to the Clojure. Chris Nuernberger has been working on this library for last year as a part of bigger tech.ml stack.

I've started to test the library and help to fix uncovered bugs. My main goal was to compare functionalities with the other standards from other platforms. I focused on R solutions: dplyr, tidyr and data.table.

During conversions of the examples I've come up how to reorganized existing tech.ml.dataset functions into simple to use API. The main goals were:

  • Focus on dataset manipulation functionality, leaving other parts of tech.ml like pipelines, datatypes, readers, ML, etc.
  • Single entry point for common operations - one function dispatching on given arguments.
  • group-by results with special kind of dataset - a dataset containing subsets created after grouping as a column.
  • Most operations recognize regular dataset and grouped dataset and process data accordingly.
  • One function form to enable thread-first on dataset.

Important! This library is not the replacement of tech.ml.dataset nor a separate library. It should be considered as a addition on the top of tech.ml.dataset.

If you want to know more about tech.ml.dataset and dtype-next please refer their documentation:

Join the discussion on Zulip

Documentation

Please refer detailed documentation with examples.

The old documentation (till the end of 2023) is here.

Usage example

(require '[tablecloth.api :as tc])
(-> "https://raw.githubusercontent.com/techascent/tech.ml.dataset/master/test/data/stocks.csv"
    (tc/dataset {:key-fn keyword})
    (tc/group-by (fn [row]
                    {:symbol (:symbol row)
                     :year (tech.v3.datatype.datetime/long-temporal-field :years (:date row))}))
    (tc/aggregate #(tech.v3.datatype.functional/mean (% :price)))
    (tc/order-by [:symbol :year])
    (tc/head 10))

_unnamed [10 3]:

:symbol :year summary
AAPL 2000 21.74833333
AAPL 2001 10.17583333
AAPL 2002 9.40833333
AAPL 2003 9.34750000
AAPL 2004 18.72333333
AAPL 2005 48.17166667
AAPL 2006 72.04333333
AAPL 2007 133.35333333
AAPL 2008 138.48083333
AAPL 2009 150.39333333

Contributing

Tablecloth is open for contribution. The best way to start is discussion on Zulip.

Development tools for documentation

Documentation is written in the Kindly convention and is rendered using Clay composed with Quarto.

The old documentation was written in RMarkdown and is kept under docs/old/.

Documentation contains around 600 code snippets which are run during build. There are three relevant source files:

(notebooks/index.clj was generated by dev/conversion.clj from the earlier Rmarkdown-based index.Rmd with asome additional manual editing. Starting at 2024, it will diverge from that source, that will no longer be maintained.)

README generation

To generate README.md, run the generate! function at the dev/readme_generation.clj script.

Detailed documentation generation

Build docs/index.html from notebooks/index.clj from the repository root:

clojure -M:docs dev/gendocs.clj

The :docs alias provides everything the notebook needs on the JVM: Clay, dataset-io for the Excel, Arrow and Parquet examples, and the larger thread stack Clay needs. Two external tools must be on your PATH:

  • The Quarto CLI. The current site was rendered with Quarto 1.9.37.
  • R with the data.table, knitr and rmarkdown packages. The data.table comparison section runs R chunks through knitr and downloads flights14.csv from GitHub, so the build also needs network access.

The command exits nonzero, leaving docs/index.html unchanged, if a notebook form throws, a file or URL fails to load, or Quarto fails. Commit the regenerated docs/index.html with the notebook change.

Code Generation

Four source files are generated. Do not edit them by hand: change their generators instead, then run this from the repository root:

clojure -M:regen

It regenerates these files in this order, because each step reads the files before it:

  1. src/tablecloth/column/api/operators.clj, from tablecloth.column.api.lift-operators
  2. src/tablecloth/api/operators.clj, from tablecloth.api.lift-operators
  3. src/tablecloth/column/api.clj, from tablecloth.column.api.api-template
  4. src/tablecloth/api.clj, from tablecloth.api.api-template

Regenerate, and commit the result, whenever you change a template, a lift-operators namespace or tablecloth.utils.codegen, or change or upgrade anything the generated files export. If any step fails, all four files are restored to their previous contents. Running the command twice produces identical files.

Guideline

  1. Before commiting changes please perform tests. I ususally do: lein do clean, check, test and build documentation as described above (which also tests whole library).
  2. Keep API as simple as possible:
    • first argument should be a dataset
    • if parametrizations is complex, last argument should accept a map with not obligatory function arguments
    • avoid variadic associative destructuring for function arguments
    • usually function should working on grouped dataset as well, accept parallel? argument then (if applied).
  3. Follow potemkin pattern and import functions to the API namespace using tech.v3.datatype.export-symbols/export-symbols function
  4. Functions which are composed out of API function to cover specific case(s) should go to tablecloth.utils namespace.
  5. Always update README-source.md, CHANGELOG.md, notebooks/index.clj, tests and function docs are highly welcomed.
  6. Always discuss changes and PRs first

Tests

Tests are written and run using midje. To run a test, evaluate a midje form. If it passes, it will return true, if it fails details will be printed to the REPL.

TODO

  • elaborate on tests
  • tutorials

Licence

Copyright (c) 2020 Scicloj

The MIT Licence

About

Dataset manipulation library built on the top of tech.ml.dataset

Topics

Resources

Stars

370 stars

Watchers

10 watching

Forks

Releases

Packages

Used by

Contributors

Languages

⌂