v0.60.0

Breaking

• Renaming xan from --path to --root.
• Renaming xan separate -T/--txt to -L/--lines.

Features

• Adding xan count -c/--check-alignment & -H/--human-readable.
• Adding xan sort -e -z/--compress.
• Adding xan sample -g -S/--sorted.
• Adding xan from -f=(json|ndjson) --model.
• Adding xan cat rows -I/--intersection & -U/--union.
• Adding xan top -g -S/--sorted.
• Adding the sort, dedup, flatten & flat_map moonblade function.
• Adding support for list & map columns when using xan from -f=parquet.

Fixes

• Fixing text wrapping across tool, especially xan flatten -w & xan flatten -F.
• Fixing xan select -e & xan map plural clause flattening given list.
• Fixing xan from -f=parquet not emitting correct column names for list & map columns.

Performance

• Introducing a fast path for xan sort -e when input fits in a single chunk.
• Faster xan from -f=ndjson & -f=parquet.
• Amortizing xan sample allocations.
• Improving performance of xan freq & xan p freq.

Quality of Life

• Better record size estimation for xan sort -e.
• Better xan flatten -c when string width cannot be computed correctly (because of emojis typically).

v0.60.0-rc.1

Bump 0.60.0-rc.1

v0.59.0

Breaking

• Bumping MSRV to 1.85.0 and edition 2024.
• Overhauling how xan scrape takes its inputs. It now targets HTML files on disk by default now.
• xan plot --density-scale now defaults to log.
• xan freq -X/--approx-algo & xan p freq -X/--approx-algo now default to heavy-keeper.
• Dropping xan eval -S/--serialize. The default behavior of the command is now to output the serialized value.

Features

• Adding xan search -x/--pattern-file.
• Adding xan cat --glob, xan merge --glob, xan parallel --glob.
• Adding the hostname moonblade function.
• Adding xan scrape --paths, --path-column, --docs, --docs-column, -D/--stdin-doc & --glob.
• Adding xan from -f=(json|ndjson) --path <path>.
• Adding xan from -f toml & -f raw.
• Adding xan spark --hide-all & --repeat-x-axis.

Fixes

• Fixing moonblade string with bstring equality.
• Fixing percentages shown by xan spark -P.
• Fixing xan complete correctness in presence of duplicate values.
• Fixing xan spark -c discretization & legend.

Performance

• Improving performance of xan scrape.
• Improving performance of xan from -f ndjson.

Quality of Life

• Better legends & axis for xan spark.
• Highlighting empty labels when using xan heatmap.

v0.59.0-rc.1

Bump 0.59.0-rc.1

v0.58.0

Breaking

• Stopping to serialize moonblade lists either as joined by some separator or JSON. This was awkard, error-prone & potentially lossy. Use the join function manually to format output when required.
• As per previous point, dropping xan scrape --sep.
• Dropping implicit unary function calls in moonblade pipelines. This feature was not well-known, confusing (an indentifier, could be understood as a call in a pipeline, only if not in first position...), and mostly useless now that moonblade has had a proper dot operator.
• xan plot -A/--aggregate does not take an expression anymore but has an automatic selection of two modes: sum and mean. It should also be faster.
• Renaming the index function as row_index for clarity.
• xan agg -C/--along-columns & -M/--along-matrix & xan groupby -C/--along-columns & -M/--along-matrix will not map current column index to the result of the index() function. The col_index() can be now used instead for this very purpose.
• xan window -g/--groupby does not require the file to be sorted anymore. This means using -g/--groupby will now require the whole file to be buffered into memory by the command. The old behavior can still be used through the -S/--sorted flag, thus aligning the xan window command with the rest of the tool.
• row_index will now error if the expression has no concept of row index, instead of returning nothing.
• xan parallel -z/--compress now take the desired compression (either gzip or zstd).
• Retiring the xan grep command in favor of xan search -Z/--fast-parser.
• xan tokenize --keep short flag becomes -k instead of -K to harmonize with other commands.
• Retiring the xan flatmap command in favor of xan explode -e.
• Retiring the xan fuzzy-join command in favor of a consolidated xan join command.
• Changing xan from -f txt -c <name> default to line instead of value.
• Renaming xan join -L/--prefix-left & -R/--prefix-right short flags to -l & -r respectively to avoid colliding with the added -R/--reverse flag that can be used for merge joins.
• Dropping xan plot -B/--bars. It never worked very well and its use-case will be redirected to xan spark.
• Changing xan heatmap --width short flag from -w to -W so that adding a -H/--height flag remain consistent and avoids clashing with -h/--help.
• Dropping xan heatmap --show-gradients in favor of xan help gradients.
• Renaming xan search -A/--all flag to --every-column for clarity and avoid clash with -A/--after-context.
• Dropping xan sort -U/--unstable. It was never used and the performance boost it supposedly provides cannot be observed.

Features

• Adding xan parallel --dont-chunk.
• Adding nullary col, col_index & header variants, to work with expression applied in series to multiple columns at once.
• Adding prev_col & next_col functions.
• Adding xan (search|filter) -B/--before-context & -A/--after-context.
• Adding xan window -O/--overwrite.
• Adding xan map -C/--along-columns.
• Adding xan window -C/--along-columns.
• Adding xan cat rows --raw, -P/--preprocess & -H/--shell-preprocess.
• Improving xan select DSL star selectors. You can now do stuff like vec_*_count, *[1], vec_*[1] etc.
• xan p -H/--shell-preprocess now works on Windows.
• Adding native zsh completions (@apcamargo).
• Adding xan dedup --u32.
• Adding xan explode -e/--evaluate, -f/--evaluate-file, --pad & -k/--keep.
• xan to npy is now able to stream.
• Adding xan parallel top & xan top -p/--parallel, -t/--threads.
• Adding xan network edgelist --range.
• Adding xan network nodelist.
• Adding the xan run command.
• Adding xan view --name.
• Adding xan join -S/--sorted, -R/--reverse & -N/--numeric.
• Adding xan parallel --run & xan cat rows --run.
• Adding xan to md -l/--limit.
• Adding the xan spark command.
• Adding xan stats -R/--report, --color, --cols, --sep.
• Adding xan (freq|p freq) -X/--approx-algo.
• Adding xan plot -D/--density-gradient, --density-scale, --hide-legend, --hide-x-axis, --hide-y-axis, --hide-all & -Q/--square.
• xan separate will now avoid emitting columns with an empty name given to --into.
• Adding xan separate --txt & --F/--filter.
• Adding pow & sqrt scales.

Fixes

• Fixing issues related to nested lambdas in expressions.
• Fixing xan rename consistency regarding CRLF newlines and first row normalization when using -n/--no-headers.
• Fixing xan map --overwrite --filter.
• Fixing lead window function when there is not enough rows ahead.
• Fixing xan network --format not being validated early enough.
• Fixing xan explode -D/--drop-empty when selecting multiple columns.
• Fixing xan merge -u row precedence.
• Fixing xan join -D/--drop-key automatic selection when using --full.
• Fixing granularity inference of xan plot -T.
• Fixing xan from -f (json|ndjson) to emit empty outputs from empty inputs.
• Fixing xan headers layout when input files have a very large number of columns (>= 1000).
• Fixing arity validation of top, argtop, most_common & most_common_counts aggregation functions.

Performance

• moonblade expressions are now faster overall and allocate more cautiously, thus saving memory.
• Improving performance of xan transform, xan flatmap, xan agg & xan groupby.
• Improving performance of xan rename.
• Faster xan range.
• Faster xan parallel -H/--shell-preprocess.
• Faster xan tokenize words.
• Adding fast path for xan explode when only a single column is selected.
• Faster xan sort -e.

Quality of Life

• xan plot will now display <empty> label in legends.
• xan cat rows will now error when input have inconsistent columns.
• Automatic column alignement with xan to md.
• xan from now consider .log files as text lines.

v0.57.1

Fixes

• Fixing xan sort --check -n & xan dedup --check -n printed report.
• Fixing xan parallel cat -nS.
• Fixing CSV parsing rare edge case panics.
• Fixing commands relying on zero-copy CSV parsing for performance.

Performance

• Improving performance of xan flatten, xan view, xan to & xan network.
• Faster xan network -f nodelist in some cases.

v0.57.0

The temporal update.

Breaking

• xan select -n will not error anymore on empty inputs and, generally, empty files should not trigger selection errors when using commands with -n/--no-headers.
• xan heatmap -C/--cram becomes a flag accepting either auto, always or never.
• Dropping -C short flag for xan sort --cells (it could be confused with --columns or --check).
• Completely overhauled how datetimes work in moonblade.
• xan separate will not trim splitted values with some modes by default anymore.
• Dropping xan network --stats in favor of -f stats.
• -D becomes short flag for xan network --degrees instead of --disjoint-keys.
• xan separate --capture-groups is dropped in favor of -c/--captures & -C/--all-captures.
• Renaming xan search --breakdown shortflag to -b to allow for future -B/--before-context.

Features

• Adding xan matrix count & xan matrix adj.
• Adding front_coding window function.
• Timestamp support with xan plot -LT.
• Adding xan rename -n/--no-headers support for -p/--prefix & -x/--suffix.
• Adding xan from -f parquet (requires the parquet feature).
• Adding xan to latex.
• Adding xan top -L/--lexicographic.
• Adding xan heatmap flags: -w/--width, -F/--fill, -a/--align, -U/--unit, -Z/--show-normalized, -A/--ascii, -l/--label & -v/--values.
• Adding new gradients to xan heatmap.
• Adding range & repeat moonblade functions.
• Adding xan sort --columns.
• Adding xan view -T/--tee.
• Adding now, fractional_days, to_timezone, to_local_timezone, with_timezone, with_local_timezone, without_timezone, to_timestamp, to_timestamp_ms, from_timestamp, from_timestamp_ms, span, date & time moonblade functions.
• Better type inference with xan stats, and the type & types aggregation functions, now including more types for temporal values (zoned_datetime, datetime, date & time).
• Adding xan input -T/--tolerant.
• Adding xan separate --trim.
• Adding xan grep -B/--before-context & -A/--after-context.
• Adding xan network -f=components, -S/--simple, --union-find, --minify & --sample-size <n>.
• Adding xan plot --timezone.
• Adding xan hist --log shorthand flag for --scale=log.
• Adding log_dist sparkline column to xan stats -q output.
• Adding dist & log_dist aggregation functions.
• Adding xan search -L/--levenshtein <k> & -D/--damerau-levenshtein <k>.

Fixes

• Fixing xan separate automatic column prefix extraction.
• Fixing xan heatmap -n.
• Fixing xan heatmap --repeat-headers --cram always not repeating x-axis legend.
• Fixing correctness of xan plot -T and increase resolution to microseconds.
• Fixing moonblade column-related functions returning incorrect results wrt -n/--no-headers.
• xan search should now properly error when handling invalid utf-8 in relevant modes.
• Fixing xan search -iR & xan search -i --replacement-column.

Performance

• Improving performance of xan complete, xan top, xan plot -T & xan hist.
• Improving overall performance of xan network.
• Slightly optimizing xan vocab by allowing needless heap allocation & indirection.
• Improving performance and memory usage of xan separate.

Quality of Life

• Adding proper help to xan heatmap.

v0.56.0

Features

• Adding xan bisect.
• Adding xan flatten -N/--non-empty.
• Adding the soundex, refined_soundex & phonogram moonblade functions for phonetic encoding.

Fixes

• Fixing xan to (md|html) --no-headers.
• Fixing xan plot -R/--regression-line.

Quality of Life

• Adding xan to markdown as an alias for xan to md.
• xan flatten & xan view will stop masquerading trimmed empty cells as empty.

v0.55.0

Breaking

• Changing how xan separate generates default column names.
• xan from -f=(json|ndjson|jsonl) will now emit column in input order by default.
• Changing xan to -B/--buffer-size to --sample-size to harmonize flag names with xan from.

Features

• Adding the xan complete command.
• Adding an optional unit to ceil, floor, round & trunc moonblade function. E.g. floor to nearest decade: floor(year, 10).
• Adding basename & dirname moonblade functions.
• Adding parse_py_literal moonblade functions. Useful to deal with files dubiously serialized using pandas.
• Adding xan view --repeat-headers=(auto|always|never).
• Adding xan view --reveal-whitespace=(auto|always|never).
• Adding --color support to XAN_VIEW_ARGS.
• Adding xan from -f json --sample-size -1 to sample the whole file.
• Adding xan from -f json --single-object.
• Adding xan from --sort-keys.
• Adding xan to (json|ndjson|jsonl) --sample-size -1 to sample the whole file.
• Adding xan to (json|ndjson|jsonl) --strings flag.
• Adding xan separate --prefix.
• Adding xan heatmap -C short flag for --cram.
• Adding xan heatmap --repeat-headers.
• Adding rank, cume_dist, percent_rank and ntile window functions.
• Adding xan help --color.

Fixes

• Fixing xan select -ne incorrectly emitting headers.

Quality of Life

• xan view -p will not print bottom header anymore by default.
• xan view will not reveal problematic whitespace if output is not colored anymore, by default.
• Better xan hist error messages and help.
• Testing more file name variants when searching for a .gzi index.

v0.54.1

Fixes

• Fixing xan freq --groupby incorrectly unescaping group cells.
• Fixing help related to xan pivot & xan unpivot.
• Upgrading simd-csv to get safety fixes.