Skip to content

CHANGELOG.md

Latest commit

 

History

History
793 lines (616 loc) · 35.1 KB

CHANGELOG.md

File metadata and controls

793 lines (616 loc) · 35.1 KB
Raw

Changelog

0.60.1 (provisional)

Fixes

  • Fixing xan rename with non-comma delimiters.
  • Fixing xan from -f=parquet not converting timestamp columns.

Quality of Life

  • Better moonblade error messages when values are not yet materialized.
  • Better xan count -H/--human-readable.

0.60.0

Breaking

  • Renaming xan from --path to --root.
  • Renaming xan separate -T/--txt to -L/--lines.

Features

  • Adding xan count -c/--check-alignment & -H/--human-readable.
  • Adding xan sort -e -z/--compress.
  • Adding xan sample -g -S/--sorted.
  • Adding xan from -f=(json|ndjson) --model.
  • Adding xan cat rows -I/--intersection & -U/--union.
  • Adding xan top -g -S/--sorted.
  • Adding the sort, dedup, flatten & flat_map moonblade function.
  • Adding support for list & map columns when using xan from -f=parquet.

Fixes

  • Fixing text wrapping across tool, especially xan flatten -w & xan flatten -F.
  • Fixing xan select -e & xan map plural clause flattening given list.
  • Fixing xan from -f=parquet not emitting correct column names for list & map columns.

Performance

  • Introducing a fast path for xan sort -e when input fits in a single chunk.
  • Faster xan from -f=ndjson & -f=parquet.
  • Amortizing xan sample allocations.
  • Improving performance of xan freq & xan p freq.

Quality of Life

  • Better record size estimation for xan sort -e.
  • Better xan flatten -c when string width cannot be computed correctly (because of emojis typically).

0.59.0

Breaking

  • Bumping MSRV to 1.85.0 and edition 2024.
  • Overhauling how xan scrape takes its inputs. It now targets HTML files on disk by default now.
  • xan plot --density-scale now defaults to log.
  • xan freq -X/--approx-algo & xan p freq -X/--approx-algo now default to heavy-keeper.
  • Dropping xan eval -S/--serialize. The default behavior of the command is now to output the serialized value.

Features

  • Adding xan search -x/--pattern-file.
  • Adding xan cat --glob, xan merge --glob, xan parallel --glob.
  • Adding the hostname moonblade function.
  • Adding xan scrape --paths, --path-column, --docs, --docs-column, -D/--stdin-doc & --glob.
  • Adding xan from -f=(json|ndjson) --path <path>.
  • Adding xan from -f toml & -f raw.
  • Adding xan spark --hide-all & --repeat-x-axis.

Fixes

  • Fixing moonblade string with bstring equality.
  • Fixing percentages shown by xan spark -P.
  • Fixing xan complete correctness in presence of duplicate values.
  • Fixing xan spark -c discretization & legend.

Performance

  • Improving performance of xan scrape.
  • Improving performance of xan from -f ndjson.

Quality of Life

  • Better legends & axis for xan spark.
  • Highlighting empty labels when using xan heatmap.

0.58.0

Breaking

  • Stopping to serialize moonblade lists either as joined by some separator or JSON. This was awkard, error-prone & potentially lossy. Use the join function manually to format output when required.
  • As per previous point, dropping xan scrape --sep.
  • Dropping implicit unary function calls in moonblade pipelines. This feature was not well-known, confusing (an indentifier, could be understood as a call in a pipeline, only if not in first position...), and mostly useless now that moonblade has had a proper dot operator.
  • xan plot -A/--aggregate does not take an expression anymore but has an automatic selection of two modes: sum and mean. It should also be faster.
  • Renaming the index function as row_index for clarity.
  • xan agg -C/--along-columns & -M/--along-matrix & xan groupby -C/--along-columns & -M/--along-matrix will not map current column index to the result of the index() function. The col_index() can be now used instead for this very purpose.
  • xan window -g/--groupby does not require the file to be sorted anymore. This means using -g/--groupby will now require the whole file to be buffered into memory by the command. The old behavior can still be used through the -S/--sorted flag, thus aligning the xan window command with the rest of the tool.
  • row_index will now error if the expression has no concept of row index, instead of returning nothing.
  • xan parallel -z/--compress now take the desired compression (either gzip or zstd).
  • Retiring the xan grep command in favor of xan search -Z/--fast-parser.
  • xan tokenize --keep short flag becomes -k instead of -K to harmonize with other commands.
  • Retiring the xan flatmap command in favor of xan explode -e.
  • Retiring the xan fuzzy-join command in favor of a consolidated xan join command.
  • Changing xan from -f txt -c <name> default to line instead of value.
  • Renaming xan join -L/--prefix-left & -R/--prefix-right short flags to -l & -r respectively to avoid colliding with the added -R/--reverse flag that can be used for merge joins.
  • Dropping xan plot -B/--bars. It never worked very well and its use-case will be redirected to xan spark.
  • Changing xan heatmap --width short flag from -w to -W so that adding a -H/--height flag remain consistent and avoids clashing with -h/--help.
  • Dropping xan heatmap --show-gradients in favor of xan help gradients.
  • Renaming xan search -A/--all flag to --every-column for clarity and avoid clash with -A/--after-context.
  • Dropping xan sort -U/--unstable. It was never used and the performance boost it supposedly provides cannot be observed.

Features

  • Adding xan parallel --dont-chunk.
  • Adding nullary col, col_index & header variants, to work with expression applied in series to multiple columns at once.
  • Adding prev_col & next_col functions.
  • Adding xan (search|filter) -B/--before-context & -A/--after-context.
  • Adding xan window -O/--overwrite.
  • Adding xan map -C/--along-columns.
  • Adding xan window -C/--along-columns.
  • Adding xan cat rows --raw, -P/--preprocess & -H/--shell-preprocess.
  • Improving xan select DSL star selectors. You can now do stuff like vec_*_count, *[1], vec_*[1] etc.
  • xan p -H/--shell-preprocess now works on Windows.
  • Adding native zsh completions (@apcamargo).
  • Adding xan dedup --u32.
  • Adding xan explode -e/--evaluate, -f/--evaluate-file, --pad & -k/--keep.
  • xan to npy is now able to stream.
  • Adding xan parallel top & xan top -p/--parallel, -t/--threads.
  • Adding xan network edgelist --range.
  • Adding xan network nodelist.
  • Adding the xan run command.
  • Adding xan view --name.
  • Adding xan join -S/--sorted, -R/--reverse & -N/--numeric.
  • Adding xan parallel --run & xan cat rows --run.
  • Adding xan to md -l/--limit.
  • Adding the xan spark command.
  • Adding xan stats -R/--report, --color, --cols, --sep.
  • Adding xan (freq|p freq) -X/--approx-algo.
  • Adding xan plot -D/--density-gradient, --density-scale, --hide-legend, --hide-x-axis, --hide-y-axis, --hide-all & -Q/--square.
  • xan separate will now avoid emitting columns with an empty name given to --into.
  • Adding xan separate --txt & --F/--filter.
  • Adding pow & sqrt scales.

Fixes

  • Fixing issues related to nested lambdas in expressions.
  • Fixing xan rename consistency regarding CRLF newlines and first row normalization when using -n/--no-headers.
  • Fixing xan map --overwrite --filter.
  • Fixing lead window function when there is not enough rows ahead.
  • Fixing xan network --format not being validated early enough.
  • Fixing xan explode -D/--drop-empty when selecting multiple columns.
  • Fixing xan merge -u row precedence.
  • Fixing xan join -D/--drop-key automatic selection when using --full.
  • Fixing granularity inference of xan plot -T.
  • Fixing xan from -f (json|ndjson) to emit empty outputs from empty inputs.
  • Fixing xan headers layout when input files have a very large number of columns (>= 1000).
  • Fixing arity validation of top, argtop, most_common & most_common_counts aggregation functions.

Performance

  • moonblade expressions are now faster overall and allocate more cautiously, thus saving memory.
  • Improving performance of xan transform, xan flatmap, xan agg & xan groupby.
  • Improving performance of xan rename.
  • Faster xan range.
  • Faster xan parallel -H/--shell-preprocess.
  • Faster xan tokenize words.
  • Adding fast path for xan explode when only a single column is selected.
  • Faster xan sort -e.

Quality of Life

  • xan plot will now display <empty> label in legends.
  • xan cat rows will now error when input have inconsistent columns.
  • Automatic column alignement with xan to md.
  • xan from now consider .log files as text lines.

0.57.1

Fixes

  • Fixing xan sort --check -n & xan dedup --check -n printed report.
  • Fixing xan parallel cat -nS.
  • Fixing CSV parsing rare edge case panics.
  • Fixing commands relying on zero-copy CSV parsing for performance.

Performance

  • Improving performance of xan flatten, xan view, xan to & xan network.
  • Faster xan network -f nodelist in some cases.

0.57.0

The temporal update.

Breaking

  • xan select -n will not error anymore on empty inputs and, generally, empty files should not trigger selection errors when using commands with -n/--no-headers.
  • xan heatmap -C/--cram becomes a flag accepting either auto, always or never.
  • Dropping -C short flag for xan sort --cells (it could be confused with --columns or --check).
  • Completely overhauled how datetimes work in moonblade.
  • xan separate will not trim splitted values with some modes by default anymore.
  • Dropping xan network --stats in favor of -f stats.
  • -D becomes short flag for xan network --degrees instead of --disjoint-keys.
  • xan separate --capture-groups is dropped in favor of -c/--captures & -C/--all-captures.
  • Renaming xan search --breakdown shortflag to -b to allow for future -B/--before-context.

Features

  • Adding xan matrix count & xan matrix adj.
  • Adding front_coding window function.
  • Timestamp support with xan plot -LT.
  • Adding xan rename -n/--no-headers support for -p/--prefix & -x/--suffix.
  • Adding xan from -f parquet (requires the parquet feature).
  • Adding xan to latex.
  • Adding xan top -L/--lexicographic.
  • Adding xan heatmap flags: -w/--width, -F/--fill, -a/--align, -U/--unit, -Z/--show-normalized, -A/--ascii, -l/--label & -v/--values.
  • Adding new gradients to xan heatmap.
  • Adding range & repeat moonblade functions.
  • Adding xan sort --columns.
  • Adding xan view -T/--tee.
  • Adding now, fractional_days, to_timezone, to_local_timezone, with_timezone, with_local_timezone, without_timezone, to_timestamp, to_timestamp_ms, from_timestamp, from_timestamp_ms, span, date & time moonblade functions.
  • Better type inference with xan stats, and the type & types aggregation functions, now including more types for temporal values (zoned_datetime, datetime, date & time).
  • Adding xan input -T/--tolerant.
  • Adding xan separate --trim.
  • Adding xan grep -B/--before-context & -A/--after-context.
  • Adding xan network -f=components, -S/--simple, --union-find, --minify & --sample-size <n>.
  • Adding xan plot --timezone.
  • Adding xan hist --log shorthand flag for --scale=log.
  • Adding log_dist sparkline column to xan stats -q output.
  • Adding dist & log_dist aggregation functions.
  • Adding xan search -L/--levenshtein <k> & -D/--damerau-levenshtein <k>.

Fixes

  • Fixing xan separate automatic column prefix extraction.
  • Fixing xan heatmap -n.
  • Fixing xan heatmap --repeat-headers --cram always not repeating x-axis legend.
  • Fixing correctness of xan plot -T and increase resolution to microseconds.
  • Fixing moonblade column-related functions returning incorrect results wrt -n/--no-headers.
  • xan search should now properly error when handling invalid utf-8 in relevant modes.
  • Fixing xan search -iR & xan search -i --replacement-column.

Performance

  • Improving performance of xan complete, xan top, xan plot -T & xan hist.
  • Improving overall performance of xan network.
  • Slightly optimizing xan vocab by allowing needless heap allocation & indirection.
  • Improving performance and memory usage of xan separate.

Quality of Life

  • Adding proper help to xan heatmap.

0.56.0

Features

  • Adding xan bisect.
  • Adding xan flatten -N/--non-empty.
  • Adding the soundex, refined_soundex & phonogram moonblade functions for phonetic encoding.

Fixes

  • Fixing xan to (md|html) --no-headers.
  • Fixing xan plot -R/--regression-line.

Quality of Life

  • Adding xan to markdown as an alias for xan to md.
  • xan flatten & xan view will stop masquerading trimmed empty cells as empty.

0.55.0

Breaking

  • Changing how xan separate generates default column names.
  • xan from -f=(json|ndjson|jsonl) will now emit column in input order by default.
  • Changing xan to -B/--buffer-size to --sample-size to harmonize flag names with xan from.

Features

  • Adding the xan complete command.
  • Adding an optional unit to ceil, floor, round & trunc moonblade function. E.g. floor to nearest decade: floor(year, 10).
  • Adding basename & dirname moonblade functions.
  • Adding parse_py_literal moonblade functions. Useful to deal with files dubiously serialized using pandas.
  • Adding xan view --repeat-headers=(auto|always|never).
  • Adding xan view --reveal-whitespace=(auto|always|never).
  • Adding --color support to XAN_VIEW_ARGS.
  • Adding xan from -f json --sample-size -1 to sample the whole file.
  • Adding xan from -f json --single-object.
  • Adding xan from --sort-keys.
  • Adding xan to (json|ndjson|jsonl) --sample-size -1 to sample the whole file.
  • Adding xan to (json|ndjson|jsonl) --strings flag.
  • Adding xan separate --prefix.
  • Adding xan heatmap -C short flag for --cram.
  • Adding xan heatmap --repeat-headers.
  • Adding rank, cume_dist, percent_rank and ntile window functions.
  • Adding xan help --color.

Fixes

  • Fixing xan select -ne incorrectly emitting headers.

Quality of Life

  • xan view -p will not print bottom header anymore by default.
  • xan view will not reveal problematic whitespace if output is not colored anymore, by default.
  • Better xan hist error messages and help.
  • Testing more file name variants when searching for a .gzi index.

0.54.1

Fixes

  • Fixing xan freq --groupby incorrectly unescaping group cells.
  • Fixing help related to xan pivot & xan unpivot.
  • Upgrading simd-csv to get safety fixes.
  • Fixing evaluation of moonblade commands related to column indexing.

0.54.0

The SIMD update.

Breaking

  • Bumping MSRV to 1.83.0.
  • Dropping xan plot -Y/--add-series. It is now possible to select multiple columns as <y> in xan plot <x> <y> instead.
  • Dropping the -C/--force-colors flag in flatten, heatmap, hist, plot and view in favor of the more standardized and flexible --color=(auto|never|always) flag.
  • xan join will now automatically drop joined columns from one the files when it is obviously safe to do so.
  • xan behead & xan rename do not normalize the output anymore to be as fast as possible.
  • The new SIMD CSV parser might not deal with CSV irregular cases the same way rust-csv did. In any case, xan input will still continue to use rust-csv.
  • xan slice -B/--byte-offset & xan slice -A/--accumulate are now mutually exclusive.
  • xan input has been overhauled.
  • Dropping xan count --sample-size.
  • Overhauling xan fixlengths to accept streams by shifting default from double-pass read to buffering the whole stream into memory.
  • xan plot --x-scale log & --y-scale log are now natural log. Use log10 for the base10 log as before.
  • Dropping xan reverse -m/--in-memory flag. Behavior is now automatically detected.
  • Dropping xan shuffle -m/--in-memory flag. Loading the file into memory is now the default. The xan shuffle -e/--external flag has been added if you want the old default behavior.
  • xan bins now outputs <empty> values instead of <nulls>.
  • Overhauling xan bins. The default is now to find nice boundaries for the bins. Use -e/--exact to revert to the old behavior. The default number of bins is now 10, and won't use Freedman-Diaconis rule by default. A -H/--heuristic flag has been added if you want to automatically select a suitable number of bins.

Features

  • Adding xan flatten -F/--flatter.
  • xan pivot can now target multiple columns.
  • Adding the xan grep command for fast but coarse filtering.
  • Adding xan search -f/--flag.
  • Adding xan map -F/--filter.
  • xan search -B/--breakdown now consolidates the results when multiple patterns have a same name.
  • Adding xan flatten --row-separator.
  • Adding xan flatten --csv.
  • Adding xan headers --color.
  • Adding the xan join <columns> <input1> <input2> arity as a convenience when joined column names are the same in both inputs.
  • Adding xan join -D/--drop-key=(none|both|left|right).
  • Adding xan fuzzy-join -D/--drop-key=(none|both|left|right).
  • Adding xan plot -A/--aggregate.
  • Adding support for plural selection clauses in both xan select -e & xan map e.g. xan map 'full_name.split(" ") as (first_name, last_name).
  • Adding xan search -P/--add-pattern.
  • Adding xan groupby -M/--along-matrix.
  • Adding xan groupby -T/--total.
  • Adding support for .ndjson & .jsonl files. Those are considered as headless TSV files with null byte quoting so you can easily use them with xan commands.
  • Adding out-of-the-box support for .vcf, .sam, .bed, .gtf & .gff2 files.
  • Adding a xan cat cols alias to xan cat columns.
  • Adding zstd support.
  • Adding earliest & latest moonblade functions.
  • Adding xan dedup -f/--flag.
  • Adding -k short flag for xan dedup --keep-duplicates, and -C short flag for xan dedup --choose.
  • Adding xan fixlengths -H/--trust-header.
  • Adding xan separate.
  • Adding full log scale support to xan plot.
  • Adding xan hist --scale.
  • xan window is now able to run total aggregations.
  • Adding thousands_sep, comma and significance kwargs to numfmt moonblade function.

Fixes

  • Fixing xan dedup --check bug where the first record was ignored.
  • Fixing xan hist -D when a same date is found multiple times.
  • Fixing xan from -f xls datetime conversion.
  • Fixing xan flatten & xan view when column names contain line breaks.
  • Fixing invalid argument parsing error being printed to stdout instead of stderr.
  • Fixing xan progress SIGINT corrupting output.
  • Fixing xan enum -A/--accumulate.
  • Fixing xan from -f tar when tarball archive is not gzipped.
  • Fixing min & max moonblade function when passing a list of numbers.
  • Fixing xan flatten -H edge cases.
  • Fixing commands requiring seekable streams accepting unindexed compressed files by error.
  • Fixing xan plot --count --y-scale log.

Performance

  • Wildly improving performance of most of xan commands by leveraging a novel SIMD CSV parser/writer.
  • Improving performance of xan from -f txt & xan from -f npy.
  • Improving memory footprint of hash-based commands (e.g. frequency, groupby, dedup etc.).
  • Improving performance of xan progress, xan range, xan enum, xan behead, xan rename.

Quality of Life

  • xan parallel cat now flushing more consistently.
  • Better highlighting of problematic strings in xan flatten, xan view & xan headers.
  • xan parallel will now generally stop as soon as an error is detected in a subprocess and cleanly report errors.
  • Better argv parsing error UX in general.
  • The -p flag will now avoid going further than 16 to avoid issues on server with many CPUs where hogging the resources is an issue and where using too much threads at once could hurt performance. The -t flag remain available to tweak the number of threads.
  • xan hist will now dim bars having a 0 count so you can easily distinguish them from non-empty bars.

0.53.0

Breaking

  • xan partition now normalizes filenames to lowercase to correctly deal with case-insensitive filesystems. xan partition also gets a related -C/--case-sensitive flag.

Features

  • Adding all and any moonblade higher-order functions.
  • Allowing moonblade printf function to be called with lists.
  • Adding -f/--evaluate-file flag to map, filter, flatmap & transform commands.
  • Adding xan map -O/--overwrite.

Fixes

  • Fixing xan top -T/--ties edge case.
  • Fixing broken pipe panics for some commands.
  • Dropping remnant dbg! macro when reading files in reverse.
  • xan flatten -H now correctly working on more data types.

Performance

  • Using jemallocator for musl builds.

Quality of Life

  • Better moonblade printf function error messages.

0.52.0

Breaking

  • xan search --count will not emit rows with 0 matches anymore unless --left is used.

Features

  • xan transform is now able to work on a selection of columns, rather than on a single column.
  • Adding the xan unpivot command.
  • Adding the xan pivot command.
  • Adding xan join --semi & xan join --anti commands.
  • Adding xan slice --raw.
  • Adding default expression argument to lead & lag window functions.
  • Adding shlex_split, cmd and shell moonblade functions.
  • Adding aarch64-apple-darwin and aarch64-unknown-linux-gnu to CI builds.
  • Adding to_fixed moonblade function.
  • Adding decimal places optional argument to ratio & percentage aggregation functions.
  • Adding frac & dense_rank aggregation functions to xan window.

Fixes

  • Loosening xan partition sanitizer to allow hyphens, dashes and points.
  • Fixing xan parallel --progress display.
  • Fixing logic error in xan search -B when using without --left.
  • Fixing xan parallel cat when working on file chunks with -P or -H.
  • Fixing moonblade list/string slicing with some combinations of negatives indices.
  • Fixing moonblade split function not using regex patterns properly.
  • Fixing moonblade parsing wrt regex patterns and comments (using a regex pattern containing # was not possible).
  • Fixing lead window aggregation function when working on any column that is not the first one.
  • Fixing xan view -S/--significance being overzealous, especially wrt integers.

Performance

  • Improving performance of xan parallel when working on file chunks.

Quality of Life

  • xan headers now report more useful information when files have diverging headers.
  • Better error messages for read_json and parse_json moonblade functions.
  • xan view -p will not engage pager when input errored or is empty.
  • xan select -e & -f become boolean flags instead of error-inducing invocation variants.

0.51.0

The parallel update.

Breaking

  • Dropping undocumented xan index and related interactions (in xan count, xan sample, xan slice & xan split --jobs).
  • Dropping now useless coalesce moonblade function.
  • xan split now accepts its output directory as an optional flag.
  • xan partition now accepts its output directory as an optional flag.
  • xan split -s becomes xan split -S to avoid confusion with the -s/--select flag used everywhere else.
  • Dropping useless xan count --csv flag.
  • Dropping xan freq -t/--threshold. Use xan freq | xan filter 'count >= n' instead.
  • Adding xan slice -I/--indices taking care of xan slice -i polymorphism taking multiple indices before.
  • xan parallel freq now follows xan freq behavior regarding limits.
  • Dropping xan url-join & xan regex-join. Both commands have been merged into a new xan fuzzy-join command using the -u/--url-prefix & -r/--regex flags respectively.
  • xan from --sheet becomes --sheet-name and is no longer the default. --sheet-index 0 becomes the default.
  • Dropping xan foreach. It is not distinctive enough as you can use xan map for the same purpose and get useful information about the results of evaluated side effects or write to /dev/null.
  • Renaming xan agg --cols to xan agg --along-rows.
  • Changing cell placeholder to anonymous _ value in xan agg -R/--along-rows.
  • Dropping moonblade commands -E/--errors flags. A lot has changed since they were created. They will be reevaluated in the future if required. You can rely on the try & warn moonblade functions instead, for now.
  • Dropping xan select -A/--append. Latest xan map is now actually equivalent to xan select -eA.
  • Changing xan map to accept a selection expression able to create multiple columns at once rather than a single expression and a column name. This means xan map 'expr' col_name becomes xan map 'expr as col_name'.

Features

  • Adding xan count -a/--approx.
  • Adding xan slice --end-byte.
  • Adding xan slice -S/--start-condition & xan slice -E/--end-condition.
  • Adding xan slice -L/--last.
  • Allowing -n/--no-headers and -d/--delimiter flags to appear before subcommands.
  • Adding backtick quoted strings to moonblade.
  • Adding moonblade printf function.
  • Adding moonblade pad, lpad & rpad functions.
  • Adding xan select -f/--evaluate-file.
  • Adding multi-member gzip files support (to handle files compressed with bgzip notably).
  • Adding xan split -f & xan partition -f short flag for --filename.
  • Adding xan split -c/--chunks & xan split --segments.
  • Adding xan sample -§/--cursed.
  • Adding xan search -B/--breakdown and the related --name-column flag.
  • Adding CSV file chunking capabilities to xan parallel.
  • Adding xan from md.
  • Adding xan parallel map.
  • Adding -p/--parallel & -t/--threads to count, freq, stats, search, agg & groupby commands.
  • Adding piped column access to expression given to xan flatmap -r.
  • Adding xan rename -R/--replace & xan rename -x/--suffix.
  • Adding xan parallel freq -l/--limit, -A/--all, -a/--approx & -N/--no-extra.
  • Adding xan search -U/--unique-matches & --sep & --left.
  • Adding parallelization through novel file segmentation of files compressed with bgzip when a .gzi index can be found.
  • Adding the xan window command for window aggregations like rolling averages, cumulative sums, lags etc.
  • Adding xan help window.
  • Adding xan head & xan tail as aliases over xan slice -l & xan slice -L respectively.
  • Adding xan from --sheet-index & --list-sheets.
  • Adding xan flatten -H/--highlight & -i/--ignore-case.
  • Adding xan agg -C/--along-cols & xan agg -M/--along-matrix.
  • Adding xan groupby -C/--along-cols.
  • Adding support for xan search -l -p -t.
  • Adding rms moonblade aggregation function.
  • Adding xan scrape -E/--encoding.
  • Adding CDX files support.
  • Adding regex moonblade function.
  • Adding header, col_index & col_index? moonblade functions.
  • Adding find & find_index moonblade functions.
  • Adding -l/--limit support to xan search -p & xan filter -p.
  • Adding xan pivot-longer alias to xan unpivot.
  • Adding xan pivot-wider alias to xan pivot.

Fixes

  • Adding missing highlight for NULL values in xan view & xan flatten.
  • Fixing moonblade slicing wrt negative indexing and nontrivial inner expression.
  • Fixing moonblade get function for bytes.
  • Fixing xan sort -e skipping first record of each chunk.
  • Fixing xan sort -e stability.
  • More accurate xan sort -e memory usage calculations.
  • Fixing xan transform -n.
  • Fixing xan view -g -s.
  • Fixing moonblade concretization wrt branching.
  • Fixing xan behead -o and xan behead -Ao.
  • Reorganizing xan help functions.
  • Fixing lexicographic extent merging in xan parallel stats.
  • Fixing xan to md width alignment.
  • Renaming xan parallel --shell-preprocessing short flag to be -H because it was being overriden by -S/--source-column.
  • Adding missing subcommand completions for xan parallel & xan cat.
  • Better default threads count heuristics.
  • Better xan plot -T date parsing.
  • Fixing xan search replacements when using the -s/--select flag with a non-full selection.
  • Adding the xan view -r/--right flag to force right alignment for a selection of columns.
  • Fixing xan flatten broken pipe panics when piped.
  • Fixing xan plot -R/--regression-line when linear function endpoints are out of bounds.
  • xan parallel early exits when a target file does not exist.
  • Fixing moonblade list slicing.
  • Fixing cols() & headers() moonblade functions without arguments.
  • Fixing cols() & headers() not working with dynamic arguments.
  • Fixing moonblade indexing parsing.
  • Fixing aggregation arity validation.
  • Fixing xan agg & xan groupby behavior wrt -n/--no-headers.
  • Fixing shortcircuiting of and and or moonblade functions.
  • Fixing issue with degenerate cases in xan bins --nice.
  • Fixing bin allocation in xan bins --nice.
  • Fixing xan bins --nice first and last bound to stick to min & max.
  • Fixing negative indexing with col*(name, pos) moonblade functions.
  • Fixing argmin & argmax parallel stability.
  • Fixing panic with xan plot when using log scales and min/max are <= 0.

Performance

  • Switching hashmaps to ahash.
  • Optimizing moonblade pipelines with more than a single underscore substitution.
  • Improving xan reverse performance.
  • Reducing memory footprint of aggregators.
  • Optimizing xan select -e allocations.

Quality of Life

  • Prepending xan subcommand to error messages.
  • Better error messages when moonblade expressions cannot be parsed.
  • Displaying number of threads actually used when using xan parallel.
  • xan view now automatically right-align columns containing only integers.
  • Better moonblade casting errors.
  • xan bins formatted bound will now be padded for better readability.

0.50.0

Features

  • Adding moonblade function log2, log10 and support for custom base with an optional argument of log.
  • Adding \0, \x.. and \u{......} literals to moonblade strings.
  • Adding moonblade function col?.

Fixes

  • Better color support for legacy Windows terminals.
  • Fixing to_timezone function with UTC timestamps on some platforms where jiff is built "bundled".
  • Fixing moonblade commands (e.g. filter, map etc.) when using -n/--no-headers.

0.49.3

Fixes

  • Adding missing -M/--hide-info support with XAN_VIEW_ARGS.
  • Pinning MSRV to 1.81.0 in CI builds to avoid Windows Defender false positives.

0.49.2

Fixes

  • Overhauling & fixing CI builds.

0.49.1

Fixes

  • Fixing compilation with musl.
  • Fixing xan cat rows -n.

0.49.0

Breaking

  • Dropping mostly useless -p/--parallel & -c/--chunk-size flags in xan agg & xan groupby. They were only useful when the inner aggregated expression was costly (i.e. reading files) and you can use xan map -p upstream for this instead. See also xan parallel (agg | groupby) if you want to work in parallel over multiple files.

Features

  • Adding xan input --tabs.
  • Adding xan input -H/--skip-headers.
  • Adding xan input -L/--skip-lines.
  • Adding xan input -R/--skip-rows.
  • Adding xan input --vcf, --gtf, --gff.
  • Adding xan search -R/--replace & xan search --replacement-column.
  • Adding xan rename -S/--slugify.
  • Adding moonblade function sum.
  • Adding support for .psv, .ssv & .scsv file extensions.
  • Adding xan headers -s/--start.
  • Adding xan to txt.
  • Adding xan behead -A/--append.
  • Adding xan hist -G/--compress-gaps.
  • Adding xan agg --cols.

Fixes

  • xan view --no-headers now automatically toggles --hide-headers.
  • xan from correctly decompress some gzipped formats.
  • xan fill -v correctly fills empty cells at beginning of files.
  • xan parallel -t will not use more threads than inputs.
  • Fixing xan stats panicking when encountering NaN values.
  • Allowing tabs to be understood as whitespace in moonblade expressions.
  • Fixing xan join --cross when joined files don't have the same number of columns.
  • Adding missing -n/--no-headers & -d/--delimiter to xan to.
  • Fixing xan progress -B with gzipped files.
  • Adding missing -C/--force-colors to xan plot.

0.48.0

Breaking

  • Dropping xan union-find. See xan network now for similar utilities.
  • xan explode --singular becomes xan explode --singularize.
  • xan implode --plural becomes xan implode --pluralize.

Features

  • Adding scraping selectors prev_sibling, next_sibling, find_ancestor & last.
  • Adding xan scrape images.
  • Adding xan search -u/--url-prefix.
  • Adding xan network -f nodelist and xan network --degrees.
  • Adding various format aliases to xan from.
  • Adding xan explode -D/--drop-empty.
  • Adding xan from tar.
  • Adding xan url-join.

Fixes

  • More inflection cases supported in both xan explode -S and xan implode -P.
  • Better error reporting with xan scrape.
  • Fixing xan scrape processing values when selection is empty.
  • Fixing parent scraping selector.
  • Adding missing higher-order functions documentation to xan help.

Performance

  • Improving performance of xan explode.
  • Improving performance of xan implode.

0.47.1

Fixes

Fixing CI builds.

0.47.0

Breaking

  • Moonblade strftime() and all other date formatting functions such as ymd() do not support timezones any more, see to_timezone and to_local_timezone instead.

Features

  • Adding moonblade function to_timezone and to_local_timezone
  • Improving xan help cheatsheet.
  • Adding moonblade function try.
  • Adding moonblade functions int & float.
  • Adding moonblade functions lru & urljoin.
  • moonblade now tolerates line-breaks as whitespace when parsing. This makes it possible to write your expressions on multiple lines.
  • moonblade now accepts comments starting with # .
  • Adding xan scrape command.
  • Adding xan help scraping subcommand.
  • Adding moonblade function html_unescape.

Fixes

  • Fixing xan flatten -w wrt line breaks.
  • Fixing underscore expansion when contained in map & list expressions.
  • Fixing xan to md cell escaping.

Performance

  • Decrease moonblade-related commands memory consumption.

0.46.2

Fixes

  • Fixing Windows compilation.

0.46.1

Fixes

  • Fixing Windows compilation.

0.46.0

Breaking

  • Moonblade concat operator becomes ++ instead of ..
  • Overhauling moonblade cli documentation:
    • Dropping --functions, --cheatsheet and --aggs everywhere
    • Adding a proper xan help command
  • Dropping xan glob.
  • Adding access & call operator to moonblade:
    • Member access: map.name (same as get(map, "name"))
    • Function call: string.len() (same as len(string))
  • Help is now printed in stdout (typically when using the -h/--help flag).

Features

  • Adding xan to html.
  • Adding xan to md.
  • Adding xan to npy.
  • Adding xan from npy.
  • Adding -R/--regression-line to xan plot.
  • Adding xan t alias for xan transpose.
  • Adding map substitution to fmt moonblade function.
  • Adding xan sort -C/--cells.
  • Adding xan search --count --overlapping.
  • Adding xan tokenize words -F/--flatmap.

Fixes

  • Fixing xan search --pattern-column.
  • Fixing autocompletion with range and multiple selection.
  • Fixing url highlighting in xan view and xan flatten.
  • Fixing datetime highlighting in xan view and xan flatten wrt Z-terminated timestamp formats.
  • Fixing stats date & url inference.
  • Fixing moonblade support for Z-terminated timestamp formats.
  • Fixing xan plot -T granularity inference.
  • Fixing missing fractional seconds to default datetime serialization.
  • Fixing bin allocation with xan bins --nice.
  • Fixing xan search --patterns -i.
  • Fixing xan search -r -i --patterns --count results.

Performance

  • Optimizing aggregator memory consumption.