CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

[Unreleased]

[1.7.5] - 15-07-26

Added

• Add _recovery_point system space introduced in upcoming Tarantool 3.8 to crud.schema system spaces.
• Add crud.locate() method to find whether a tuple is in memtx or vinyl engine. Works for spaces managed by the enterprise cooler module.
• Support for mode, balance, prefer_replica and request_timeout options in crud.len.

Fixed

• Allow read-only operations (get, select, pairs, count, min, max) to execute successfully using healthy replicas when all cluster masters are down.
• Add tnt_crud_storage_nil_bucket_id_compat_total metric to track operations performed without bucket_ref (legacy router compatibility mode).
• Fixed storage compatibility with routers < 1.7.0 by handling nil bucket_id in get, update, and delete operations. Now storage skips bucket referencing and logs a rate-limited warning about reduced rebalance safety during rolling upgrades.

[1.7.4] - 12-02-26

Fixed

• Optimize crud.select() and crud.pairs() pagination with after cursor by using native after option on Tarantool 2.10+ for O(1) cursor positioning. Readview operations use native after only on Tarantool 3.x (#488).

[1.7.3] - 03-02-26

Added

• Add tnt_crud_router_cache_clear_ts metric to router to help user correctly disable safe mode in cluster.

Fixed

• Return bucket_ref error as array in crud.*_many methods.
• Move bucket_unref out of transaction.
• Prevent duplicate metrics if init function is called multiple times.
• Prevent creating duplicate triggers on _crud_settings_local space if init function is called multiple times.
• Fix deadlock in crud.schema() after schema reload error (#479).

[1.7.2] - 28-01-26

Fixed

• Remove tnt_crud_storage_safe_mode_enabled metric from router.

[1.7.1] - 16-01-26

Changed

• Stop marking/killing fast-mode iproto fibers on safe-mode switch. Storage ops are asserted by yield_checks in tests.
• Move switch to safe mode from on_commit trigger to on_replace trigger.
• vinyl spaces always work in safe mode.

[1.7.0] - 26-12-25

Added

• Implement "safe" mode to prevent writing data to wrong replicaset when vshard rebalance is in progress (#448).
• Auto switch to safe mode when rebalance process starts.
• Manual return to fast mode.

Fixed

• drop wrap_box_space_func_result to cut allocations and speed up storage calls.

[1.6.1] - 19-09-25

Added

• Support for vshard's request_timeout parameter for calls with mode = 'read'

Changed

• Auto-fill bucket_id for primary-key CRUD requests (get/update/delete) when the sharding index is the primary index and bucket_id is passed as box.NULL.

Fixed

[1.6.0] - 09-09-25

Changed

• bumped: vshard version in rockspec.

Added

• Role model support in CRUD is provided (commit).
• Validation of bucket_id. Invalid values now raise BucketIDError before routing (commit).

Fixed

• Fixed compatibility with cartridge 2.16.0.
• crud.schema no longer returns TCF system space _cdc_state.
• crud.schema no longer returns system space _gc_consumers with Tarantool 3.2+.
• crud.schema no longer returns tt system space _tt_migrations.
• Tests of schema with Tarantool 3.2+.
• Fixed bad error handling for method call.single
• Added support for working in 3.1 with data from 2.11, previously there was an error due to the inability to find the replicasets by name (commit).

[1.5.2] - 20-05-24

Fixed

• insert_many, insert_object_many, replace_many, replace_object_many, upsert_many, upsert_object_many operations no longer fail with ShardingHashMismatchError if a space has custom sharding info and every tuple/object in the request has bucket_id set (#437).

[1.5.1] - 27-04-24

Added

• Clarify license in the rockspec (#434).

[1.5.0] - 03-04-24

Added

• Asynchronous bootstrap support for storages (#412).
• Tarantool 3 roles for setting up crud routers and storages (#415).
• Ability to configure crud through Tarantool 3 roles configuration (#415).

Changed

• Explicitly forbid datetime interval conditions (#373).
• Storage initialization is now asynchronous by default for Tarantool 3.0+ (#412).
• Additionally check backoff error on storage info fetch (#427).

Fixed

• Working with datetime conditions in case of non-indexed fields or non-iterating indexes (#373).
• Precision loss for decimal conditions in case of non-indexed fields or non-iterating indexes (#373).
• Passing errors from storages for merger operations (crud.select, crud.pairs, readview:select, readview:pairs) (#423).
• Working with nil operand conditions in case of non-indexed fields or non-iterating indexes (#422).

[1.4.3] - 05-02-24

Fixed

• Compatibility with vshard configuration if UUIDs are omitted (#407).
• Compatibility with automatic master discovery in vshard (#409).
• Secondary conditions for index operands with operations >=, <=, >, < no longer cause missing part of the actual result for scan operations (crud.select, crud.pairs, crud.count, readview:select, readview:pairs) (#418).

[1.4.2] - 25-12-23

Added

• mode option for crud.min and crud.max (#404).

Fixed

• Compatibility with vshard 0.1.25 name_as_key identification mode for Tarantool 3.0 (#403).
• Propagating noreturn and fetch_latest_metadata options in case of intermediate nullable fields update for Tarantool 2.7 and older (#404).

[1.4.1] - 23-10-23

Changed

• Dropped external tuple merger and tuple keydef modules installation from the package build (#390).

Fixed

• Compatibility with Tarantool 3.0 binary protocol change (#390).
• Compatibility with Tarantool 3.0 tuple objects (#390).

[1.4.0] - 16-10-23

Added

• Space schema introspection API crud.schema (#380).

Changed

• deps.sh installs the vshard instead of the cartridge by default (#364). You could to specify an environment variable CARTIRDGE_VERSION to install the cartridge and run tests cases with it.
• Return explicit error for *_many call with no tuples/objects (#377).
• Quickstart section in the README.md focuses on usage with vshard instead of Cartridge (#366).

Fixed

• crud.readview resource cleanup on garbage collect (#379).
• doc/playground.lua does not work with Tarantool 3 (#371).
• Tests with Tarantool 3 (#364).
• VShard storage user have not execution rights for internal functions (#366).
• Compatibility with Tarantool 3.0 tuple objects (#387).

[1.3.0] - 27-09-23

Added

• Read view support for select and pairs (#343).

[1.2.0] - 07-06-23

Added

• Add noreturn option for operations: insert, insert_object, insert_many, insert_object_many, replace, replace_object, replace_many, insert_object_many, upsert, upsert_object, upsert_many, upsert_object_many, update, delete (#267).

Fixed

• Crud DML operations returning stale schema for metadata generation. Now you may use fetch_latest_metadata flag to work with latest schema (#236).

[1.1.1] - 24-03-23

Changed

• Yield on select/pairs storage tuple lookup (#312).
• Naming of DEV environment variable to TARANTOOL_CRUD_ENABLE_INTERNAL_CHECKS (#250).

Fixed

• Loaded functions misleading coverage (#249).

[1.1.0] - 13-03-23

Added

• Add versioning support (PR #342).

Fixed

• Pre-hotreload cartridge support (older than 2.4.0) (PR #341).
• Tarantool version-dependent features for 3.x (PR #344).

[1.0.0] - 02-02-23

Added

• Add timeout condition for the validation of master presence in replicaset and for the master connection (#95).
• Support Cartridge clusterwide configuration for crud.cfg (#332).

Changed

• Breaking: forbid using space id in crud.len (#255).

Fixed

• Add validation of the master presence in replicaset and the master connection to the utils.get_space method before receiving the space from the connection (#331).
• Fix fiber cancel on schema reload timeout in call_reload_schema (PR #336).

[0.14.1] - 10-11-22

Added

• skip_nullability_check_on_flatten option for insert_object, insert_object_many, replace_object, replace_object_many. false by default. By setting the option to true you allow setting null values to non-nullable fields, which can be useful if non-nullable field value is generated by sequence. Warning: there is no native support for sequences in sharded systems since each replicaset has its own sequence. If sequence field is a part of the sharding key (which is true by default), choosing the bucket id is the sole responsibility of the developer (#328).

Changed

• Rework NonInitialized error message to be more helpful for troubleshooting (#326).

[0.14.0] - 13-09-22

Added

• Support vshard_router option in operations for Cartridge vshard groups or non-default vshard routers (#44).

Changed

• Deprecate using space id in crud.len (#255).

[0.13.0] - 29-08-22

Added

• crud.storage_info function to get storages status (#229).

Fixed

• Fix specifying vshard sharding funcs (#314).

[0.12.1] - 21-07-22

Fixed

• Fetching invalid ddl configuration (sharding key for non-existing space) is no longer breaks CRUD requests (#308).
• ddl space record delete no more throws error if crud is used (#310).
• crud sharding metainfo is now updated on ddl record delete (#310).

[0.12.0] - 28-06-22

Added

• Batch insert/upsert operation crud.insert_many()/crud.insert_object_many()/ crud.upsert_many()/crud.upsert_object_many() crud.replace_many()/crud.replace_object_many() with partial consistency (#193).

[0.11.3] - 15-06-22

Changed

• Optimize crud.select() without conditions and with after.
• Behaviour of potentially long select and count calls: a critical log entry containing the current stack traceback is created upon such function calls — an user can explicitly request a full scan through by passing fullscan=true to select or count options table argument in which case a log entry will not be created (#276).
• Make select error description more informative when merger built-in module or tuple-merger external module is used in case of disabled/uninit storage (#229).

Fixed

• crud.select() if a condition is '<=' and it's value < after.

  Before this patch:

  tarantool> crud.select('developers', {{'<=', 'id', 3}}, {first = 10, after = rows[5]}).rows

  • • [2, 401, 'Sergey', 'Allred', 21]
    • [1, 477, 'Alexey', 'Adams', 20] ...

  After this patch:

  tarantool> crud.select('developers', {{'<=', 'id', 3}}, {first = 10, after = rows[5]}).rows

  • • [3, 2804, 'Pavel', 'Adams', 27]
    • [2, 401, 'Sergey', 'Allred', 21]
    • [1, 477, 'Alexey', 'Adams', 20] ...

• crud.select() filtration by a first condition if the condition is '>' or '>=' and it's value > after.

  Before this patch:

  tarantool> crud.select('developers', {{'>=', 'id', 5}}, {first = 10, after = rows[2]}).rows

  • • [3, 2804, 'Pavel', 'Adams', 27]
    • [4, 1161, 'Mikhail', 'Liston', 51]
    • [5, 1172, 'Dmitry', 'Jacobi', 16]
    • [6, 1064, 'Alexey', 'Sidorov', 31] ...

  After this patch:

  tarantool> crud.select('developers', {{'>=', 'id', 5}}, {first = 10, after = rows[2]}).rows

  • [5, 1172, 'Dmitry', 'Jacobi', 16]
  • [6, 1064, 'Alexey', 'Sidorov', 31] ...

• crud.select() results order with negative first.

• crud.select() if a condition is '=' or '==' with negative first.

  Suppose we have a non-unique secondary index by the field age field and a space:

  tarantool> crud.select('developers', nil, {first = 10})

  • metadata: [{'name': 'id', 'type': 'unsigned'}, {'name': 'bucket_id', 'type': 'unsigned'}, {'name': 'name', 'type': 'string'}, {'name': 'surname', 'type': 'string'}, {'name': 'age', 'type': 'number'}] rows:
    • [1, 477, 'Alexey', 'Adams', 20]
    • [2, 401, 'Sergey', 'Allred', 27]
    • [3, 2804, 'Pavel', 'Adams', 27]
    • [4, 1161, 'Mikhail', 'Liston', 27]
    • [5, 1172, 'Dmitry', 'Jacobi', 27]
    • [6, 1064, 'Alexey', 'Sidorov', 31]
  • null ...

  Before this patch:

  tarantool> crud.select('developers', {{'=', 'age', 27}}, {first = -10, after = rows[4]}).rows

  • [] ...

  After this patch:

  tarantool> crud.select('developers', {{'=', 'age', 27}}, {first = -10, after = rows[4]}).rows

  • • [2, 401, 'Sergey', 'Allred', 27]
    • [3, 2804, 'Pavel', 'Adams', 27] ...

[0.11.2] - 23-05-22

Added

• Make metrics quantile collector age params configurable (#286).
• Add separate latency_average and latency_quantile_recent fields to crud.stats() output (#286).

Fixed

• Preset stats_quantile_tolerated_error crud.cfg parameter is no more lost on configuration update (#284).

[0.11.1] - 06-05-22

Added

• Make metrics quantile collector tolerated error configurable (#281).

Changed

• Change metrics quantile collector default tolerated error from 1e-2 to 1e-3 (#281).

Fixed

• Requests no more fail with "Sharding hash mismatch" error if ddl set and bucket_id is explicitly specified (#278).

[0.11.0] - 20-04-22

Caution: Use CRUD 0.11.1 instead of 0.11.0. It fixes critical bug for some requests (see #278).

Added

• crud.count() function to calculate the number of tuples in the space according to conditions.
• Support bucket id calculating using sharding func specified in DDL schema or in _ddl_sharding_func space.
• Statistics for CRUD operations on router (#224).
• Integrate CRUD statistics with metrics (#224).
• Added LICENSE file into the repository (BSD-2-Clause).

Changed

• Optimize select with known bucket_id (#234).
• Deprecate manual sharding schema reload (#212).

Fixed

• Fix processing storage error for tuple-merger implementation of select/pairs (#271).
• Do not change input tuple object in requests.
• Add automatic reload of DDL schema (#212).

[0.10.0] - 01-12-21

Added

• CRUD operations calculates bucket id automatically using sharding key specified with DDL schema or in _ddl_sharding_key space. NOTE: CRUD methods delete(), get() and update() requires that sharding key must be a part of primary key.

Fixed

• Use tuple-merger backed select implementation on tarantool 2.10+ (it gives less pressure on Lua GC).
• DDL sharding key now can be extracted from select conditions even if there are no separate index.

[0.9.0] - 20-10-21

Added

• crud.len() function to calculate the number of tuples in the space for memtx engine and calculate the maximum approximate number of tuples in the space for vinyl engine.
• Testing: added integration with service coveralls.io (PR #195).
• Testing: added integration with luacov that allows to generate report with code coverage statistics (PR #195).

Changed

• Names of errors generated by CRUD operations have been unified (PR #184).
• Opmimize crud.select() / crud.pairs() for one replicaset case (say, when bucket_id is passed or deducible from conditions). It gives 13% boost on the case from #220 (PR #226).

Fixed

• Ignoring of an error in crud.pairs() (#144).
• Damaging of opts table by CRUD methods (#192).
• Ignoring of bucket_id option in crud.select()/crud.pairs() (#220).

[0.8.0] - 02-07-21

Fixed

• Invalid results for pagination queries when filtering on a part of a composite index

Added

• Added jsonpath indexes support for queries
• tuple-merger module updated to 0.0.2

[0.7.1] - 2021-06-21

Fixed

• Compatibility with Tarantool versions that don't support key_def and merger modules.
• Ignoring opts.first on crud.pairs call
• External keydef compatibility with built-in merger

[0.7.0] - 2021-05-27

Fixed

• Fixed error for partial result option if field contains box.NULL.
• Fixed incorrect crud results during reverse pagination without specifying batch_size.
• Fixed crud roles reload:
  • before this patch reload wasn't fair - reloading tuple.merger and tuple.keydef modules failed, and crud started to use crud.select.compat.select_old module with naive merger implementation;
  • fair reloading tuple.merger and tuple.keydef led to the error that was fixed by caching loaded module in the special global variable not cleaned on reloading roles;
  • ability of using tuple.merger and tuple.keydef modules now is checked by calling package.search, built-in merger and keydef modules are used if present in package.loaded. It allows to avoid ignoring errors on checking modules existing via pcall(require, '<module_name>').
• Fixed some cases when module ignored schema updates.
• Preserve "Bootstrap vshard" button in WebUI operable even if vshard roles are hidden explicitly.

Added

• cut_rows and cut_objects functions to cut off scan key and primary key values that were merged to the select/pairs partial result.
• Functions stop() for the roles crud-storage and crud-router.
• Option flag force_map_call for select()/pairs() to disable the bucket_id computation from primary key.
• crud.min and crud.max functions to find the minimum and maximum values in the specified index.
• Added support for jsonpath for select.

[0.6.0] - 2021-03-29

Fixed

• Fixed not finding field in tuple on crud.update if there are is_nullable fields in front of it that were added when the schema was changed for Tarantool version <= 2.2.

• Pagination over multipart primary key.

Added

• mode, prefer_replica and balance options for read operations (get, select, pairs). According to this parameters one of vshard calls (callrw, callro, callbro, callre, callbre) is selected

[0.5.0] - 2021-03-10

Fixed

• Fixed not finding field in tuple on crud.update if there are is_nullable fields in front of it that were added when the schema was changed.
• Fixed select crash when dropping indexes
• Using outdated schema on router-side
• Sparsed tuples generation that led to "Tuple/Key must be MsgPack array" error

Added

• Support for UUID field types and UUID values
• fields option for simple operations and select/pairs calls with pagination support to get partial result

[0.4.0] - 2020-12-02

Fixed

• Fixed typo in error for case when failed to get bucket_id
• Fixed select by part of sharding key equal. Before this patch selecting by equality of partially specified multipart primary index value was misinterpreted as a selecting by fully specified key value.
• Fixed iteration with pairs through empty space returned nil.

Added

• truncate operation
• iterator returned by pairs is compatible with luafun

[0.3.0] - 2020-10-26

Fixed

• Select by primary index name
• Fix error handling select with invalid type value
• Get rid of performing map-reduce for single-replicaset operations

Added

• crud-router Cartridge role
• bucket_id option for all operations to specify custom bucket ID. For operations that accepts tuple/object bucket ID can be specified as tuple/object field as well as opts.bucket_id value.

Changed

• CRUD-router functions are exposed to the global scope, so it's possible to call crud-operations via net.box.call
• crud.init is removed in favor to crud.init_storage and crud.init_router

[0.2.0] - 2020-10-07

Fixed

• Select with == conditions bugs
• Select with conditions by fields with collations

Added

• CRUD operations:
• replace
• upsert
• Output format for CRUD operations changed to set of rows and metadata
• Insert/replace/upsert methods now accept tuples. To process unflattened objects *_object methods are introduced.
• pairs accepts use_tomap flag to return tuples or objects

Changed

• checks is disabled for internal functions by default
• limit option is renamed to first
• Reverse pagination (negative first) is supported
• after option accepts a tuple

[0.1.0] - 2020-09-23

Added

• Basic CRUD operations:
  • insert
  • get
  • select
  • update
  • delete
• pairs function to iterate across the distributed space