Using targets with in place operations and data.table

[al-obrien]

Prework

• [X ] I have read and understand help guide.
• [ X] I understand and agree to contributing guide.
• [X ] I am confident this issue is a bug in targets itself, not a user error or a bug in a different package. I have done troubleshooting on my own to verify this. User questions which are not bugs in targets should be posted as discussions instead of bug reports.
• [X ] I am posting a runnable reprex to help others troubleshoot.

Description

Loading a data.table object from disk (e.g. serialized qs2 object), it does not automatically assign a new valid memory address. This led to a confusing bug when trying to do some in-place operations for memory efficiency.

It seems having a custom format the enforces setDT or similar touchpoint to reset the pointer could address the issue; however, I dont know if this is the best way to implement the fix when perhaps it can be built on top of the existing qs format (or other formats for that matter).

What Im trying to get at ultimately is: what is the targets approach for dealing with data.table across multiple targets while still allowing for in place operations to occur within a targets step.

reprex

Memory address not valid

This will not allow in place changes to occur right away

tar_dir({ 
    tar_script({
        library(targets)
        library(tarchetypes)
        tar_option_set(format = 'qs', packages = c('data.table', 'qs2'))
        list(
            tar_target(mydt, as.data.table(iris))
        )
    })
    tar_make()
    tar_load(mydt)
    testinplace <- function(data, val) {data[,testdata := val]}
    
    print(data.table:::selfrefok(mydt))
    testinplace(mydt, 'first'); print(mydt)
    
    print(data.table:::selfrefok(data.table::setDT(mydt)))
    testinplace(mydt, 'second'); print(mydt)

})

With the helper loader

This will allow inplace to occur

tar_dir({ 
    tar_script({
        qs_dt_format <- targets::tar_format(
            read = function(path) {
                res <- qs2::qs_read(path)
                data.table::setDT(res)
                res
            },
            write = function(object, path) {
                qs2::qs_save(object, path)
            }
        )
        
        library(targets)
        library(tarchetypes)
        tar_option_set(format = 'qs', packages = c('data.table', 'qs2'))
        list(
            tar_target(mydt, as.data.table(iris), format = qs_dt_format )
        )
    })
    tar_make()
    tar_load(mydt)
    testinplace <- function(data, val) {data[,testdata := val]}
    
    print(data.table:::selfrefok(mydt))
    testinplace(mydt, 'first'); print(mydt)
    
    print(data.table:::selfrefok(data.table::setDT(mydt)))
    testinplace(mydt, 'second'); print(mydt)

})

[wlandau]

@al-obrien, this reflects an inherent difference between the goals and philosophy of targets and those of data.table. targets is functional and declarative, so it treats each return value as immutable. data.table supports mutable objects by default in order to avoid deep copies. Your setDT() workaround is appropriate, either in custom formats like the one you proposed, or applied in the interactive session after an informal tar_read() or tar_load().

[al-obrien]

Thanks for confirming the work around. I like using data.table with targets especially if the object is large and can benefit from in place operations within a function to minimize RAM use whenever possible, even if it requires a copy at the start.

I based the workaround on the help page for tar_format() but looking at the built in formats, what I wrote seemed very simplistic in comparison. Just wondering if there's any other resources on constructing these formats?