Migrate Workflow from R Kumu to PyKumu (Python)

[phuong808]

Project Overview

The purpose of this project is to migrate an existing R analysis notebook to Python, verify that the Python implementation produces equivalent results, refactor the workflow into a modular API, and integrate the Kaiaulu data processing pipeline to support a complete and automated data analysis process.

Milestone 1

• Fully migrate the existing R notebook into a Python notebook and confirm that both implementations produce identical outputs.

Milestone 2

• Refactor the Python notebook into reusable Python modules and expose its core functionality through an API.

Milestone 3

• Integrate the Kaiaulu data file into the Python workflow by parsing and preprocessing the data prior to performing causal analysis.

Task List

Sections needed to be migrated from R notebook

• Feature Engineering
  • Formatting Data Types
    • CVE Data Type
  • Feature Renaming
  • Missing Data Handling
    • Convert "start" to Unix Timestamp
  • 1-Time Lag Features
  • Remove Short CVEs
  • Addressing Determinism and High Intercorrelation Among Features
• Non-Parametric Transformation
  • Binarized CVE Indicators
  • Add Null Features
  • Keep only 5 null indicator features
• FGES Null Variable Search
• BOSS Null Variable Search
• Deriving the 1 PNEF Threshold
  • Graph Examination
  • Deriving 1 PNEF
• Non-Null Causal Search
  • Domain Knowledge Causal Search without Null Variables
  • Causal Search
• Applying 1PNEF Threshold
  • Applying 1PNEF Threshold
• Results
  • Full Causal Graph 1-PNEF Trimmed
  • Sub-Graphs of Effort Variables and Parents

[phuong808]

Hi @carlosparadis,

Sorry for the delay and non communications for the last two days, based on the next steps you have provided, I have worked on reviewing code snippets you provided in the other discussion about the 3 thresholds that I am prioritizing for this week's issue. This 3 thresholds being score thresholds, bootstrapping thresholds, and FGES search thresholds. And the thresholds compared to python is similar in pytetrad based on code

search = search.TetradSearch(data) 
search.use_sem_bic(penalty_discount=2) 
search.set_bootstrapping(numberResampling=10, percent_resample_size=100, with_replacement=True, 
                         add_original=True, resampling_ensemble=1, seed=4130213) 
search.run_fges()

In terms of what I want to look for in the JAVA API documentation is only the relevant things to the 3 thresholds and that can be found for example

search.use_sem_bic(penalty_discount=2)

import pytetrad.tools.TetradSearch as search

I should be looking for pytetrad.tools.TetradSearch. I can also look for all the parameters by looking in TetradSearch.py and one of the example function parameters are

def use_sem_bic(self, penalty_discount=2, structurePrior=0, sem_bic_rule=1, singularity_lambda=0.0): 
     self.params.set(Params.PENALTY_DISCOUNT, penalty_discount) 
     self.params.set(Params.SEM_BIC_STRUCTURE_PRIOR, structurePrior) 
     self.params.set(Params.SEM_BIC_RULE, sem_bic_rule) 
     self.params.set(Params.SINGULARITY_LAMBDA, singularity_lambda) 
     self.SCORE = score_.SemBicScore() 

And for the Kumu .Rmd is doing, its concatenating all the numbers in the data file in the csv file into a string and pass them into Causal CMD.

Also I looked the documentation for contributing code and pull requests in the kaiaulu folder, I was able to make some progress on the code and wanted to get feedback on code review so I was wondering what is the process for contributing code and creating/naming branches in the pykumu repository since there are no other contributed yet?

Thank you,
Phuong

[carlosparadis]

Hi Phuong,

The daily communication is only necessary to the extent you are still understanding your task, if that is clear then no worries.

In regards to the code, for now you can just attach the notebook here. We can go over the PR process next call.

Do you have a notebook that can execute the FGES using the data I gave you? If so, can you also attach the outputs?

[phuong808]

Hi @carlosparadis,

I don't have working code yet at the moment but I am still in the working on it and will continue to work to get as much done as I can by tomorrow morning and I can attach what I have to this issue.

Thank you,
Phuong

[phuong808]

Hi @carlosparadis,

I wanted to send you an update on what I've been working on with migrating FGES causal search from R Kumu to PyKumu.

The notebook starts with importing all the necessary libraries. I imported pandas and numpy for data handling, the os module for file operations, and the TetradSearch class from pytetrad.tools which is the main tool that allows us to yse the Tetrad causal discovery algorithms. I then loaded the null_variable_dt.csv file which contains 4,870 rows and 138 columns. After checking the data, I converted all the columns to float64 data types because the FGES algorithm needs continous variables to work properly.

Once the data was ready, I moved on to configuring the SEM-BIC scoring function. I created a TetradSearch object and passed in the dataset, then configured the SEM-BIC score with specific parameters. I set the penalty discount to 2, used sem_bic_rule 1, set structurePrior to 0 which means no prior knowledge about the structure, and kept singularity_lambda at 0.0. The SEM-BIC score is really important because it's what the algorithm uses to evaluate how good different causal graphs are. Next up was configuring the bootstrapping parameters. I set it to do 500 bootstrap iterations, with each resample using 90% of the original data. I set a random seed of 32 to make sure the results would be reproducible if someone runs it again. I also set it to include the original dataset and use sampling with replacement, and used ensemble method 1. The notebook explained that this means FGES will actually run 501 times total, once on the full dataset and then 500 times on bootstrap resamples. This is important for getting robust results that aren't just based on one specific sample.

After bootstrapping was configured, I set the FGES parameters. I set time_lag to 0 because we're not doing time series analysis, and I turned verbose mode to True so the algorithm would give us detailed output while it's running. The cell that runs FGES set parameters like max_degree to 1000, faithfulness_assumed to True, symmetric_first_step to True, and parallelized to False. The algorithm took about 91 seconds to run because it had to go through all 501 iterations.

After FGES finished running, the next cells extracted the results. First, I got the graph string representation which shows the causal graph in a text format. Then I extracted the adjacency matrix, which is a 138 by 138 matrix showing all the connections between variables. The notebook displayed the first 10x10 section of the matrix and explained that the numbers represent different endpoint types - 0 means no endpoint, 1 is a circle endpoint, 2 is an arrow endpoint, and 3 is a tail endpoint. After that, I ran a cell that converted the adjacency matrix into an edge list, which is basically a more readable format that just lists out all the edges. The algorithm found 101 total edges, and the notebook displayed the first 20 edges so I could see what the causal relationships looked like.

I created a directory called fges_results and saved three files - the adjacency matrix as a CSV file which came out to 39 KB, the edge list as a CSV file which was 2 KB, and the graph string as a text file. I run into one error at the very end. When the notebook tried to save the graph string to a text file, it gave me this error: "TypeError: write() argument must be str, not java.lang.String". I'm going to continue to work on this issue on the time being.

I'm also having problems attaching the .ipynb file onto this comment. it keeps saying failed to upload filles. Should I attach screenshots of the .ipynb file instead?

Thank you,
Phuong

[carlosparadis]

Hey Phuong,

Could you please format your comment with a few "Enters"? It is impossible to read.

[phuong808]

Hi @carlosparadis,

I have revised the format of the previous comment, can you please advise me on what I should do next?

Thank you,
Phuong

[carlosparadis]

Hi,

Thanks. Please fork this repo to your own account, commit the code there and link me here. In the future, it is easier if you just do that instead of describing in words your notebook!

Tomorrow I will go over the permissios to this and the other repos.

it gave me this error: "TypeError: write() argument must be str, not java.lang.String".

You need to find out how to convert a Java string object to a Python string object. The answer you seek is likely some function from JPype.

[phuong808]

Hi @carlosparadis,

Thank you for your guidance. I have forked this repo and pushed my notebook changes to branch issue-1. I have added you as collaborator, can you please accept my invite and review the changes made?

Thank you,
Phuong

[carlosparadis]

Hi,

The notebook is 80MB+

https://github.com/phuong808/pykumu/blob/0d4f8e46445ac8090398206a11e40e62a4c5a3df/fges_causal_search_migration.ipynb

I can't even render it on my browser to see it, so I can't review it.

I suspect you may be printing entire tables in the notebook. Please use head() functions to only display a portion of the tables and resubmit.

When you do, please link here so I know I am looking at the right version.

Thanks!

[phuong808]

Hi @carlosparadis,

I have made changes to the notebook so it should be readable now, here is the link to the notebook: https://github.com/phuong808/pykumu/blob/Issue-1/fges_causal_search_migration.ipynb

Thank you,
Phuong

[carlosparadis]

Well done! This is what I was looking for. Here's something I wanted you to test: The file you saved (.txt) or any other should be able to be loaded on the Tetrad GUI. Try to run the GUI (i.e. the jar you are using is likely it), and then see if you can load the file you are saving on the GUI.

Send me a screenshot here if you are able to. If not, we want to find how to save the file after running the search so we can be loaded on the other tool. This is useful for a collaborator we have that prefers using the GUI.

Then we will work on refactoring the code in an API but we can discuss that on call.

[phuong808]

Hi @carlosparadis,

I was able to load the graph onto Tetrad GUI. Below is a screenshot of how the graph looks like on the GUI.

Thank you,
Phuong

[carlosparadis]

@phuong808 well done. Can you confirm what is the file that is being loaded there? Is it the .txt?

I would also like you to confirm if there is any way we can export a json out of the code you have instead of a .txt so it can be loaded on Tetrad.

Regarding the visualization:

Tetrad GUI should have a way to display the graph you loaded as a circle, could you try displaying in that manner and paste a new screenshot in a new comment?

And lastly, could you check if there is any way to plot the graph you see on the Python side? Don't spend too much time looking on this visualization piece, I can ask my collaborator if he knows of any otherwise.