Performance Discrepancy in Reproducing Results for AdvHotPotQA and FEVER

[hipros]

First of all, thank you for your great work!

While reproducing the results from the paper, I noticed that the performance I obtained was lower than the reported results.
I would like to check if I might have missed anything in my setup.

I conducted experiments on AdvHotPotQA and FEVER using GPT-3.5-turbo, and the results are as follows:

Model	Dataset	Paper Performance	Reproduced Performance
GPT-3.5-turbo	AdvHotPotQA	42.9	0.3636
GPT-3.5-turbo	FEVER	63.1	0.605

Below are the scripts I used for my experiment:

• Wikidata DB Script

$ wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz
$ python preprocess_dump.py --input_file ./latest-all.json.gz --out_dir ./processed_wiki
$ python build_index.py --input_dir ./processed_wiki --output_dir ./processed_wiki/indices --num_chunks 16

$ python server.py --data_dir ./processed_wiki --chunk_number 0 --host_ip <server IP> --port 23546
$ python server.py --data_dir ./processed_wiki --chunk_number 1 --host_ip <server IP> --port 23547
…
$ python server.py --data_dir ./processed_wiki --chunk_number 15 --host_ip <server IP> --port 23562

• ToG 2.0 Execution Script

$ python main_tog2.py \
--dataset hotpot_e \
--max_length 256 \
--temperature_exploration 0 \
--temperature_reasoning 0 \
--width 3 \
--depth 3 \
--remove_unnecessary_rel True \
--LLM_type_rp gpt-3.5-turbo-16k \
--LLM_type gpt-3.5-turbo \
--opeani_api_keys <openai_api_key> \
--embedding_model_name bge-bi \
--relation_prune_combination True \
--num_sents_for_reasoning 10 \
--topic_prune True \
--self_consistency_threshold 0.8 \
--clue_query True

I would appreciate any insights on what might be causing this performance discrepancy.
Please let me know if there are any additional steps or configurations I should check.

[liujilong0116]

First of all, thank you for your great work!

While reproducing the results from the paper, I noticed that the performance I obtained was lower than the reported results. I would like to check if I might have missed anything in my setup.

I conducted experiments on AdvHotPotQA and FEVER using GPT-3.5-turbo, and the results are as follows:

Model Dataset Paper Performance Reproduced Performance
GPT-3.5-turbo AdvHotPotQA 42.9 0.3636
GPT-3.5-turbo FEVER 63.1 0.605
Below are the scripts I used for my experiment:

• Wikidata DB Script

$ wget https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.json.gz
$ python preprocess_dump.py --input_file ./latest-all.json.gz --out_dir ./processed_wiki
$ python build_index.py --input_dir ./processed_wiki --output_dir ./processed_wiki/indices --num_chunks 16

$ python server.py --data_dir ./processed_wiki --chunk_number 0 --host_ip <server IP> --port 23546
$ python server.py --data_dir ./processed_wiki --chunk_number 1 --host_ip <server IP> --port 23547
…
$ python server.py --data_dir ./processed_wiki --chunk_number 15 --host_ip <server IP> --port 23562

• ToG 2.0 Execution Script

$ python main_tog2.py \
--dataset hotpot_e \
--max_length 256 \
--temperature_exploration 0 \
--temperature_reasoning 0 \
--width 3 \
--depth 3 \
--remove_unnecessary_rel True \
--LLM_type_rp gpt-3.5-turbo-16k \
--LLM_type gpt-3.5-turbo \
--opeani_api_keys <openai_api_key> \
--embedding_model_name bge-bi \
--relation_prune_combination True \
--num_sents_for_reasoning 10 \
--topic_prune True \
--self_consistency_threshold 0.8 \
--clue_query True

I would appreciate any insights on what might be causing this performance discrepancy. Please let me know if there are any additional steps or configurations I should check.

Hi,
I would like to reproduce the results from the paper, but I’m unsure whether my machine can support the deployment of Wikidata.
I would greatly appreciate it if you could let me know how much ROM and RAM the deployment of Wikidata uses on your machine.

[hipros]

Here's the detailed information regarding ROM (storage) and RAM usage for deploying Wikidata:

[Summary] My Resource Setup

• RAM: 2.5 TB
• Disk Storage (ROM): ~580–600 GB

---

1. Disk Storage (ROM)

• Raw Wikidata dump:

  Approximately 133 GB

• Final processed Wikidata size (after preprocessing):

  Approximately 426 GB

Total ROM required: about 558 GB

Therefore, having around 580–600 GB of storage is recommended for comfortable usage.

2. Memory (RAM)

Due to the practical difficulty of obtaining a single server with 2.5 TB RAM,
I deployed Wikidata using multiple servers. RAM usage per server was calculated simply as:

• RAM per server = (Total RAM required: 2.5 TB) / (Number of servers)

I recommend allocating slightly more RAM than the calculated requirement per server to ensure stable operations.

Also, for indexing, I modified the code to disable multiprocessing, processing each data chunk sequentially to manage RAM usage.

[liujilong0116]

Here's the detailed information regarding ROM (storage) and RAM usage for deploying Wikidata:

[Summary] My Resource Setup

• RAM: 2.5 TB
• Disk Storage (ROM): ~580–600 GB

1. Disk Storage (ROM)

• Raw Wikidata dump:
  Approximately 133 GB
• Final processed Wikidata size (after preprocessing):
  Approximately 426 GB

Total ROM required: about 558 GB

Therefore, having around 580–600 GB of storage is recommended for comfortable usage.

2. Memory (RAM)

Due to the practical difficulty of obtaining a single server with 2.5 TB RAM, I deployed Wikidata using multiple servers. RAM usage per server was calculated simply as:

• RAM per server = (Total RAM required: 2.5 TB) / (Number of servers)

I recommend allocating slightly more RAM than the calculated requirement per server to ensure stable operations.

Also, for indexing, I modified the code to disable multiprocessing, processing each data chunk sequentially to manage RAM usage.

Thank you very much for your response and for sharing the details regarding the ROM and RAM usage. This is extremely helpful for evaluating the feasibility of deploying Wikidata on my end. I really appreciate your time and support.