ironstrings alloca_probe stack size calculation errors

[llebout]

Hello!

I have a function at 0x1802b4d40 in this Windows PE x64 file
sample.txt (ignore extension its a dll, github wouldnt upload otherwise)

text:00000001802B4D40 000                 mov     [rsp-8+arg_0], rbx
.text:00000001802B4D45 000                 push    rbp
.text:00000001802B4D46 008                 push    rsi
.text:00000001802B4D47 010                 push    rdi
.text:00000001802B4D48 018                 push    r12
.text:00000001802B4D4A 020                 push    r13
.text:00000001802B4D4C 028                 push    r14
.text:00000001802B4D4E 030                 push    r15
.text:00000001802B4D50 038                 lea     rbp, [rsp-0F00h]
.text:00000001802B4D58 038                 mov     eax, 1000h
.text:00000001802B4D5D 038                 call    __alloca_probe
.text:00000001802B4D62 1038                sub     rsp, rax
.text:00000001802B4D65 1038                mov     rdi, r8
.text:00000001802B4D68 1038                mov     r14, rdx
.text:00000001802B4D6B 1038                mov     r15, rcx
.text:00000001802B4D6E 1038                xor     esi, esi

It seems like alloca_probe is messing with the stack size calculations, I also have another function where it did this, it specially happens on functions with large stacks. Do you have a clue how to fix it? I tried but couldnt figure it out. There is many stack strings in those functions, they seem to be constructed in part with XMM registers, though some smaller functions that did the same thing ironstrings was able to find strings without an issue. I tried other tools as well like flare-floss, stackstrings, nothing worked. ironstrings is what found the most strings in this sample, but still missing quite a few.

Here's the program output:

ironstrings2.log

Thanks a lot!

[mr-tz]

Thanks for the details! I'll take a look here in the next couple of days.

[mr-tz]

Thanks for the details. They helped to debug this a bit quicker. Please take a look at my proposed fix in #128.