creating an image file that can be used to mount on the emmc on a rockpi-s

[MeestorX]

Describe the bug
image file is over double the size of the disk that it's copying.
The whole size is listed as 2.53G, but the image file is 5349MiB

To Reproduce
run the program on a 2.5G disk

Expected behavior
2.5G or smaller .img file

Harware (please complete the following information):
Rock Pi-S, Armbian 24.8.3

Additional context
Here's the last of the log:

Number of files: 71,917 (reg: 60,281, dir: 8,382, link: 3,254)
Number of created files: 71,916 (reg: 60,281, dir: 8,381, link: 3,254)
Number of deleted files: 1 (dir: 1)
Number of regular files transferred: 60,277
Total file size: 2.53G bytes
Total transferred file size: 2.53G bytes
Literal data: 2.53G bytes
Matched data: 0 bytes
File list size: 2.23M
File list generation time: 0.005 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2.53G
Total bytes received: 1.20M

sent 2.53G bytes  received 1.20M bytes  6.91M bytes/sec
total size is **2.53G**  speedup is 1.00
## Rsync done...
## Please stand by...
## Finalizing filesystem...
armbi_root: /lost+found not found.  CREATED.

       71923 inodes used (21.03%, out of 342048)
          33 non-contiguous files (0.0%)
          57 non-contiguous directories (0.1%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 68758/1
      707696 blocks used (51.74%, out of 1367920)
           0 bad blocks
           1 large file

       60277 regular files
        8382 directories
           0 character device files
           0 block device files
           0 fifos
           4 links
        3254 symbolic links (3155 fast symbolic links)
           0 sockets
------------
       71917 files
## Remounting for autoexpansion...
## Mounting img root partition...
## Enabling fs-autoexpand...
## Armbian OS detected...
## Armbian filesystem autoresizing at boot...
## Backup done.
#####################################################################################
## Write to logfile: true                                                           #
## Autoexpand filesystem at boot: true                                              #
## Use exclude.txt: true                                                            #
## /mnt/2025-01-01.img is **5359MiB** with 100MiB [extra space] included                #
## Please wait for the system to reboot after restoring an image with autoexpansion #
#####################################################################################
## Exiting and cleaning up...
## Please stand by...
## Done.
## Elapsed time: 06.59

[MeestorX]

More from the log, Why is LOCAL_AUTORESIZE_MIN set to 6.5G? Where did it get LOCAL_USED_SPACE to be 5.3G?

2025-01-02 01:33:51 [INFO]  - Running function: get_dev_variables
2025-01-02 01:33:51 [INFO]  - No boot partition detected
2025-01-02 01:33:51 [DEBUG] - LOCAL_DEV_ROOT_PATH=/dev/mmcblk1p1
2025-01-02 01:33:51 [DEBUG] - LOCAL_ROOT_UUID=117b851b-2be0-4bb7-8572-7edce1ef7375 | LOCAL_ROOT_PARTUUID=ec99e6ce-01
2025-01-02 01:33:51 [DEBUG] - LOCAL_ROOT_PARTN=1
2025-01-02 01:33:51 [INFO]  - Calculating size for dd to cover bootsector and adding 5MiB (5242880 bytes) to overlap into root (only used in img creation)
2025-01-02 01:33:51 [DEBUG] - LOCAL_ROOT_START=16777216 bytes | LOCAL_BOOTSECTOR=16776704 bytes | LOCAL_DDBOOTSECTOR=43007 512B blocks
2025-01-02 01:33:51 [INFO]  - Calculating recommended root size
2025-01-02 01:33:51 [DEBUG] - BLOCKSIZE=4096 bytes
2025-01-02 01:33:51 [INFO]  - ext4 filesystem detected, using resize2fs to set recommended root size
2025-01-02 01:33:51 [DEBUG] - LOCAL_AUTORESIZE_MIN=6541316096 bytes
2025-01-02 01:33:51 [INFO]  - Calculating TOTAL (space needed for files on root) by adding LOCAL_USED_SPACE and ADDED_SPACE
2025-01-02 01:33:51 [DEBUG] - LOCAL_USED_SPACE=5363929088 bytes | ADDED_SPACE=104857600 bytes
2025-01-02 01:33:51 [DEBUG] - TOTAL=5468786688 bytes
2025-01-02 01:33:51 [INFO]  - Calculating .img file size by adding LOCAL_BOOTSECTOR to TOTAL
2025-01-02 01:33:51 [DEBUG] - TRUNCATE_TOTAL=5485563392 bytes
2025-01-02 01:33:51 [WARNING] - Adding WIGGLEROOM (128MiB) because manually requested ADDED_SPACE is less than autocaluclated size
2025-01-02 01:33:51 [DEBUG] - AUTORESIZE_WARNING=true | WIGGLEROOM=134217728 bytes | TRUNCATE_TOTAL=5619781120 bytes
2025-01-02 01:33:52 [INFO]  - Do you want to continue? [y/n]
2025-01-02 01:33:54 [INFO]  - Y or y pressed to confirm

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            164M     0  164M   0% /dev
tmpfs            48M  1.2M   47M   3% /run
/dev/mmcblk1p1   58G  2.5G   55G   5% /
tmpfs           237M     0  237M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           8.0G     0  8.0G   0% /tmp
/dev/zram1       47M  1.1M   43M   3% /var/log
tmpfs            48M     0   48M   0% /run/user/0

[UnconnectedBedna]

Please provide the entire log.

A quick glance tells me the storage device is 58G, and why resize2fs says you need 6G when df reports 2.5G used, well, that is a question you have to ask the devs of resize2fs. (don't do that, IIRC they have a disclaimer the -P function can be hit or miss)
As stated in the log file:

2025-01-02 01:33:51 [INFO]  - ext4 filesystem detected, using resize2fs to set recommended root size
2025-01-02 01:33:51 [DEBUG] - LOCAL_AUTORESIZE_MIN=6541316096 bytes

The exact line is at 398 in shrink-backup:

debug 'INFO' 'ext4 filesystem detected, using resize2fs to set recommended root size'
LOCAL_AUTORESIZE_MIN=$(resize2fs -P "$LOCAL_DEV_ROOT_PATH" 2>/dev/null | awk '{print $7}') # blocks
LOCAL_AUTORESIZE_MIN=$(( LOCAL_AUTORESIZE_MIN * BLOCKSIZE )) # bytes

Your root path gets detected as 2025-01-02 01:33:51 [DEBUG] - LOCAL_DEV_ROOT_PATH=/dev/mmcblk1p1

So the line used is: sudo resize2fs -P /dev/mmcblk1p1 (returns blocks) and then calculate how many bytes needed (blocks * 4096 = bytes), maybe the variable BLOCKSIZE (assuming 4096, 4k) gets set incorrectly (I need to see logfile to verify).
I think it is resize2fs being "safe" and returning a way bigger number than needed.
Try that line on your system and see what it tells you.

I have no control of how resize2fs works.
A solution could be to use df or du instead to calculate size (I have to make an option that lets you select), OR you could try manually adding space instead of relying on the automatic calculations, for example: sudo shrink-backup -tl /path/to/backup.img 2048
Please see: Smallest image possible

See if the "manually added space" method works first for you and we go from there.

[MeestorX]

I tried it again earlier, and was able to get an image created. However, what I was looking to do was to "burn" it to the eMMC on a rockpi-s, so without really knowing what I was doing, I dd'd it there, and it didn't boot.
Likely this script isn't really what I'm looking for, which is a way to emulate armbian-install from a commandline.
Sorry for wasting your time!

[UnconnectedBedna]

That is exactly how it should work, if it doesn't, it IS a bug and you are not wasting my time. :)
But that was not what you reported, you reported the img getting "too big".
You should be able to use dd, balena etcher, win64imgwriter, suse imgwriter etc etc etc, it SHOULD NOT MATTER how you get the img to the device.

The img file you create should be the same files as on the original system. The difference is the partition size gets shrunk.
Then if you do NOT disable the autoresize function (-e) when creating/updating the img, the filesystem and partition will be resized at first boot on that img.
Does it work if you create the img with -e and write and boot it? Maybe it is the autoresize function that is not working correctly for you.

If it does not work, I consider that a bug.
So if you want to keep looking at it, I am willing.
But without further information there is nothing I can do to help. 😃

Unless you want to dd the entire system into a 64G img file, I have not found any other backup utility that can do this with rsync without a boot partition. :)

[MeestorX]

Cool. Appreciate the willingness to help!

What I have is a rockpi-s that has a µSD slot and eMMC. I typically use armbian-install to copy a working "disk" on the µSD card to the eMMC, which works great. However, I want to automate the process, and armbian-install is menu-driven and quite complex.
If I were able to use your program to just do what I need, that would be amazing!
Basically, live-clone a working µSD card while inserted and running on the rockpi-s to its eMMC. If you can let me know how to do that, I will test the shit out of it! :-)

[UnconnectedBedna]

Oh, NOW I think I understand what you mean.
You change from mmcblk to emmc as root device?
Hmmmm, I have to look into that.
shrink-backup states it supports Armbian, so I should make the time to find out what is going on here.

I saw an email about a comment on armbian forums, was that you? (I need to find credentials to log in there)
I have to make a small disclaimer here, I am not the same person as created the script, that person has sadly passed but was kind enough to prepare and leave credentials for logins and such.
I am the closest relative to him, and his spirit will live on in this script. ❤️
I just wanted to mention this, because even though I have a TON of documentation he handcrafted and did not put online, it could take a while for me to learn 100% how the script works. Please do not make a thing about it...

[MeestorX]

No problem. That was me on the forum. I figured there was the place to have a discussion, and just report bugs here. This bug was most likely operator error on my part.
Sorry for your loss, I can empathize and have no issue being patient. I'm currently spending a lot of time trying to strip out everything I DON'T need within armbian-install to make it into simple script that will work for me. However, if you're able to make it work with this script, I think it would be more useful as I can also use it to create backup images, which currently is a bit of a PITA.
Let me know if you have something to try and I'll see how it goes. :-)
Happy New Year!

[UnconnectedBedna]

Thank you! ❤️

I am trying to find armbian-install but all I can find is armbian-config
I have no idea what configs I need to change to change boot device on armbian, so if you could provide a link it would probably make things significantly easier for me. :)

Edit:
Found the boot loaders etc: https://github.com/armbian/build/tree/b20254123c7a730d105de35cfd54cd413691b856/packages/bsp/common/usr/lib/armbian-install
And I suppose this it the config stuff I have to look into: https://github.com/armbian/build/tree/b20254123c7a730d105de35cfd54cd413691b856/packages/bsp/common/usr/lib/armbian

But it would be useful to see what it is exactly you are working with.

[MeestorX]

Yeah, very odd that armbian-install has no repository...
Here's a link to it, I assume I'm not breaking any rules by posting it here...

[MeestorX]

Fair warning, it's REALLY convoluted with not much in the way of commenting.

[UnconnectedBedna]

# Tool to transfer the rootfs of an already running Armbian installation from boot media
# to NAND, eMMC, SATA, NVME or USB storage. In case of eMMC it's also possible to transfer
# the bootloader to eMMC in a single step so from then on running without SD card is
# possible. In case of UEFI it's possible to install it side by side with Windows 10/11 or
# to an empty target.

Sounds like what we are looking for. :)

I will look closer at this. But after a first quck glance, this is what you are after right?
Line 888

	for choice in $choices
	do
		case $choice in
			1)
				title='MMC (SD/eMMC) boot | USB/SATA/NVMe root install'
				command='Reboot'
				check_partitions
				show_warning "This script will erase your device $DISK_ROOT_PART. Continue?"
				format_disk "$DISK_ROOT_PART"
				create_armbian "" "$DISK_ROOT_PART"
				;;

Edit:
On second thought, that is not what you are after, you want to run EVERYTHING on your emmc and not use the sd-card at all right?
End edit.

Just a quick question to make sure you actually need all of this.
Have you tried writing the img (made by shrink-backup) to emmc WHILE booted with the sd-card, then use armbian-config to select emmc as boot device, shutdown the device, remove the sd-card and boot?
They should both have the same UUID (so my guess is you don't need to change anything in fstab, just make sure you remove the sd-card before you boot).

Important

If you have 2 devices with the same UUID when you boot, you will get a kernel panic.

When trying, create the img and include -e option to disable the autoexpansion to minimize the risk of bugs.
You can expand the partition/filesystem in armbian-config too (IIRC).
shrink-backup uses this to autoexpand: armbian-resize-filesystem.service

ExecStart=/usr/lib/armbian/armbian-resize-filesystem start

I do not own a armbian device with an emmc, only normal sd-card devices so I have never done anything like this (except for on raspberry pi os, I have confirmed it works to move between mmcblk and sdx, but this is armbian).
But as far as I understand it, both sd-cards AND emmc both gets /dev/mmcblk* or am I incorrect with that statement?

[MeestorX]

I'm not sure how you "select emmc as boot device". The way it works is that it boots from the eMMC if there's no SD card inserted. If there's an SD card inserted and it has a bootable image, it boots from that.
The 2 devices cannot have the same UUID, as there will be need to mount the emmc when booting from the SD card, and I think when the UUID match, it puts the mounted file system into read-only.
On the Rockpi-s, mmcblk0 is always the emmc, mmcblk1 is always the SD. However, if you want something more generic, you'll have to determine which is which, even if it has a different name, I would expect...

[UnconnectedBedna]

Then I'm confused. (I thought you had to select boot device in armbian-config, my bad)
What is going wrong if you do what I suggested earlier?
If you create a backup FROM the sd-card, including the -e option, write the img to the emmc, shut down, remove the sd-card and boot, what goes wrong?

[MeestorX]

It doesn't boot. But, I never really found out if I was doing the procedure correctly.

1. Use your tool to create an image of the SD card that's currently running on the Rock-Pi, storing it in /mnt/ so it won't get added to the list while running...
2. use dd if=/mnt/backup.img of=/dev/mmcblk0 BS=8M status=progress to copy the image file to the emmc
3. Shut down the Rockpi and pull the sd card
4. Power on the RockPi

Does not boot, or at least I cannot SSH to it - not sure what it's doing because it has no monitor. If I knew how to read a dmesg from the emmc while booted on the SD card, perhaps it would show if it even tried to boot?

[MeestorX]

I guess another way to test would be to copy the image file to my computer, use Balena to burn it to an SD card, and then try booting with that SD card. I would assume that would work. Not sure if my dd command is the proper way to copy the image to the emmc. Does the image or emmc have to be mounted?

[MeestorX]

Success!
I took the sd.img file I had created, dd'd it to the eMMC, shut down, rebooted without the SD card and it started up!
So... Something must be wrong with the settings I used previously?

Was it the auto expansion? Why did you suggest to not allow the auto expansion?

[UnconnectedBedna]

I suggested it to minimize the risk, because I actually do not know if there are things happening on hardware level when you switch from one boot device to another.
Maybe it is that, maybe it is not.
Could be a random fluke. Could be gremlins, who knows...
Only way to know is for you to try with an img you create without using the -e option. :)

[MeestorX]

Ok. Certainly having matching UUID's looks to be an issue. It's taking quite a bit of work to try and get the rp to boot from the SD card while it has a matching UUID. Had to run armbian-install to fix it.
Is that something you might be able to solve?

[MeestorX]

Good news! I did an image without the -e and it booted! It's not using ALL of the available space, but it's close, so that's very cool!
If you are able to resolve the duplicate UUID issue, then this could be a winner!

[MeestorX]

My guess is that you'd have to make a copy of /boot/armbianEnv.txt, change the UUID line and use that in the image file. Not sure if /etc/fstab would also need to be copied and edited?
Check the create_armbian() section of armbian-install for some ideas.

[UnconnectedBedna]

If you are able to resolve the duplicate UUID issue, then this could be a winner!

Not sure what issue you are referring to, can you clarify?
Sounds to me there never were any issue to begin with but rather some fluke that made it not work the first time.

[MeestorX]

Oh, I expect it was operator error on my part the first time. The script does what it's supposed to and quite well!
However, for my needs, there's 2 things I still need to figure out:

1. How to use as much of the available space on the device for the partition as possible
2. How to have a different UUID for the image

I understand that these improvements may be out of the scope of what you wish to do with your program, and if so, I'll see if I can figure that out.

[UnconnectedBedna]

1. You should ask armbian about that, because the script relies on armbian-resize-filesystem.service
   To my knowledge, it correctly expands the partition and filesystem to cover the rest of the device it is on.
   What do you mean by "almost covers all". You have to start providing logs of statements you make.
2. Easiest way is to modify shrink-backup and edit the line where the filesystem is created and just remove UUID.

shrink-backup/shrink-backup

Line 1380 in 7870444

if ! output=$(mkfs.ext4 -U "$UUID" -L "$LABEL" "$IMG_DEV_ROOT_PATH" 2>&1 | tee "$TTY_AVAILABILITY" ); then

- if ! output=$(mkfs.ext4 -U "$UUID" -L "$LABEL" "$IMG_DEV_ROOT_PATH" 2>&1 | tee "$TTY_AVAILABILITY" ); then
+ if ! output=$(mkfs.ext4 -L "$LABEL" "$IMG_DEV_ROOT_PATH" 2>&1 | tee "$TTY_AVAILABILITY" ); then

You then have to use lsblk to see what new UUID the filesystem got and then use sed to edit fstab and change the UUID of root (or manually do that before you boot using that device).
This is not something that will be implemented in the script.
It's a backup. A backup should have the same UUID as the system it was made from.

Edit
Wait a minute. I might actually include that option. The downside it will create yet another dependency, uuidgen.
The package seems to be uuid-runtime.
I will look into this, because the option would actually be very easy to implement.
All I have to do is create a new option, if option is selected cat /proc/sys/kernel/random/uuid into $UUID (instead of copying the existing UUID from mounted filesystem) and sed the $UUID to fstab...

Since you came up with the idea, I let you select what letter I should use for the option.. 🥳

[MeestorX]

logically, --uuid

I'm not sure what you mean by "almost covers all"? Did I say that?
Folks at armbian don't really respond anymore, they seem to be understaffed and overworked from what they say. Bugs I've reported aren't getting verified let alone addressed, so I expect the resize issue I'll have to look into myself.

Thank you for offering to add the unique UUID feature!

[UnconnectedBedna]

logically, --uuid

Is not a letter. xD
A LETTER. The option is for "create new UUID". Check shrink-backup -h to see all options taken so far.

I'm not sure what you mean by "almost covers all"? Did I say that?

yes

#55

It's not using ALL of the available space, but it's close

WHAT is the "issue". What makes you think it is not working correctly?!?
Can you provide some proof of that? Like parted /dev/mmcblk1 print.
Your claim that shrink-backup had a bug was disprooven, twice.. So all I am trying to say is... be a bit careful about swinging accusations about bugs left and right. People might stop listening...

Yes, they are very few ppl, so bothering them with standard linux questions is probably not a good idea.
They also do all of it for free, so calling it "work" is a bit of a stretch...

[MeestorX]

Re: armbian-resize-filesystem.service: It might be the 5% that it's not using because the eMMC is only 4G

Jan 03 01:39:26 rp-satellite-xx systemd[1]: Starting armbian-resize-filesystem.service - Armbian filesystem resize...
Jan 03 01:39:26 rp-satellite-xx armbian-resize-filesystem[452]: Fri Jan  3 01:39:26 AM UTC 2025 | Rockpi S | 24.11.1 | arm64 | aarch64 | 6.6.63-current-rockchip64
Jan 03 01:39:26 rp-satellite-xx armbian-resize-filesystem[452]: ### [resize2fs] Trying to resize partition /dev/mmcblk0p1:
Jan 03 01:39:26 rp-satellite-xx armbian-resize-filesystem[452]: diskdevname: mmcblk0
Jan 03 01:39:26 rp-satellite-xx armbian-resize-filesystem[452]: diskdev: /dev/mmcblk0
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: partindex: 1
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: partstart: 32768s
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: partend: 7102134s
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: capacity: 7569408s
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: sectorsize: 512B
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: 4GiB or smaller media - leaving 5% spare area
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: Old partition table:
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: major minor  #blocks  name
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]:  179        0    3784704 mmcblk0
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]:  179        1    3534683 mmcblk0p1
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]:  251        0     242504 zram0
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]:  251        1      51200 zram1
Jan 03 01:39:27 rp-satellite-xx armbian-resize-filesystem[452]: Executing fdisk:

[UnconnectedBedna]

What is interesting is what userspace tools actually say:
parted /dev/mmcblk0 print

[MeestorX]

In my case...

parted /dev/mmcblk0 print
Model: SD CS032 (sd/mmc)
Disk /dev/mmcblk0: 3876MB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system  Flags
 1      16.8MB  3674MB  3657MB  primary  ext4

[MeestorX]

With only 4G I need every inch of it! But, not a showstopper. If the UUID thing gets addressed, I'll be very happy!

[UnconnectedBedna]

Disk /dev/mmcblk0: 3876MB
End: 3874MB

Guess you are correct, I have never noticed that.
If 2MB matters to you then...
If you want to deep dive, you can do it manually. look at how shrink-backup does it on raspberry pi:

shrink-backup/shrink-backup

Line 1843 in 7870444

cat << EOF2 > "${TMP_DIR}/expand-fs.sh"

You can read the content of what is being inserted into the file and then ran at boot. You can just manually run those commands exchanging the variables to what fits on your system.
Since they mention 5%, maybe they only expand it 95% instead of the 100% used in that version.

Good luck!

Edit
If you dont select a letter, I will do it when I get around to implementing the UUID thing on the testing branch.
Edit2
Actually, we can use --new-uuid if you want, because --uuid would be confusing.

[MeestorX]

because --uuid would be confusing

Like -e you mean? lol.

[UnconnectedBedna]

No, an e can mean anything, and -e is used already.

[UnconnectedBedna]

If you are just going to be silly about it, lets just close the thread.
I have already spent HOURS on you, for a bug that never existed.