Clonning to block device should allow partition to be bigger than the archive

[Jonathas-Conceicao]

When testing bita for applying changes to a ext4 formatted I've came across the following error:

Error: Size of output device (1430 MiB (1499661312 bytes)) differ from size of archive target file (142 MiB (149934080 bytes))

full command output

root@uh-qemu-x86-64:~# bita clone --seed-output updatehub-image.ext4.bita /dev/vda3
Archive:
  Built with version: 0.7.1
  Archive size: 29 MiB (30866271 bytes)
  Header checksum: 1635e9a121ca79f2cc1fb5e1bf23da9c8a64a5d95e146ba6299c900200d74f0cede30d88f6589abd34d9d07aef580a3d1041b3e722fc3ebc69a12421e03d0511
  Chunking algorithm: RollSum
  Rolling hash window size: 64 bytes
  Chunk minimum size: 16 KiB (16384 bytes)
  Chunk maximum size: 16 MiB (16777216 bytes)
  Chunk average target size: 64 KiB (65536 bytes) (mask: 0b111111111111111)
Source:
  Source checksum: cf895d5a02488c426a6a60dfda220f201a19bc1ba0c7eb4a5c8113319b21e81b2f0bc45923b5d3d21ebc288c03ed134fea5b76a2e42f9c5dc995177bef6e68b0
  Chunks in source: 1463 (unique: 1455)
  Average chunk size: 100 KiB (102909 bytes)
  Source size: 142 MiB (149934080 bytes)

Cloning archive updatehub-image.ext4.bita to /dev/vda3...
Error: Size of output device (1430 MiB (1499661312 bytes)) differ from size of archive target file (142 MiB (149934080 bytes))

Which was caused by the following piece of code:

bita/src/clone_cmd.rs

Lines 136 to 145 in e6d3a9e

	if output.is_block_dev() {
	let size = output.size().await?;
	if size != archive.total_source_size() {
	return Err(anyhow!(
	"Size of output device ({}) differ from size of archive target file ({})",
	size_to_str(size),
	size_to_str(archive.total_source_size())
	));
	}
	}

I believe that error should only be raised in case that the partition is smaller than the archive. A bigger partition should not be an issue.

Commenting out that block of the code in order to test the clone as a hole did work for applying the desired changes, but indexing the partition took more time than I would expect. I think the hole output device is being processed in order to perform the clone. If that's the case, wouldn't it be possible (for cases such as this) to index only the chunks up to the archive total source size?

[oll3]

I believe that error should only be raised in case that the partition is smaller than the archive. A bigger partition should not be an issue.

Yes, I think you are right that it should not be an error to write an image which is smaller than the destination block device.

If that's the case, wouldn't it be possible (for cases such as this) to index only the chunks up to the archive total source size?

I'm not sure about this though... There might be cases where a new image has decreased in size compared to the last image. Then you would probably wanna scan as much of the old image on the block device as possible to make sure full re-use.

Maybe could be a flag to only scan the size of the new archive. Or maybe we could add some cli parameter to bita to set an upper (and lower maybe?) bound to scan of a specific seed to avoid having to go through the full block device.

Another alternative could be to seed from stdin and use something like dd to limit the scan.

[otavio]

Maybe could be a flag to only scan the size of the new archive. Or maybe we could add some cli parameter to bita to set an upper (and lower maybe?) bound to scan of a specific seed to avoid having to go through the full block device.

I missed the case where a previous image was larger and we could, possibly, reuse some blocks after the end of the new file range. One thing I am wondering is: Do you foresee a way to "know" the size of the previous image?

This way we could read up to the end of old image and avoid going over it.

[oll3]

I missed the case where a previous image was larger and we could, possibly, reuse some blocks after the end of the new file range. One thing I am wondering is: Do you foresee a way to "know" the size of the previous image?

I have been thinking about building some kind of cache functionality. Where the chunk index of a written output would be stored on clone. That way the seed scan could be skipped completely next clone when/if a cache for that seed file is available. Still one would verify chunk checksums so that the cache and the seed file hasn't diverged. Then of course one need to decide what to do on checksum mismatch, fall back to a regular scan or be done or else.
Other than that no. And have not started this work yet and not sure when I will have time to do neither.

[oll3]

Closing since fixed from release 0.9.0.