Repository navigation
__pack_tensor__ should be in the beginning of the file to avoid seeking the whole file #162
Description
Hi @FrancescAlted ,
I have another concern about __pack_tensor__. According to hexedit, the __pack_tensor__ entry is located in the end of .bl2 file. I think this is an inefficient choice for large files. Suppose I have a 10 GiB bl2 file. I don't want to read it entirely, but knowing its shapes is essential for almost any usecase. So in order to read the shape, the c-blosc2 would need to fseek() up to the end of file. Of course, seeking is much faster than reading the content, but the file I/O would still need to hop over the inodes of the fragmented representation of big file in the filesystem. So why not to eliminate all this extra load on the filesystem by always placing metadata nodes in the beginning of the file? Is there an industry standard or practice that requires metadata to be placed in the end of file?
Activity
FrancescAlted commentedon Jan 29, 2024
Variable length metadata (aka user metadata) is at the end of the file on purpose; reason is that you always need to provide space for including more meta, and doing so at the beginning of the actual data would require a rewrite of it.
Now,
__pack_tensor__is really meant to provide kind of a quick and dirty support for different flavor of tensors (Torch, TF, NumPy). Isn't theb2ndmetalayer (which is at the beginning of the file) enough for you? What are you trying to achieve?dmikushin commentedon Jan 29, 2024
I'm just trying to work in C++ with the existing .bl2 files that were created in the following way:
Therefore, I am given
__pack_tensor__, it's not my choice. Should I try to create the .bl2 files in Python is some different way? Is there a similar Python way to useb2ndout of the box? It's highly desirable that I create .bl2 files the same way likely to be used by others, that's why I use the standard way above that you provide.OK,
__pack_tensor__is a variable-length metadata. But is it really varying that much? I think it's size could have a good estimate.FrancescAlted commentedon Jan 29, 2024
I just wanted to check whether you want to store the kind of tensor or if storing and retrieving a multidimensional dataset would be enough. If the latter, you can store your NDim dataset (see e.g. https://www.blosc.org/python-blosc2/getting_started/tutorials/02.ndarray-basics.html) and retrieve it from the C side quite easily too (see e.g. https://github.com/Blosc/c-blosc2/blob/main/examples/b2nd/example_serialize.c).
If you still want to get the additional info about the kind of tensor you are storing and you don't want to do seeks (although my experience is that they are very effective, and you should not need more speed for most of the cases), then you can still create your own fixed-length metalayer (e.g. https://www.blosc.org/python-blosc2/getting_started/tutorials/02.ndarray-basics.html#Metalayers-and-variable-length-metalayers) and read it from the C side withouth the additional seek(s). Mind that this could be a bit too involved for the (small) benefits you can get.
dmikushin commentedon Jan 29, 2024
Oh cool, perhaps ndarray is what I need, will go and study it, thanks a lot! 🙏