Differences

This shows you the differences between two versions of the page.

--- faq [2018/12/19 01:16] – flack
+++ faq [2018/12/19 01:40] – flack
@@ Line 26: / Line 26: @@
 </code>
-If you have pigz installed on your system you can replace gzip with pigz in the commands above to speed up the process, especially when you want compressed chunks.
+If you have pigz installed on your system you can replace gzip with pigz in the commands above to speed up the process, especially when you want compressed chunks. You can typically install it with <code>sudo apt install pigz</code> or a similar command.
+====Can I get a Mcule database SDF file in smaller chunks?====
+If you have access to a unix based system and awk you can use the below commands to split large, gzipped SDF files into smaller chunks.
+To split an sdf.gz file into multiple uncompressed chunks, use a command like this:
+<code>
+gzip -dc your.sdf.gz | awk -v name=<chunk_name> -v ext=sdf -v size=<size> 'BEGIN{size=size}(NR==1){file1=sprintf("%s%0.10d.%s",name,counter,ext)}{print $0 > file1}{if($0=="$$$$"){file2=sprintf("%s%0.10d.%s",name,int(++counter/size),ext);{if(file1!=file2){close(file1);file1=file2}}}}'
+</code>
+Just replace your.sdf.gz with your filename, <chunk_name> with the name of the files you want and <size> with the intended chunk size.
+For example to split the Mcule Purchasable (Full) sdf.gz file into 1M uncompressed chunks use:
+<code>
+gzip -dc mcule_purchasable_full_180817.sdf.gz | awk -v name=mcule_purchasable_full_180817__ -v ext=sdf -v size=1000000 'BEGIN{size=size}(NR==1){file1=sprintf("%s%0.10d.%s",name,counter,ext)}{print $0 > file1}{if($0=="$$$$"){file2=sprintf("%s%0.10d.%s",name,int(++counter/size),ext);{if(file1!=file2){close(file1);file1=file2}}}}'
+</code>
+To split an sdf.gz file into multiple gzip compressed chunks, use a command like this:
+<code>
+gzip -dc your.sdf.gz | awk -v name=<chunk_name> -v ext=sdf.gz -v size=<size> 'BEGIN{size=size}(NR==1){file1=sprintf("%s%0.10d.%s",name,counter,ext)}{print $0 | "gzip -9 > "file1""}{if($0=="$$$$"){file2=sprintf("%s%0.10d.%s",name,int(++counter/size),ext);{if(file1!=file2){close("gzip -9 > "file1"");file1=file2}}}}'
+</code>
+Just replace your.sdf.gz with your filename, <chunk_name> with the name of the files you want and <size> with the intended chunk size.
+For example to split the Mcule Purchasable (Full) sdf.gz file into 1M gzip compressed chunks use:
+<code>
+gzip -dc mcule_purchasable_full_180817.sdf.gz | awk -v name=mcule_purchasable_full_180817__ -v ext=sdf.gz -v size=1000000 'BEGIN{size=size}(NR==1){file1=sprintf("%s%0.10d.%s",name,counter,ext)}{print $0 | "gzip -9 > "file1""}{if($0=="$$$$"){file2=sprintf("%s%0.10d.%s",name,int(++counter/size),ext);{if(file1!=file2){close("gzip -9 > "file1"");file1=file2}}}}'
+</code>
+If you have pigz installed on your system you can replace gzip with pigz in the commands above to speed up the process, especially when you want compressed chunks. You can typically install it with <code>sudo apt install pigz</code> or a similar command.
+Please note that the process can take a while.