An Architect has designed a data pipeline that Is receiving small CSV files from multiple sources. All of the files are landing in one location…

According to the Snowflake documentation, the data loading performance can be improved by

following some best practices and guidelines for preparing and staging the data files. One of the

recommendations is to aim for data files that are roughly 100-250 MB (or larger) in size compressed,

as this will optimize the number of parallel operations for a load. Smaller files should be aggregated

and larger files should be split to achieve this size range. Another recommendation is to use a multi-

cluster warehouse for loading, as this will allow for scaling up or out the compute resources

depending on the load demand. A single-cluster warehouse may not be able to handle the load

concurrency and throughput efficiently. Therefore, by creating a multi-cluster warehouse and

merging smaller files to create bigger files, the data loading performance can be

improved. Reference:

Data Loading Considerations

Preparing Your Data Files

Planning a Data Load