1

We can upload the data to Snowpark in the same way as we did in Scenario 1.

2

In Snowpark, users are limited to using the libraries pre-provided by the environment. To employ third-party packages, users must either establish a connection between their cluster and the local environment, utilizing the Jupyter Notebook interface, or navigate a cumbersome work-around as detailed in the article.

3

Worksheets does not provide a graphical output as they are py files and therefore users are left with the following choices: Connect to the local jupyter notebook. Save plots and view them manually outside a worksheet. Use the “chart” functionality of worksheets. The worksheets do provide a “chart” option to visualize data but are limited to only 4 kinds of in-built charts.

4

A new notebook was created with auto generated code

5

In Snowpark there is a potential issue to run the functions by pasting its name in settings and running. This annoying feature may lead to issues during development, and we tend to forget adding it. We have the capability to perform data cleansing with ease. The key distinction from Databricks is that the utilization of third-party packages within the worksheet could potentially pose an issue We can use a snowpark dataframe by using inbuilt libraries present in snowflake.ml.preprocessing else users can use the anaconda-python libraries. We have been able to successfully test by importing a lot of libraries.

6

We were able to write and test a lot of functions in order to test the differences between snowpark’s native dataframe and pandas dataframe. The two code samples use the same kind of transformations between snowpark dataframe and pandas dataframes and as follows with minor differences in using subsetting features of pandas(tempPdDf['ENGINE']) vs snowpark dataframe(with_column). Snowpark dataframe transformation code using UDF fix_values to get a clean modular code:

7

Similar differences were observed while trying to create a function for imputation ‘na’ as ‘Empty’. In snowpark dataframe as we need to use dictionary to pass the parameters eg: cat_column_dictonary = {feature:'Empty' for feature in feature_list[0]} while in pandas we can further concise If the data type of the provided replacement value does not match the column data type, fillna will produce a warning and skip that column. This can be avoided by specifying for each column what the replacement value should be, using a dictionary

8

Another distinction found while trying to impute the median values with in-built ‘median’ function was that it returns a view which we need to collect and use transformation before extracting the final values. median(feature).as_(feature)).collect()[0].as_dict().get(feature)

9

The features like categorical and feature engineering were smooth and came through without issues. But there are differences in usages between pyspark vs pandas vs snowpark datasets

10

11

Snowpark has support for feature-store based on the link and link however this was not found to be a part of snowsight/worksheets and only available while doing local development. This is in contrast to what we have explored as a part of Databricks.

12

In snowpark we can train sklearn and models provided by snowflake’s ml modeling package - link

13

Snowpark's worksheet is limited to using hyperopt but this is limited to use only with pandas dataframe and we could not use it with snowpark native dataframe. The installation of third-party libraries like OPTUNA can prove to be cumbersome. Moreover, built-in support for parallel/distributed hyperparameter tuning is absent. Nonetheless, achieving this can be done through an extended approach. In this approach we need to create a UDTF (User Defined Table Functions) as compared to Databricks where we used a function with hyperopt with pyspark.

14

Based on the approach we have been able to also try distributed hyperparameter tuning but could not use snowpark native dataframe and had to change to pandas with one model RandomForestRegressor but with a range of hyperparameters and the result was satisfactory.

15

While trying out MLflow we found that Snowpark leverages mlflow via snowpark/anaconda which means we can use MLflow to log training jobs from python stored procedure. Reference to link1 and link 2. But this will still use Azure for experiment tracking and to see the model artifacts/ runs and their corresponding metrics. Since we were interested in only trying to use a snowpark dataframe we did not delve deeper into this as pyspark Mllib has native support for MLflow.

16

In worksheets users can utilize the SHAP libraries to perform calculations, however, to see the plots users need to save the plot and view manually. To avoid that, users can use the support for connection with the local environment to run jupyter notebook on VS Code and view plots in real-time.

17

Models can only be deployed for batch inferencing and there is no support for real-time or streaming deployments. For batch deployment, there were two approaches which were discovered:

18

We can train the model locally, upload it to a stage and load it from the stage when the UDF is called.

19

20

In snowpark we can train sklearn and models provided by snowflake's ml modeling package - link

21

We can see a ‘procedure definition’ being available which shows the underlying code used to convert the model output to SPROC.

22

Git support is not available on snowflake’s Snowpark, unlike repos on Databricks the team could not find any direct support for git on the environment. For snowpark deployment we saw a newly available Snowpark Container Service from the link.Snowpark Container Services (SPCS) is a fully managed container offering that allows you to easily deploy, manage, and scale containerized services, jobs, and functions, all within the security and governance boundaries of Snowflake, and requiring zero data movement. Since this was only available to a subset of customers as a private view, we could not explore it further. Note: Using and connecting with a local Jupyter notebook on a platform such as VS Code we can overcome this issue and push code to GitHub from our local machine.