About the spark thrift server using spark cluster question: Initially, we used the Spark operator created by the RADAnalytics team (https://radanalytics.io/) to implement the Data Catalog architecture. RADAnalytics has this approach of long-running spark clusters as it was something in mind that customers would potentially look for. On the other hand, there's the Spark operator created by Google in which approach is app-centric, meaning that you only need to manage spark apps and they will use the k8s scheduler to provision the spark cluster. This last approach has the problem of being ephemeral by design, which needs other components like Spark History Server to track all spark app executions.

The change to use k8s scheduler with Spark thrift server was on our radar, but we decide to focus our development efforts on Trino because of some performance issues we found with Spark Thrift Server, as well as lack of Data Security implementation and others that blocks our path to a full Data Governance story. At last, we don't have a final decision if we still want to maintain a spark operator on ODH since kubeflow already does that, and RADAnalytics is in maintenance status.