[Open Data Hub Contributors] Re: Using custom spark images in opendatahub operator

Agree with #2 On Mon, Aug 19, 2019 at 9:53 AM Václav Pavlín <vasek(a)redhat.com&gt; wrote: > I agree with #2 - ODH should work out of box, so we need to provide the > image (which is a no for #1), and #3 sounds like an overkill > > Thanks, > V. > > On Mon, Aug 19, 2019 at 3:43 PM Alex Corvin <acorvin(a)redhat.com&gt; wrote: > >> I think my vote is for #2. Option #1 will continue to be supported for >> groups that need it, but we can make it easier for people to get up and >> running by curating an official image. >> >> >> On August 19, 2019 at 9:33:44 AM, Ricardo Martinelli de Oliveira ( >> rmartine(a)redhat.com) wrote: >> >> Hi, >> >> I'm integrating Spark SQL Thrift server into ODH operator and I need to >> use a custom spark image (other than the RADAnalytics image) with >> additional jars to access Ceph/S3 buckets. Actually, both thrift server and >> the spark cluster will need this custom spark image in order to access the >> buckets. >> >> With that being said, I'd like to discuss some options to get this done. >> I am thinking about these options: >> >> 1) Let the customer specify the custom image in the yaml file (this is >> already possible) >> 2) Create that custom spark image and publish on quay.io opendarahub >> organization >> 3) Add a buildconfig object and make operator create the custom build >> and set the image location into the deploymentconfig objects >> >> Although the third option automate everything and deliver the whole set >> with the custom image, there's this thing about supporting custom images >> within operators. We'd need to add a spark_version variable where the build >> could download the spark distribution corresponding to that version and the >> artifacts related and run the build. In the first option, we simply don't >> create the build objects and document that in order to use Thrift server in >> ODH operator, both spark cluster and thrift must use a custom spark image >> containing the jars needed to access Ceph/S3. At last, the middle term >> between both is option two, so we don't need to worry about delegate this >> task to the user or the operator. >> >> What do you think? What could be the best option for this scenario? >> >> -- >> >> Ricardo Martinelli De Oliveira >> >> Data Engineer, AI CoE >> >> Red Hat Brazil <https://www.redhat.com/> >> >> Av. Brigadeiro Faria Lima, 3900 >> >> 8th floor >> >> rmartine(a)redhat.com T: +551135426125 >> M: +5511970696531 >> @redhatjobs <https://twitter.com/redhatjobs> redhatjobs >> <https://www.facebook.com/redhatjobs> @redhatjobs >> <https://instagram.com/redhatjobs> >> <https://www.redhat.com/> >> _______________________________________________ >> Contributors mailing list -- contributors(a)lists.opendatahub.io >> To unsubscribe send an email to contributors-leave(a)lists.opendatahub.io >> >> _______________________________________________ >> Contributors mailing list -- contributors(a)lists.opendatahub.io >> To unsubscribe send an email to contributors-leave(a)lists.opendatahub.io >> > > > -- > Open Data Hub, AI CoE, Office of CTO, Red Hat > Brno, Czech Republic > Phone: +420 739 666 824 > > _______________________________________________ > Contributors mailing list -- contributors(a)lists.opendatahub.io > To unsubscribe send an email to contributors-leave(a)lists.opendatahub.io > -- Landon LaSmith Sr.Software Engineer Red Hat, AI CoE - Data Hub