Hello Petar,

You have two options to update Hadoop version in s2i-spark-notebook:

1) From a notebook, run "!pip install pyspark==<any version that contains Haddop >= 2.8.0" and restart the kernel. You will get hadoop version updated install will be lost when you restart the server
2) Follow https://github.com/vpavlin/jupyter-notebooks#creating-custom-notebook-images to create a custom notebook image containing the desired hadoop version. This is the recommended approach as it will persist the installation in the image no matter if you restart the server.

Let me know if you have any questions.

On Fri, May 7, 2021 at 12:11 PM <pimilosevic@croz.net> wrote:
Hi all,

I'm a little new at OpenShifting and I need to deploy a whole ODH stack.. We have OCS set up to use the s3 storage but the s2i-spark-notebook that comes with the Operator uses a Hadoop version that refuses to change the URL style with hadoopConf.set("fs.s3a.path.style.access", "true")
I get a big error log saying the URL of my bucket is unreachable and the URL that gets used is the bucket.s3.storage.whatever where it should be s3.storage.whatever/bucket

Upon looking around online i found that it could be a bug that was solved in Hadoop version 2.8 so I'd like to upgrade to that if at all possible but don't really understand how to do it. I appreciate any advice.

Stay good,
Petar
_______________________________________________
Users mailing list -- users@lists.opendatahub.io
To unsubscribe send an email to users-leave@lists.opendatahub.io


--

Ricardo Martinelli De Oliveira

Senior Software Engineer, AI CoE

Red Hat Brazil

Av. Brigadeiro Faria Lima, 3900

8th floor

rmartine@redhat.com    T: +551135426125    
M: +5511970696531