10:05 a.m.
Hi all,
I'm a little new at OpenShifting and I need to deploy a whole ODH stack.. We have OCS
set up to use the s3 storage but the s2i-spark-notebook that comes with the Operator uses
a Hadoop version that refuses to change the URL style with
hadoopConf.set("fs.s3a.path.style.access", "true")
I get a big error log saying the URL of my bucket is unreachable and the URL that gets
used is the bucket.s3.storage.whatever where it should be s3.storage.whatever/bucket
Upon looking around online i found that it could be a bug that was solved in Hadoop
version 2.8 so I'd like to upgrade to that if at all possible but don't really
understand how to do it. I appreciate any advice.
Stay good,
Petar
10:43 a.m.
Hello Petar,
You have two options to update Hadoop version in s2i-spark-notebook:
1) From a notebook, run "!pip install pyspark==<any version that contains
Haddop >= 2.8.0" and restart the kernel. You will get hadoop version
updated install will be lost when you restart the server
2) Follow
https://github.com/vpavlin/jupyter-notebooks#creating-custom-notebook-images
to create a custom notebook image containing the desired hadoop version.
This is the recommended approach as it will persist the installation in the
image no matter if you restart the server.
Let me know if you have any questions.
On Fri, May 7, 2021 at 12:11 PM <pimilosevic(a)croz.net> wrote:
Hi all,
I'm a little new at OpenShifting and I need to deploy a whole ODH stack..
We have OCS set up to use the s3 storage but the s2i-spark-notebook that
comes with the Operator uses a Hadoop version that refuses to change the
URL style with hadoopConf.set("fs.s3a.path.style.access", "true")
I get a big error log saying the URL of my bucket is unreachable and the
URL that gets used is the bucket.s3.storage.whatever where it should be
s3.storage.whatever/bucket
Upon looking around online i found that it could be a bug that was solved
in Hadoop version 2.8 so I'd like to upgrade to that if at all possible but
don't really understand how to do it. I appreciate any advice.
Stay good,
Petar
_______________________________________________
Users mailing list -- users(a)lists.opendatahub.io
To unsubscribe send an email to users-leave(a)lists.opendatahub.io
--
Ricardo Martinelli De Oliveira
Senior Software Engineer, AI CoE
Red Hat Brazil <https://www.redhat.com/>
Av. Brigadeiro Faria Lima, 3900
8th floor
rmartine(a)redhat.com T: +551135426125
M: +5511970696531
@redhatjobs <https://twitter.com/redhatjobs> redhatjobs
<https://www.facebook.com/redhatjobs> @redhatjobs
<https://instagram.com/redhatjobs>
<https://www.redhat.com/>
11:10 a.m.
Hi Petar,
Yes this is a known issue with the version of Hadoop. A simple workaround
is to rename your bucket with all capital letters, such as "MYBUCKET".
That will force the URL to resolve properly. Others on the mailing list
may know if there is a notebook image available with a more recent version
of Hadoop. I thought there was something done in this area but will check
to see.
Thanks,
Sherard
On Fri, May 7, 2021 at 11:11 AM <pimilosevic(a)croz.net> wrote:
Hi all,
I'm a little new at OpenShifting and I need to deploy a whole ODH stack..
We have OCS set up to use the s3 storage but the s2i-spark-notebook that
comes with the Operator uses a Hadoop version that refuses to change the
URL style with hadoopConf.set("fs.s3a.path.style.access", "true")
I get a big error log saying the URL of my bucket is unreachable and the
URL that gets used is the bucket.s3.storage.whatever where it should be
s3.storage.whatever/bucket
Upon looking around online i found that it could be a bug that was solved
in Hadoop version 2.8 so I'd like to upgrade to that if at all possible but
don't really understand how to do it. I appreciate any advice.
Stay good,
Petar
_______________________________________________
Users mailing list -- users(a)lists.opendatahub.io
To unsubscribe send an email to users-leave(a)lists.opendatahub.io
--
Thanks,
Sherard Griffin
1081
days inactive
1084
days old
3 comments
4 participants
participants (4)
-
Petar Ivankovic Milosevic -
pimilosevic＠croz.net -
Ricardo Martinelli de Oliveira -
Sherard Griffin