+ opendathub contributor mailing list

Maybe somebody can help Rich with ceph here?

marcel
On Thu, May 2, 2019 at 12:09 PM Rich Lucente <rlucente@redhat.com> wrote:
All,

After getting OpenDataHub installed, I discovered that the ceph pod is not listening on port 8000 but instead port 8080.  When I adjusted the aws command line to use 8080 I was able to create an S3 bucket and upload data.  My concern is that other components may depend on port 8000 (for instance the ceph loadbalancer service) and during the configuration of jupyterhub there's a prompt for the ceph s3 endpoint URL.  Can someone confirm that ceph indeed is listening on port 8080 and this is not something particular to my environment, and then also what if any changes would be needed to opendatahub to accommodate the port change?

I'm running OpenDataHub using this OCP environment ... https://github.com/rlucente-se-jboss/ocp-on-nuc
I'm installing OpenDataHub using these scripts ... https://github.com/rlucente-se-jboss/opendatahub-demo

Rich Lucente

Principal Solutions Architect

Red Hat Public Sector

rlucente@redhat.com   
M: 240-994-0562    





On Wed, May 1, 2019 at 10:04 PM Rich Lucente <rlucente@redhat.com> wrote:
Hi Marcel,

I was able to install opendatahub successfully, but I'm not able to connect to the ceph-nano-0 pod.  Please see the following:

$ oc port-forward ceph-nano-0 8000 &
[1] 31222
$ Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000

$ curl http://localhost:8000
Handling connection for 8000
E0501 21:51:07.407755   31222 portforward.go:331] an error occurred forwarding 8000 -> 8000: error forwarding port 8000 to pod 5da3e18eceea7e2c0491f7ea856fe82415b36244b1279f39c43ec66496b7aa3c, uid : exit status 1: 2019/05/01 21:51:07 socat[21464] E connect(7, AF=2 127.0.0.1:8000, 16): Connection refused
curl: (52) Empty reply from server

I'm not sure why the connection is being refused.  The atomic-openshift-node process fails when attempting to connect to port 8000 on the pod.  On the compute node, I see the following error:

# journalctl -a -u atomic-openshift-node | grep httpstream | tail -1
May 01 21:51:07 console atomic-openshift-node[23539]: E0501 21:51:07.404800   23539 httpstream.go:251] error forwarding port 8000 to pod 5da3e18eceea7e2c0491f7ea856fe82415b36244b1279f39c43ec66496b7aa3c, uid : exit status 1: 2019/05/01 21:51:07 socat[21464] E connect(7, AF=2 127.0.0.1:8000, 16): Connection refused

Rich Lucente

Principal Solutions Architect

Red Hat Public Sector

rlucente@redhat.com   
M: 240-994-0562    





On Fri, Apr 12, 2019 at 7:24 AM Marcel Hild <mhild@redhat.com> wrote:
+Václav Pavlín for installing opendatahub, but the prometheus anomaly detection app doesnt need opendatahub.

I'd install maistra (which will include istio and kiali) and then the istio bookstore demo. That'll give you an application, prometheus, grafana and a load generator. That's all you need for the anomaly detection demo.

No need for jupyterhub

On Thu, Apr 11, 2019 at 10:24 PM Rich Lucente <rlucente@redhat.com> wrote:
So, I tried the following with minishift (cdk 3.8.0-2):

oc login -u admin -p admin $(minishift ip):8443
oc new-project opendatahub
apb registry add opendatahub --type quay --org opendatahub
apb bundle list
apb bundle provision jupyterhub-apb -s admin -f -vvv

 and it failed with the attached output and "KeyError: apis"

Is there a consolidated set of instructions somewhere to configure this with Minishift, OCP, etc?  I'm confused whether I first install Istio on top of 3.11 via Maistra then install the apb bundle for jupyterhub?

Rich Lucente

Principal Solution Architect

Red Hat Public Sector

rlucente@redhat.com    M: 240-994-0562    



On Wed, Apr 10, 2019 at 11:18 AM Marcel Hild <mhild@redhat.com> wrote:
You can easily ignore those. In a minimal setup the code does not need to store the model to S3.

All you need is a cluster with a prometheus instance running. Then deploy the anomaly-detection thing via a build pipeline, point it to prometheus and give it one metric to do the anomaly detection on.

I would use https://maistra.io/docs/ to setup an environment, use the included bookstore demo to create some load.
https://github.com/kiali/kiali/tree/master/hack has also stuff to create a cluster and create load.

Then use the anomaly detector on one istio metric.

+Anand Sanmukhani might help you out a bit, as he was involved in creating this POC.



On Wed, Apr 10, 2019 at 4:12 PM Rich Lucente <rlucente@redhat.com> wrote:
Thanks Marcel.  Any help you can offer is appreciated.

The first question has to do with deploying this to minishift (or any OCP instance for that matter).  The "make oc_deploy" target requires the followingparameters:
  • bearer_token
  • block_storage_access_key
  • block_storage_secret_key
  • block_storage_bucket_name
  • block_storage_endpoint_url
Can you give a sentence or two on what's expected for these?  Do I need an external block storage solution (e.g. Ceph or S3) for this to work or can it all run locally using local file storage?

Rich Lucente

Principal Solution Architect

Red Hat Public Sector

rlucente@redhat.com    M: 240-994-0562    



On Tue, Apr 9, 2019 at 12:12 PM Marcel Hild <mhild@redhat.com> wrote:
Hey,
I'm currently on PTO, so expect slow replies.

Unfortunately, the demo hasn't been touched lately, but I wanted to run it with RH service mesh and the istio bookstore demo.
The requirements should be just a minishift instance and that's all.

I can give you more pointers if you plan to polish that demo


On Mon, Apr 8, 2019 at 6:34 PM Rich Lucente <rlucente@redhat.com> wrote:
Hi Marcel,

Ted pointed me to your efforts around anomaly detection with Prometheus.  We are looking to take the demo on the road, potentially with limited bandwidth at a government tech conference.  What are the minimal set of requirements to run this?  Do I need a Ceph storage cluster in addition to the OCP components?

Rich Lucente

Principal Solution Architect

Red Hat Public Sector

rlucente@redhat.com    M: 240-994-0562    



On Mon, Apr 8, 2019 at 12:17 PM Ted Brunell <tbrunell@redhat.com> wrote:
FYI... Marcel knows what is being worked on.  You can reach out to them if needed.

R/
Ted

Ted Brunell

Senior Principal Solution Architect  RHCA, RHCE, RHCVA

CHIEF ARCHITECT, DoD PROGRAMS

Red Hat Public Sector

tbrunell@redhat.com    M: 760-712-6837    



---------- Forwarded message ---------
From: Marcel Hild <mhild@redhat.com>
Date: Thu, Feb 14, 2019 at 10:55 AM
Subject: Re: Looking for an AI Ops demo
To: Ted Brunell <tbrunell@redhat.com>, shgriffi@redhat.com <shgriffi@redhat.com>


+Sherard Griffin do you have links to current OpenDataHub demos / videos?
There is also this [1] video of a latest demo of training and deploying a tensorflow model on OpenShift.

For the anomaly detection with prometheus, let me know which issues you have. I have that exact setup running on my minishift.
Here [2] are slides for that setup.

[1] https://drive.google.com/file/d/1DBxtw1mbOecgCWZLIHmKNc3Rnb3Ozbc2/view?usp=sharing
[2] https://docs.google.com/presentation/d/1un00-UuLOvHVq334ClmWhRhROfN0Y-sMKixFt8NoBzo/edit?usp=sharing

On Thu, Feb 14, 2019 at 9:52 AM Ted Brunell <tbrunell@redhat.com> wrote:
Hi Marcel,

Thanks for responding.  I wasn't really looking for a product (there aren't any) but instead looking for a demo of OpenShift being used to host an AI application. 

Some of the AIOps projects on Mojo, specifically the Anomaly Detection with Promotheus.  I think that a log analysis applications would resonate very well with the people that attend that conference.  However, there seems to be some problems with the set of instructions that I found yesterday for getting Promotheus and Grafana running on MiniShift. 

I'll take a closer look at the OpenDataHub site.  Are the videos and workshop materials all posted there?

Thanks,
Ted

Ted Brunell

Senior Principal Solution Architect  RHCA, RHCE, RHCVA

CHIEF ARCHITECT, DoD PROGRAMS

Red Hat Public Sector

tbrunell@redhat.com    M: 760-712-6837    



On Wed, Feb 13, 2019 at 3:45 PM Marcel Hild <mhild@redhat.com> wrote:
Hey Ted,
there arent so many demos as in actual products that would show AIOps features. 
Are you looking for AI workloads as in running and developing AI on top of openshift?

You could show some of the opendatahub.io stuff we're working on. And we also have some workshop and video material related to that.

On Wed, Feb 13, 2019 at 2:43 PM Ted Brunell <tbrunell@redhat.com> wrote:
Halo Marcel,

I hope thay you are doing well. 

I saw that you are part of the AI CoE group and seem to be focused on AI Ops and I am hoping that you can help. 

We are preparing for a large cyber conference that is scheduled for May 14-16 where we will a booth presence.   The conference is highly attended by members of the Defense Information Systems Agency (DISA) who have begun to explore how AI can be integrated into their work.  We would love to be able to show a demonstration of the AIOps capabilities that have been developed on OpenShift.

Do you know of any demos or videos that we could show at the conference that would highlight AI workloads running in OpenShift and be visually appealing to people walking by the booth?

Thanks!
Ted


Ted Brunell

Senior Principal Solution Architect  RHCA, RHCE, RHCVA

CHIEF ARCHITECT, DoD PROGRAMS

Red Hat Public Sector

tbrunell@redhat.com    M: 760-712-6837