[Open Data Hub Contributors] Re: Looking for an AI Ops demo

All, After getting OpenDataHub installed, I discovered that the ceph pod is not listening on port 8000 but instead port 8080. When I adjusted the aws command line to use 8080 I was able to create an S3 bucket and upload data. My concern is that other components may depend on port 8000 (for instance the ceph loadbalancer service) and during the configuration of jupyterhub there's a prompt for the ceph s3 endpoint URL. Can someone confirm that ceph indeed is listening on port 8080 and this is not something particular to my environment, and then also what if any changes would be needed to opendatahub to accommodate the port change? I'm running OpenDataHub using this OCP environment ... https://github.com/rlucente-se-jboss/ocp-on-nuc I'm installing OpenDataHub using these scripts ... https://github.com/rlucente-se-jboss/opendatahub-demo Rich Lucente Principal Solutions Architect Red Hat Public Sector <https://www.redhat.com> rlucente(a)redhat.com M: 240-994-0562 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://red.ht/sig> On Wed, May 1, 2019 at 10:04 PM Rich Lucente <rlucente(a)redhat.com&gt; wrote: > Hi Marcel, > > I was able to install opendatahub successfully, but I'm not able to > connect to the ceph-nano-0 pod. Please see the following: > > $ *oc port-forward ceph-nano-0 8000 &* > [1] 31222 > $ Forwarding from 127.0.0.1:8000 -> 8000 > Forwarding from [::1]:8000 -> 8000 > > $ *curl http://localhost:8000 <http://localhost:8000>* > Handling connection for 8000 > E0501 21:51:07.407755 31222 portforward.go:331] an error occurred > forwarding 8000 -> 8000: error forwarding port 8000 to pod > 5da3e18eceea7e2c0491f7ea856fe82415b36244b1279f39c43ec66496b7aa3c, uid : > exit status 1: 2019/05/01 21:51:07 socat[21464] E connect(7, AF=2 > 127.0.0.1:8000, 16): Connection refused > curl: (52) Empty reply from server > > I'm not sure why the connection is being refused. The > atomic-openshift-node process fails when attempting to connect to port 8000 > on the pod. On the compute node, I see the following error: > > # *journalctl -a -u atomic-openshift-node | grep httpstream | tail -1* > May 01 21:51:07 console atomic-openshift-node[23539]: E0501 > 21:51:07.404800 23539 httpstream.go:251] error forwarding port 8000 to > pod 5da3e18eceea7e2c0491f7ea856fe82415b36244b1279f39c43ec66496b7aa3c, uid : > exit status 1: 2019/05/01 21:51:07 socat[21464] E connect(7, AF=2 > 127.0.0.1:8000, 16): Connection refused > > Rich Lucente > > Principal Solutions Architect > > Red Hat Public Sector <https://www.redhat.com> > > rlucente(a)redhat.com > M: 240-994-0562 > @RedHat <https://twitter.com/redhat> Red Hat > <https://www.linkedin.com/company/red-hat> Red Hat > <https://www.facebook.com/RedHatInc> > <https://red.ht/sig> > > > > > On Fri, Apr 12, 2019 at 7:24 AM Marcel Hild <mhild(a)redhat.com&gt; wrote: > >> +Václav Pavlín <vasek(a)redhat.com&gt; for installing opendatahub, but the >> prometheus anomaly detection app doesnt need opendatahub. >> >> I'd install maistra (which will include istio and kiali) and then the >> istio bookstore demo. That'll give you an application, prometheus, grafana >> and a load generator. That's all you need for the anomaly detection demo. >> >> No need for jupyterhub >> >> On Thu, Apr 11, 2019 at 10:24 PM Rich Lucente <rlucente(a)redhat.com&gt; >> wrote: >> >>> So, I tried the following with minishift (cdk 3.8.0-2): >>> >>> oc login -u admin -p admin $(minishift ip):8443 >>> oc new-project opendatahub >>> apb registry add opendatahub --type quay --org opendatahub >>> apb bundle list >>> apb bundle provision jupyterhub-apb -s admin -f -vvv >>> >>> and it failed with the attached output and "KeyError: apis" >>> >>> Is there a consolidated set of instructions somewhere to configure this >>> with Minishift, OCP, etc? I'm confused whether I first install Istio on >>> top of 3.11 via Maistra then install the apb bundle for jupyterhub? >>> >>> Rich Lucente >>> >>> Principal Solution Architect >>> >>> Red Hat Public Sector >>> <https://www.redhat.com> >>> >>> rlucente(a)redhat.com M: 240-994-0562 >>> <https://red.ht/sig> >>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>> >>> >>> On Wed, Apr 10, 2019 at 11:18 AM Marcel Hild <mhild(a)redhat.com&gt; wrote: >>> >>>> You can easily ignore those. In a minimal setup the code does not need >>>> to store the model to S3. >>>> >>>> All you need is a cluster with a prometheus instance running. Then >>>> deploy the anomaly-detection thing via a build pipeline, point it to >>>> prometheus and give it one metric to do the anomaly detection on. >>>> >>>> I would use https://maistra.io/docs/ to setup an environment, use the >>>> included bookstore demo to create some load. >>>> https://github.com/kiali/kiali/tree/master/hack has also stuff to >>>> create a cluster and create load. >>>> >>>> Then use the anomaly detector on one istio metric. >>>> >>>> +Anand Sanmukhani <asanmukh(a)redhat.com&gt; might help you out a bit, as >>>> he was involved in creating this POC. >>>> >>>> >>>> >>>> On Wed, Apr 10, 2019 at 4:12 PM Rich Lucente <rlucente(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Thanks Marcel. Any help you can offer is appreciated. >>>>> >>>>> The first question has to do with deploying this to minishift (or any >>>>> OCP instance for that matter). The "make oc_deploy" target requires the >>>>> followingparameters: >>>>> >>>>> - bearer_token >>>>> - block_storage_access_key >>>>> - block_storage_secret_key >>>>> - block_storage_bucket_name >>>>> - block_storage_endpoint_url >>>>> >>>>> Can you give a sentence or two on what's expected for these? Do I >>>>> need an external block storage solution (e.g. Ceph or S3) for this to work >>>>> or can it all run locally using local file storage? >>>>> >>>>> Rich Lucente >>>>> >>>>> Principal Solution Architect >>>>> >>>>> Red Hat Public Sector >>>>> <https://www.redhat.com> >>>>> >>>>> rlucente(a)redhat.com M: 240-994-0562 >>>>> <https://red.ht/sig> >>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>> >>>>> >>>>> On Tue, Apr 9, 2019 at 12:12 PM Marcel Hild <mhild(a)redhat.com&gt; wrote: >>>>> >>>>>> Hey, >>>>>> I'm currently on PTO, so expect slow replies. >>>>>> >>>>>> Unfortunately, the demo hasn't been touched lately, but I wanted to >>>>>> run it with RH service mesh and the istio bookstore demo. >>>>>> The requirements should be just a minishift instance and that's all. >>>>>> >>>>>> I can give you more pointers if you plan to polish that demo >>>>>> >>>>>> https://github.com/AICoE/prometheus-anomaly-detector >>>>>> >>>>>> On Mon, Apr 8, 2019 at 6:34 PM Rich Lucente <rlucente(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hi Marcel, >>>>>>> >>>>>>> Ted pointed me to your efforts around anomaly detection with >>>>>>> Prometheus. We are looking to take the demo on the road, potentially with >>>>>>> limited bandwidth at a government tech conference. What are the minimal >>>>>>> set of requirements to run this? Do I need a Ceph storage cluster in >>>>>>> addition to the OCP components? >>>>>>> >>>>>>> Rich Lucente >>>>>>> >>>>>>> Principal Solution Architect >>>>>>> >>>>>>> Red Hat Public Sector >>>>>>> <https://www.redhat.com> >>>>>>> >>>>>>> rlucente(a)redhat.com M: 240-994-0562 >>>>>>> <https://red.ht/sig> >>>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>>>> >>>>>>> >>>>>>> On Mon, Apr 8, 2019 at 12:17 PM Ted Brunell <tbrunell(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> FYI... Marcel knows what is being worked on. You can reach out to >>>>>>>> them if needed. >>>>>>>> >>>>>>>> R/ >>>>>>>> Ted >>>>>>>> >>>>>>>> Ted Brunell >>>>>>>> >>>>>>>> Senior Principal Solution Architect RHCA, RHCE, RHCVA >>>>>>>> >>>>>>>> CHIEF ARCHITECT, DoD PROGRAMS >>>>>>>> >>>>>>>> Red Hat Public Sector <https://www.redhat.com> >>>>>>>> >>>>>>>> tbrunell(a)redhat.com M: 760-712-6837 >>>>>>>> <https://red.ht/sig> >>>>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>>>>> >>>>>>>> >>>>>>>> ---------- Forwarded message --------- >>>>>>>> From: Marcel Hild <mhild(a)redhat.com&gt; >>>>>>>> Date: Thu, Feb 14, 2019 at 10:55 AM >>>>>>>> Subject: Re: Looking for an AI Ops demo >>>>>>>> To: Ted Brunell <tbrunell(a)redhat.com&gt;, shgriffi(a)redhat.com < >>>>>>>> shgriffi(a)redhat.com&gt; >>>>>>>> >>>>>>>> >>>>>>>> +Sherard Griffin <shgriffi(a)redhat.com&gt; do you have links to >>>>>>>> current OpenDataHub demos / videos? >>>>>>>> There is also this [1] video of a latest demo of training and >>>>>>>> deploying a tensorflow model on OpenShift. >>>>>>>> >>>>>>>> For the anomaly detection with prometheus, let me know which >>>>>>>> issues you have. I have that exact setup running on my minishift. >>>>>>>> Here [2] are slides for that setup. >>>>>>>> >>>>>>>> [1] >>>>>>>> https://drive.google.com/file/d/1DBxtw1mbOecgCWZLIHmKNc3Rnb3Ozbc2/view?us... >>>>>>>> [2] >>>>>>>> https://docs.google.com/presentation/d/1un00-UuLOvHVq334ClmWhRhROfN0Y-sMK... >>>>>>>> >>>>>>>> On Thu, Feb 14, 2019 at 9:52 AM Ted Brunell <tbrunell(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Marcel, >>>>>>>>> >>>>>>>>> Thanks for responding. I wasn't really looking for a product >>>>>>>>> (there aren't any) but instead looking for a demo of OpenShift being used >>>>>>>>> to host an AI application. >>>>>>>>> >>>>>>>>> Some of the AIOps projects on Mojo, specifically the Anomaly >>>>>>>>> Detection with Promotheus. I think that a log analysis applications would >>>>>>>>> resonate very well with the people that attend that conference. However, >>>>>>>>> there seems to be some problems with the set of instructions that I found >>>>>>>>> yesterday for getting Promotheus and Grafana running on MiniShift. >>>>>>>>> >>>>>>>>> I'll take a closer look at the OpenDataHub site. Are the videos >>>>>>>>> and workshop materials all posted there? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ted >>>>>>>>> >>>>>>>>> Ted Brunell >>>>>>>>> >>>>>>>>> Senior Principal Solution Architect RHCA, RHCE, RHCVA >>>>>>>>> >>>>>>>>> CHIEF ARCHITECT, DoD PROGRAMS >>>>>>>>> >>>>>>>>> Red Hat Public Sector <https://www.redhat.com> >>>>>>>>> >>>>>>>>> tbrunell(a)redhat.com M: 760-712-6837 >>>>>>>>> <https://red.ht/sig> >>>>>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Feb 13, 2019 at 3:45 PM Marcel Hild <mhild(a)redhat.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hey Ted, >>>>>>>>>> there arent so many demos as in actual products that would show >>>>>>>>>> AIOps features. >>>>>>>>>> Are you looking for AI workloads as in running and developing AI >>>>>>>>>> on top of openshift? >>>>>>>>>> >>>>>>>>>> You could show some of the opendatahub.io stuff we're working >>>>>>>>>> on. And we also have some workshop and video material related to that. >>>>>>>>>> >>>>>>>>>> On Wed, Feb 13, 2019 at 2:43 PM Ted Brunell <tbrunell(a)redhat.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Halo Marcel, >>>>>>>>>>> >>>>>>>>>>> I hope thay you are doing well. >>>>>>>>>>> >>>>>>>>>>> I saw that you are part of the AI CoE group and seem to be >>>>>>>>>>> focused on AI Ops and I am hoping that you can help. >>>>>>>>>>> >>>>>>>>>>> We are preparing for a large cyber conference that is scheduled >>>>>>>>>>> for May 14-16 where we will a booth presence. The conference is highly >>>>>>>>>>> attended by members of the Defense Information Systems Agency (DISA) who >>>>>>>>>>> have begun to explore how AI can be integrated into their work. We would >>>>>>>>>>> love to be able to show a demonstration of the AIOps capabilities that have >>>>>>>>>>> been developed on OpenShift. >>>>>>>>>>> >>>>>>>>>>> Do you know of any demos or videos that we could show at the >>>>>>>>>>> conference that would highlight AI workloads running in OpenShift and be >>>>>>>>>>> visually appealing to people walking by the booth? >>>>>>>>>>> >>>>>>>>>>> Thanks! >>>>>>>>>>> Ted >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Ted Brunell >>>>>>>>>>> >>>>>>>>>>> Senior Principal Solution Architect RHCA, RHCE, RHCVA >>>>>>>>>>> >>>>>>>>>>> CHIEF ARCHITECT, DoD PROGRAMS >>>>>>>>>>> >>>>>>>>>>> Red Hat Public Sector <https://www.redhat.com> >>>>>>>>>>> >>>>>>>>>>> tbrunell(a)redhat.com M: 760-712-6837 >>>>>>>>>>> <https://red.ht/sig> >>>>>>>>>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>>>>>>>>>> >>>>>>>>>>