I think we should not be introducing new special case behavior, nor sending policy documents to OPA, which from what we have heard and read, intends to make no use of them. Matt On Wed, Jan 29, 2020 at 4:45 PM Seena Fallah <seenafallah@xxxxxxxxx> wrote: > > I think it’s better to OR between two of the bucket policies and OPA policies. So if one of them reject certain access the request will reject as AWS do on its IAM and bucket policy. > Are you okay with this idea? > > On Wed, Jan 29, 2020 at 11:13 PM Casey Bodley <cbodley@xxxxxxxxxx> wrote: >> >> >> On 1/28/20 2:45 PM, Matthias Muench wrote: >> > Hi, >> > I think making Ceph special to what the rest of the clients in the >> > world would expect would be a bit off the idea of providing S3 like >> > service. >> > To my understanding, setting OPA to be the source of truth would >> > introduce latency (based on Casey’s comments) and will not allow to >> > set policies (based on Seena). >> > The first one brings us towards harder latency and especially >> > depending on extern systems resource capability (assume central >> > resource as the idea is and therefor not necessarily really “in reach” >> > within an acceptable latency, routing in addition, etc.). The second >> > one says simply that this would break any existing compatibility with >> > clients and use cases. To me it looks not that good to loose on both ends. >> >> Agreed. Even if one has to opt-in to this broken s3 compatiblity, I'm >> skeptical that users will find this to be a compelling target for their >> applications. >> >> The existing prototype of OPA integration sends this authorization >> request to OPA -in addition to- radosgw's own authorization logic, where >> we consult any of our user/bucket policies or ACLs that apply. In this >> model, OPA is not the only source of truth. It just has the opportunity >> to deny access that we would otherwise grant, so it doesn't require that >> we break compatibility with any S3 features that conflict with OPA's >> view of policy. >> >> Were we to change this so that OPA was the only source of truth, then >> we'd be left with two bad options: either reject all requests to modify >> policy and break existing applications, or send all policy/ACL >> information to OPA and require every OPA policy script to implement >> s3-compatible enforcement of them. I also don't see any benefit to this >> model - why, if an client wants to use s3 policy to restrict a certain >> access, would OPA want to override that and grant access instead? >> >> > I could live more with the latency issue but wouldn’t like it. >> > For the second, I can understand the idea of having simplification for >> > auditing the access but I’m not that convinced to take the burden of >> > being “the special” one that nobody wants to work with. So, I would >> > love to see the full fledged support of setting the policy by clients, >> > no matter what the result would be in terms of implementing it to >> > interact with OPA. Instead, having an additional requirement to >> > implement additional handling to set policies different from what S3 >> > actually provides would require special clients first and secondly an >> > additional path to OPA with all the additional burden to tweak >> > security to allow this path to OPA. I feel that the first wouldn’t >> > happen (special clients) and the second in practice not either because >> > of security constraints by the OPA admin folks. >> > >> > G, >> > -matt >> > >> > —————————————————— >> > Matthias Muench >> > Senior Specialist Solution Architect >> > EMEA Storage Specialist >> > matthias.muench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx> >> > Phone: +49-160-92654111 <tel:+49-160-92654111> >> > >> > Red Hat GmbH >> > Werner-von-Siemens-Ring 14 <x-apple-data-detectors://2/1> >> > 85630 Grasbrunn <x-apple-data-detectors://2/1> >> > Germany <x-apple-data-detectors://2/1> >> > _______________________________________________________________________ >> > Red Hat GmbH, http://www.de.redhat.com <http://de.redhat.com/> · >> > Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen >> > HRB 153243 · Managing Directors: Charles Cachera, Michael O'Neill, Tom >> > Savage, Eric Shander >> > >> >> On Jan 28, 2020, at 15:02, Seena Fallah <seenafallah@xxxxxxxxx> wrote: >> >> >> >> >> >> Amazon AWS S3 has two type of policies. One from bucket policy and >> >> one form IAM. I think it could be better to have two policies models >> >> in Ceph one from bucket policy and one form OPA if its enable. >> >> So if you are okay we can change the PR to make bucket policy enabled >> >> when OPA is enabled, too. Because now bucket policies not working >> >> when OPA integration is enabled. >> >> >> >> On Tue, Jan 28, 2020 at 2:57 AM Seena Fallah <seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx>> wrote: >> >> >> >> Matt When OPA integration is enabled S3 policies doesn’t work! If >> >> you want them to be worked we should bypass S3 policies to OPA >> >> for being applied and worked. >> >> Here we have conflict in OPA integration with S3 policies! >> >> >> >> On Tue, Jan 28, 2020 at 2:52 AM Matt Benjamin >> >> <mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> wrote: >> >> >> >> My take so far is that this is not a bug, and I'd like not to >> >> introduce special-case logic to override or suppress >> >> processing of >> >> native policy. >> >> >> >> Matt >> >> >> >> On Mon, Jan 27, 2020 at 5:24 PM Seena Fallah >> >> <seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>> wrote: >> >> > >> >> > I think it's very good that Ceph export its authorization >> >> and we could have external source of truth with it. S3 >> >> policies can transport to OPA and updates by users set/del >> >> policies. >> >> > But now we have conflict with OPA integration and S3 >> >> policies which is set when OPA integration is enabled, aren't >> >> work. >> >> > >> >> > Can you all please help to fix this bug? >> >> > >> >> > On Fri, Jan 24, 2020 at 1:05 PM Seena Fallah >> >> <seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>> wrote: >> >> >> >> >> >> Hi all. >> >> >> >> >> >> Any updates here? >> >> >> >> >> >> On Tue, Jan 21, 2020 at 2:50 AM Seena Fallah >> >> <seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>> wrote: >> >> >>> >> >> >>> OPA can be used in companies that uses many services like >> >> k8s, Ceph,... and want to have one central point for >> >> authorizing users so they can maintenance their access for >> >> each user on each service for example and etc. It’s just a >> >> use case and so it’s really good to have it. I think this is >> >> the biggest use case for having OPA in products that gets an >> >> option to centralize authorizing for all types of services. >> >> >>> >> >> >>> Performance for this model is issue like having keystone >> >> with Ceph. So I think it’s based on users that active this >> >> integrations at all. >> >> >>> >> >> >>> The model for writing policies to radosgw isn’t really >> >> good I think because of the reason above if this accrued >> >> there is always two copies of policies and it doesn’t sounds >> >> good for maintaining. >> >> >>> >> >> >>> If bucket policy disable, s3 clients like boto3 and etc >> >> will not work for setting polices but I think when someone is >> >> enabling OPA for authorizing it will also have an API for >> >> his/her OPA server to set/del policies and they can call >> >> these APIs to set/del policies. >> >> >>> And for extensions like PublicAccessBlock, it will >> >> disable because OPA is just authorizing requests and Ceph >> >> doesn’t authorize any request when OPA integration is enabled >> >> so OPA should handle any incoming policies were made by S3 >> >> policies. So it doesn’t make conflicts and if OPA integration >> >> is enabled it won’t work as we return 405 on each set/del >> >> policies requests and if OPA is disabled users can use this >> >> policies. >> >> >>> >> >> >>> >> >> >>> On Tue, Jan 21, 2020 at 2:05 AM Casey Bodley >> >> <cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>> wrote: >> >> >>>> >> >> >>>> I am a big fan of the IAM policy documents, both because >> >> of the >> >> >>>> flexibility and expressiveness they provide, and because >> >> they're in a >> >> >>>> format that all of our s3 clients understand. >> >> >>>> >> >> >>>> I'm not familiar enough with OPA to know what extra >> >> capabilities it >> >> >>>> offers that IAM policy cannot, but I have serious >> >> concerns about the >> >> >>>> performance and scalability of a model where radosgw has >> >> to send >> >> >>>> blocking RPCs to OPA in order to authorize each and >> >> every request. >> >> >>>> >> >> >>>> On the other hand, consider a model where a Policy Agent >> >> exercises its >> >> >>>> control over authorization by writing IAM documents to >> >> radosgw, which we >> >> >>>> use to cheaply authorize requests out of our metadata >> >> cache. I would >> >> >>>> imagine that this model could cover a lot of interesting >> >> use cases, >> >> >>>> without breaking support for existing s3 applications >> >> that rely on >> >> >>>> bucket policy - as the proposal to reject >> >> PutBucketPolicy requests would. >> >> >>>> >> >> >>>> Is this something that OPA could feasibly do? >> >> >>>> >> >> >>>> For use cases that aren't supported by the existing >> >> policy grammar, >> >> >>>> we're open to maintaining extensions to these documents. >> >> We already >> >> >>>> implement a number of s3 extensions [1][2] that are >> >> easily accessible >> >> >>>> via python/boto and the aws cli. >> >> >>>> >> >> >>>> But a model where radosgw outsources authorization >> >> entirely is a hard >> >> >>>> sell, because it conflicts with feature development >> >> going forward. One >> >> >>>> example would be support for PublicAccessBlock [3], >> >> where radosgw needs >> >> >>>> full visibility into policy to detect cases where public >> >> access would be >> >> >>>> granted. >> >> >>>> >> >> >>>> [1] >> >> >>>> >> >> https://docs.ceph.com/docs/master/radosgw/s3/python/#using-s3-api-extensions >> >> >>>> >> >> >>>> [2] >> >> >>>> >> >> https://github.com/ceph/ceph/blob/master/examples/boto3/service-2.sdk-extras.json >> >> >>>> >> >> >>>> [3] https://github.com/ceph/ceph/pull/30033 >> >> >>>> >> >> >>>> On 1/20/20 12:21 PM, Seena Fallah wrote: >> >> >>>> > I’m also agree with you Matt that it will free us from >> >> complexity of >> >> >>>> > handling S3 policy or Swift ACL if we save the current >> >> state of OPA. >> >> >>>> > But if we want use this state of OPA we should act for >> >> S3 policy and >> >> >>>> > Swift ACL that if user is setting them it shouldn’t be >> >> allowed and >> >> >>>> > return user that you can’t set them! Because now when >> >> OPA integration >> >> >>>> > is enabled and user set bucket policy it returns >> >> success but actually >> >> >>>> > it doesn’t work! >> >> >>>> > >> >> >>>> > What are your thoughts? >> >> >>>> > >> >> >>>> > On Sun, Jan 19, 2020 at 12:33 AM Seena Fallah >> >> <seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx> >> >> >>>> > <mailto:seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx>>> wrote: >> >> >>>> > >> >> >>>> > I think the other problem caused when OPA >> >> integration is enabled >> >> >>>> > and we set bucket policy is when user wants to get >> >> his/her bucket >> >> >>>> > policy. Some policies are set through OPA (for >> >> example in OPA >> >> >>>> > rules we have user A that has access to user B >> >> bucket so OPA >> >> >>>> > return true on authorizing request and it acts >> >> like bucket policy) >> >> >>>> > and some through bucket policy (s3 clients >> >> command). So when user >> >> >>>> > is getting his/her bucket policy what data should >> >> we return? The >> >> >>>> > policies that are set through bucket policy or OPA >> >> rules for that >> >> >>>> > bucket? >> >> >>>> > >> >> >>>> > I fact I think OPA rules are not static and will >> >> change in time >> >> >>>> > and so there should be a client interface for that >> >> OPA server that >> >> >>>> > users could change their rules for their buckets >> >> (giving access to >> >> >>>> > put, get, ... to someone else and etc.). So if the >> >> client exists >> >> >>>> > there is no need to bucket policy and we can make >> >> it disable (by >> >> >>>> > returning 405) when OPA integration is enabled (I >> >> repeat that >> >> >>>> > still now in Ceph latest version when OPA >> >> integration is enabled >> >> >>>> > bucket policies aren’t work!) because the policies >> >> that are set >> >> >>>> > with bucket policy can be check with OPA, too. >> >> >>>> > >> >> >>>> > What’s your opinion? >> >> >>>> > >> >> >>>> > On Fri, Jan 17, 2020 at 9:40 PM Seena Fallah >> >> >>>> > <seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx> <mailto:seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx>>> wrote: >> >> >>>> > >> >> >>>> > I think when OPA integration is enabled the >> >> source of truth >> >> >>>> > for authorizing should be OPA (it is right now >> >> in Ceph and all >> >> >>>> > requests are authorizing with OPA and Ceph >> >> doesn’t authorize >> >> >>>> > any request by it self). >> >> >>>> > When user is using bucket policy feature >> >> he/she wants to get >> >> >>>> > access to someone else so when he/she is the >> >> bucket owner, >> >> >>>> > he/she can perform this action and we should >> >> apply this policy >> >> >>>> > for him/her. If we want policies just update >> >> within OPA >> >> >>>> > server/client and S3 clients (s3cmd, aws, ...) >> >> don’t edit >> >> >>>> > policies, we should reply to them that >> >> set/delpolicy isn’t >> >> >>>> > allowed from here (return 405 for example; >> >> just for saying >> >> >>>> > that the request that user send isn’t successful). >> >> >>>> > >> >> >>>> > Yes we can have some process and >> >> simplification before sending >> >> >>>> > it to OPA but the s3 policy has a general >> >> structure so OPA >> >> >>>> > server can decode it by it self. >> >> >>>> > >> >> >>>> > On Fri, Jan 17, 2020 at 9:16 PM Matt Benjamin >> >> >>>> > <mbenjami@xxxxxxxxxx >> >> <mailto:mbenjami@xxxxxxxxxx> <mailto:mbenjami@xxxxxxxxxx >> >> <mailto:mbenjami@xxxxxxxxxx>>> wrote: >> >> >>>> > >> >> >>>> > The larger question, I think, is what OPA >> >> is supposed to >> >> >>>> > do with it. >> >> >>>> > The larger question I think it asks is >> >> whether OPA or Ceph >> >> >>>> > owns a >> >> >>>> > particular dimension of policy--or, >> >> perhaps, which owns >> >> >>>> > policy for >> >> >>>> > what portions of the namespace (at any >> >> particular point in >> >> >>>> > time). >> >> >>>> > >> >> >>>> > Without any new interaction, when OPA is >> >> configured, OPA >> >> >>>> > can make a >> >> >>>> > direct authorization decision with all >> >> available >> >> >>>> > information for >> >> >>>> > Ceph/RGW, notwithstanding any S3 or Swift >> >> ACL or S3 policy >> >> >>>> > that might >> >> >>>> > exist--including any that might have been >> >> stored prior to >> >> >>>> > turning on >> >> >>>> > this proposed feature to push policy >> >> documents to OPA. This >> >> >>>> > overriding property of the OPA integration >> >> when in use >> >> >>>> > frees us from a >> >> >>>> > lot of complexity regarding which system >> >> is the source of >> >> >>>> > truth, and >> >> >>>> > for what. >> >> >>>> > >> >> >>>> > I can see value in more sophisticated >> >> integration that >> >> >>>> > mutually >> >> >>>> > comprehends policy--but I'm having trouble >> >> with "send >> >> >>>> > policy documents >> >> >>>> > to OPA, maybe it will do something with them." >> >> >>>> > >> >> >>>> > Matt >> >> >>>> > >> >> >>>> > On Fri, Jan 17, 2020 at 12:01 PM Seena Fallah >> >> >>>> > <seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx> <mailto:seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx>>> wrote: >> >> >>>> > > >> >> >>>> > > Hello Ash >> >> >>>> > > >> >> >>>> > > With bucket policy user A can get access >> >> to user B for >> >> >>>> > putting object on bucket C. So if this >> >> policy sent to Ceph >> >> >>>> > and OPA integration is enabled it will be >> >> discard because >> >> >>>> > this policy isn’t sent to OPA server to be >> >> updated. >> >> >>>> > > Here is a documentation for bucket policy: >> >> >>>> > > >> >> https://docs.ceph.com/docs/master/radosgw/bucketpolicy/ >> >> >>>> > > >> >> >>>> > > With this PR when user set bucket >> >> policy, the data of >> >> >>>> > that policy will sent to OPA server to be >> >> applied and so >> >> >>>> > OPA can get access to user that gets >> >> access to bucket via >> >> >>>> > bucket policy. >> >> >>>> > > >> >> >>>> > > On Fri, Jan 17, 2020 at 8:24 PM Ash Narkar >> >> >>>> > <ash@xxxxxxxxx <mailto:ash@xxxxxxxxx> >> >> <mailto:ash@xxxxxxxxx <mailto:ash@xxxxxxxxx>>> wrote: >> >> >>>> > >> >> >> >>>> > >> Hello Seena, >> >> >>>> > >> >> >> >>>> > >> The OPA integration is with the RGW and >> >> the intent is >> >> >>>> > to check if an authenticated user is >> >> allowed to perform a >> >> >>>> > particular action on a particular >> >> resource. For example, >> >> >>>> > can Bob delete a bucket based on some >> >> attribute like his >> >> >>>> > location. I am not familiar with the >> >> internals of Ceph's >> >> >>>> > bucket policy command. It would be great >> >> to get some >> >> >>>> > context here and discuss if the bucket >> >> policy can be >> >> >>>> > authorized with OPA which is the intent of >> >> your PR I believe. >> >> >>>> > >> >> >> >>>> > >> Thanks >> >> >>>> > >> Ash >> >> >>>> > >> >> >> >>>> > >> On Fri, Jan 17, 2020 at 6:33 AM Seena >> >> Fallah >> >> >>>> > <seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx> <mailto:seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx>>> wrote: >> >> >>>> > >>> >> >> >>>> > >>> So when OPA integration is enabled the >> >> bucket policy >> >> >>>> > from users will not work! >> >> >>>> > >>> I think it’s about Ceph architecture >> >> not OPA because >> >> >>>> > OPA is for authorizing the requests and >> >> bucket policy is >> >> >>>> > one of the authorizing methods that OPA >> >> should support. >> >> >>>> > >>> >> >> >>>> > >>> On Fri, Jan 17, 2020 at 5:56 PM Matt >> >> Benjamin >> >> >>>> > <mbenjami@xxxxxxxxxx >> >> <mailto:mbenjami@xxxxxxxxxx> <mailto:mbenjami@xxxxxxxxxx >> >> <mailto:mbenjami@xxxxxxxxxx>>> wrote: >> >> >>>> > >>>> >> >> >>>> > >>>> Hi Seena, >> >> >>>> > >>>> >> >> >>>> > >>>> As I wrote in a comment on your PR, >> >> my current >> >> >>>> > intuition is that what >> >> >>>> > >>>> you're doing here isn't consistent >> >> with the original >> >> >>>> > intent of the OPA >> >> >>>> > >>>> integration we currently have, nor >> >> with the OPA model >> >> >>>> > in general. >> >> >>>> > >>>> >> >> >>>> > >>>> That said, I'd really like some >> >> feedback from OPA >> >> >>>> > architects, CC'd. >> >> >>>> > >>>> >> >> >>>> > >>>> regards, >> >> >>>> > >>>> >> >> >>>> > >>>> Matt >> >> >>>> > >>>> >> >> >>>> > >>>> On Thu, Jan 16, 2020 at 5:04 AM Seena >> >> Fallah >> >> >>>> > <seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx> <mailto:seenafallah@xxxxxxxxx >> >> <mailto:seenafallah@xxxxxxxxx>>> wrote: >> >> >>>> > >>>> > >> >> >>>> > >>>> > Hi all. In OPA integration from >> >> Ceph there is no >> >> >>>> > integration for bucket policy. >> >> >>>> > >>>> > When user is setting bucket policy >> >> to his/her >> >> >>>> > bucket the OPA server won't get who get's >> >> access to that >> >> >>>> > bucket so after that if the request is >> >> coming from the >> >> >>>> > user (that gets access to that bucket via >> >> bucket policy) >> >> >>>> > to access that bucket (PUT, GET,...), OPA >> >> will reject that >> >> >>>> > because of no data in database. >> >> >>>> > >>>> > I have create a pull request for >> >> this problem so if >> >> >>>> > user creates a bucket policy for his/her >> >> bucket, the >> >> >>>> > policy data will send to OPA server to be >> >> update on the >> >> >>>> > database. >> >> >>>> > >>>> > I think the main idea of having OPA >> >> is to have all >> >> >>>> > authorization in OPA and Ceph don't >> >> authorize any request >> >> >>>> > by it self. >> >> >>>> > >>>> > Here is the pull request and I >> >> would be thankful to >> >> >>>> > hear about your comments. >> >> >>>> > >>>> > https://github.com/ceph/ceph/pull/32294 >> >> >>>> > >>>> > Thanks. >> >> >>>> > >>>> > >> >> _______________________________________________ >> >> >>>> > >>>> > Dev mailing list -- dev@xxxxxxx >> >> <mailto:dev@xxxxxxx> <mailto:dev@xxxxxxx <mailto:dev@xxxxxxx>> >> >> >>>> > >>>> > To unsubscribe send an email to >> >> dev-leave@xxxxxxx <mailto:dev-leave@xxxxxxx> >> >> >>>> > <mailto:dev-leave@xxxxxxx >> >> <mailto:dev-leave@xxxxxxx>> >> >> >>>> > >>>> >> >> >>>> > >>>> >> >> >>>> > >>>> >> >> >>>> > >>>> -- >> >> >>>> > >>>> >> >> >>>> > >>>> Matt Benjamin >> >> >>>> > >>>> Red Hat, Inc. >> >> >>>> > >>>> 315 West Huron Street, Suite 140A >> >> >>>> > >>>> Ann Arbor, Michigan 48103 >> >> >>>> > >>>> >> >> >>>> > >>>> >> >> http://www.redhat.com/en/technologies/storage >> >> >>>> > >>>> >> >> >>>> > >>>> tel. 734-821-5101 >> >> >>>> > >>>> fax. 734-769-8938 >> >> >>>> > >>>> cel. 734-216-5309 >> >> >>>> > >>>> >> >> >>>> > >> >> >>>> > >> >> >>>> > -- >> >> >>>> > >> >> >>>> > Matt Benjamin >> >> >>>> > Red Hat, Inc. >> >> >>>> > 315 West Huron Street, Suite 140A >> >> >>>> > Ann Arbor, Michigan 48103 >> >> >>>> > >> >> >>>> > http://www.redhat.com/en/technologies/storage >> >> >>>> > >> >> >>>> > tel. 734-821-5101 >> >> >>>> > fax. 734-769-8938 >> >> >>>> > cel. 734-216-5309 >> >> >>>> > >> >> >>>> > >> >> >>>> > _______________________________________________ >> >> >>>> > Dev mailing list -- dev@xxxxxxx <mailto:dev@xxxxxxx> >> >> >>>> > To unsubscribe send an email to dev-leave@xxxxxxx >> >> <mailto:dev-leave@xxxxxxx> >> >> >>>> _______________________________________________ >> >> >>>> Dev mailing list -- dev@xxxxxxx <mailto:dev@xxxxxxx> >> >> >>>> To unsubscribe send an email to dev-leave@xxxxxxx >> >> <mailto:dev-leave@xxxxxxx> >> >> >> >> >> >> >> >> -- >> >> >> >> Matt Benjamin >> >> Red Hat, Inc. >> >> 315 West Huron Street, Suite 140A >> >> Ann Arbor, Michigan 48103 >> >> >> >> http://www.redhat.com/en/technologies/storage >> >> >> >> tel. 734-821-5101 >> >> fax. 734-769-8938 >> >> cel. 734-216-5309 >> >> >> >> _______________________________________________ >> >> Dev mailing list -- dev@xxxxxxx >> >> To unsubscribe send an email to dev-leave@xxxxxxx >> -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
