Seena Fallah <seenafallah@xxxxxxxxx> writes: > I have make this PR to make OPA single source of truth so OPA can get > access to not owned bucket for user. And also in OPA request we send > bucket policy, acl, ... to OPA so we can support them when OPA integration > is enabled. We will authorized user based on bucket policy, acl, ... and > OPA policies in OPA server. > Can you please take a look at here? > https://github.com/ceph/ceph/pull/32294 How does OPA decide on S3 conditionals wrt Policy here, we already support a few bucket attribute or Object based conditionals https://docs.ceph.com/docs/master/radosgw/bucketpolicy/#bucket-related-operations which can only be evaluated after reading the object in question for eg. These allow for eg. Bucket A would have access to User B to download object iff the object tag has an attribute openaccess = true. Additionally for buckets that are public https://github.com/ceph/ceph/pull/30033 would prevent access when IgnorePublicAcls are true. > > On Tue, Feb 18, 2020 at 9:52 AM Seena Fallah <seenafallah@xxxxxxxxx> wrote: > >> I have seen another bad scenario that we have two source of truth. If we >> get access to user for a bucket that he/she doesn't own it in OPA we can >> perform this action because op->verify_permission() will return -EACCES and >> so in rgw_process_authenticated function rgw_opa_authorize will not check! >> >> I think it's better to have one source of truth when we enabled OPA >> integration so we can send bucket policy, acl, ... to OPA on each >> request to be authorized. >> >> Do you have any other suggestion? >> >> On Tue, Feb 18, 2020 at 1:01 AM Seena Fallah <seenafallah@xxxxxxxxx> >> wrote: >> >>> I have changed the PR for sending bucket policy, bucket acl, iam policy >>> and user acl as a field to OPA request so OPA can decision based on this >>> parameters and we can have a external authorization for our organization >>> like AWS IAM. So we can have an S3 service that authorize based on bucket >>> policy and organizations IAM. >>> https://github.com/ceph/ceph/pull/32294 >>> Could you please review this? >>> >>> On Mon, Feb 17, 2020 at 2:39 PM Seena Fallah <seenafallah@xxxxxxxxx> >>> wrote: >>> >>>> Also we can add bucket policy result field to OPA request so in OPA >>>> policies we can act based on bucket policy results. >>>> >>>> Are you agree with it? >>>> >>>> On Thu, Feb 13, 2020 at 12:17 PM Seena Fallah <seenafallah@xxxxxxxxx> >>>> wrote: >>>> >>>>> In bucket policy we have an Effect::Pass in validation result so if we >>>>> just put OPA authorization in case of Effect::Pass I think it will be close >>>>> to what AWS do. >>>>> >>>>> On Fri, Feb 7, 2020 at 1:37 PM Seena Fallah <seenafallah@xxxxxxxxx> >>>>> wrote: >>>>> >>>>>> In organizations that has many services and want to have a centralized >>>>>> authorization server this will be a good solution to have. >>>>>> >>>>>> I mean that when we just authorize user ReadOnly in OPA but give write >>>>>> access via bucket policy, the user can’t write because OPA is rejecting. >>>>>> >>>>>> I think we can just weight bucket policy upper that OPA so bucket >>>>>> policies that apply policies specific than OPA policies can accept and >>>>>> reject at first then OPA would authorize that request. I mean bucket policy >>>>>> specify policy more specific (On bucket or on object) than OPA (OPA can set >>>>>> policy globally too like giving ReadOnly to all buckets) so it's better to >>>>>> first check bucket policy then check for OPA. this could be easy solve this >>>>>> problem. >>>>>> >>>>>> On Thu, Feb 6, 2020 at 10:11 PM Casey Bodley <cbodley@xxxxxxxxxx> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> On 2/6/20 11:58 AM, Seena Fallah wrote: >>>>>>> > The main goal of using OPA like as AWS IAM is having an external >>>>>>> > authorization so we can have out own management on policies from >>>>>>> > external source of truth (OPA) too. >>>>>>> > >>>>>>> > I think it’s better to handle bucket policy with OPA as well as AWS >>>>>>> > does so we can have a better S3 service :) >>>>>>> >>>>>>> Can you expand on why that's better than the model I suggested >>>>>>> earlier >>>>>>> in the thread, where a centralized policy service uses radosgw's >>>>>>> existing IAM APIs to manage policy instead of requiring radosgw to >>>>>>> call >>>>>>> out to an external service for every request? >>>>>>> >>>>>>> I'd also like to clarify what you mean when you say "handle bucket >>>>>>> policy with OPA" - my understanding is that it's not something that >>>>>>> OPA >>>>>>> itself does, but something very specific to your own product's OPA >>>>>>> policy script. Am I getting that right? If so, it sounds like you're >>>>>>> trying to re-engineer our OPA integration in a way that a) is not >>>>>>> useful >>>>>>> to OPA users in general, and b) duplicates functionality that radosgw >>>>>>> already provides. >>>>>>> >>>>>>> For OPA users that just want the ability to write simple scripts to >>>>>>> customize authorization for their environment, I think our current >>>>>>> level >>>>>>> of OPA integration is sufficient. >>>>>>> >>>>>>> > >>>>>>> > On Thu, Feb 6, 2020 at 8:16 PM Casey Bodley <cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx>> wrote: >>>>>>> > >>>>>>> > Is there an advantage to doing this in OPA over radosgw? Have >>>>>>> you >>>>>>> > looked >>>>>>> > at using our PutUserPolicy[1] APIs instead? We support both >>>>>>> user and >>>>>>> > bucket policy, and (as far as I know) handle the intersection of >>>>>>> > the two >>>>>>> > as you'd expect. >>>>>>> > >>>>>>> > >>>>>>> https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html >>>>>>> > >>>>>>> > On 2/5/20 9:55 AM, Seena Fallah wrote: >>>>>>> > > I'm trying to implement AWS IAM with OPA so I can have >>>>>>> external >>>>>>> > > authorization for my S3 service and also have an active bucket >>>>>>> > ACL and >>>>>>> > > bucket policy. >>>>>>> > > >>>>>>> > > On Wed, Feb 5, 2020 at 6:11 PM Casey Bodley < >>>>>>> cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>> >>>>>>> wrote: >>>>>>> > > >>>>>>> > > I'm confused by your references to AWS IAM. Are you >>>>>>> talking >>>>>>> > about >>>>>>> > > radosgw user policy? Or are you trying to implement IAM >>>>>>> policy >>>>>>> > > inside of >>>>>>> > > OPA? >>>>>>> > > >>>>>>> > > On 2/5/20 8:54 AM, Seena Fallah wrote: >>>>>>> > > > Hi all. >>>>>>> > > > >>>>>>> > > > Any updates here? :) >>>>>>> > > > >>>>>>> > > > On Sat, Feb 1, 2020 at 10:37 AM Seena Fallah >>>>>>> > > <seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx> >>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>> >>>>>>> > > wrote: >>>>>>> > > > >>>>>>> > > > I should also mention that if we get access to >>>>>>> bucket >>>>>>> > via bucket >>>>>>> > > > policy and reject it via AWS IAM, the request will >>>>>>> > reject so I >>>>>>> > > > think we should make a new behavior at what we >>>>>>> should do >>>>>>> > > with this >>>>>>> > > > two source of truth? >>>>>>> > > > >>>>>>> > > > On Sat, Feb 1, 2020 at 10:33 AM Seena Fallah >>>>>>> > > > <seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx> >>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx> >>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>>>> >>>>>>> wrote: >>>>>>> > > > >>>>>>> > > > Yes but the main problem is when the policy >>>>>>> isn't >>>>>>> > set in AWS >>>>>>> > > > IAM for example a user has >>>>>>> only AmazonS3ReadOnlyAccess >>>>>>> > > and we >>>>>>> > > > give PutObject policy via bucket policy, user >>>>>>> can >>>>>>> > put object >>>>>>> > > > to that bucket but in radosgw, OPA will deny >>>>>>> this >>>>>>> > process >>>>>>> > > > because there is only ReadOnlyAccess to that >>>>>>> > bucket for user >>>>>>> > > > and radosgw will not check bucket policy that >>>>>>> gave >>>>>>> > access to >>>>>>> > > > user. >>>>>>> > > > I think we should weight bucket policy over OPA >>>>>>> so >>>>>>> > if bucket >>>>>>> > > > policy accept that request it doesn't need to be >>>>>>> > checked >>>>>>> > > with >>>>>>> > > > OPA BUT if there is no policy according to that >>>>>>> > request it >>>>>>> > > > should check by OPA because if the policy >>>>>>> > according to that >>>>>>> > > > request isn't set bucket policy will reject that >>>>>>> > request >>>>>>> > > so it >>>>>>> > > > against failed! >>>>>>> > > > >>>>>>> > > > On Fri, Jan 31, 2020 at 1:30 AM Casey Bodley >>>>>>> > > > <cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx> >>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx> >>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>>> >>>>>>> wrote: >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > On 1/30/20 2:18 PM, Seena Fallah wrote: >>>>>>> > > > > Hi Casey, >>>>>>> > > > > >>>>>>> > > > > The main problem now is when OPA >>>>>>> integration is >>>>>>> > > enabled >>>>>>> > > > bucket >>>>>>> > > > > policies aren’t work! >>>>>>> > > > > I have checked AWS S3 that what is doing >>>>>>> > when both >>>>>>> > > > bucket policy and >>>>>>> > > > > IAM policy (the policy is set with AWS >>>>>>> panel >>>>>>> > in IAM >>>>>>> > > > section) is set it >>>>>>> > > > > will OR between two of them so now in >>>>>>> > radosgw S3 we >>>>>>> > > > don’t have this >>>>>>> > > > > feature and bucket policies won’t work >>>>>>> when OPA >>>>>>> > > > integration is enabled. >>>>>>> > > > > >>>>>>> > > > > So I think it’s better to active this >>>>>>> > feature and >>>>>>> > > > enabled bucket >>>>>>> > > > > policy when OpA integration is enabled. >>>>>>> > > > > >>>>>>> > > > > There is two solutions here in this >>>>>>> > discussion for >>>>>>> > > > enabling bucket >>>>>>> > > > > policy on OPA integration: >>>>>>> > > > > 1. Send bucket policy on set/del actions >>>>>>> to >>>>>>> > OPA server >>>>>>> > > > to be apply on >>>>>>> > > > > OPA policy rules so in this case the >>>>>>> source >>>>>>> > of truth >>>>>>> > > > will be OPA (the >>>>>>> > > > > state that we have now in OPA integration) >>>>>>> > and so >>>>>>> > > these >>>>>>> > > > policies that >>>>>>> > > > > sent from bucket policy will be applied. >>>>>>> > > > > 2. OR between bucket policy and OPA >>>>>>> policy like >>>>>>> > > AWS S3. >>>>>>> > > > So there is >>>>>>> > > > > two source of truth in this case and if >>>>>>> any of >>>>>>> > > them deny >>>>>>> > > > the request, >>>>>>> > > > > the request will be denied. >>>>>>> > > > >>>>>>> > > > What you described here in 2. is exactly >>>>>>> how it >>>>>>> > > currently >>>>>>> > > > works. >>>>>>> > > > >>>>>>> > > > > >>>>>>> > > > > Do have any other solutions we have here >>>>>>> and >>>>>>> > which of >>>>>>> > > > these solutions >>>>>>> > > > > do you prefer to have? >>>>>>> > > > > >>>>>>> > > > > On Thu, Jan 30, 2020 at 10:21 PM Casey >>>>>>> Bodley >>>>>>> > > > <cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx>> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx> >>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>> >>>>>>> > > > > <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>> >>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx >>>>>>> >>>>> >>>>>>> > > > wrote: >>>>>>> > > > > >>>>>>> > > > > Hi Seena, >>>>>>> > > > > >>>>>>> > > > > I think it would probably help if you >>>>>>> could >>>>>>> > > describe >>>>>>> > > > your use case >>>>>>> > > > > here, >>>>>>> > > > > and what role you want OPA to play in >>>>>>> the >>>>>>> > > > interpretation of these >>>>>>> > > > > bucket >>>>>>> > > > > policies. In other words, what is it >>>>>>> > that your OPA >>>>>>> > > > policy is doing >>>>>>> > > > > with >>>>>>> > > > > these bucket policy documents that >>>>>>> shouldn't >>>>>>> > > be done >>>>>>> > > > within radosgw? >>>>>>> > > > > >>>>>>> > > > > On 1/30/20 1:09 AM, Seena Fallah >>>>>>> wrote: >>>>>>> > > > > > So Matt what should we have done >>>>>>> with >>>>>>> > bucket >>>>>>> > > > policy if we enable >>>>>>> > > > > OPA >>>>>>> > > > > > integration? >>>>>>> > > > > > >>>>>>> > > > > > On Thu, Jan 30, 2020 at 1:45 AM Matt >>>>>>> > Benjamin >>>>>>> > > > > <mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>>> >>>>>>> > > > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>> >>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx>>>>>> wrote: >>>>>>> > > > > > >>>>>>> > > > > > I think we should not be >>>>>>> > introducing new >>>>>>> > > > special case >>>>>>> > > > > behavior, nor >>>>>>> > > > > > sending policy documents to OPA, >>>>>>> > which from >>>>>>> > > > what we have >>>>>>> > > > > heard and >>>>>>> > > > > > read, intends to make no use of >>>>>>> them. >>>>>>> > > > > > >>>>>>> > > > > > Matt >>>>>>> > > > > > >>>>>>> > > > > > On Wed, Jan 29, 2020 at 4:45 PM >>>>>>> > Seena Fallah >>>>>>> > > > > > <seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>>> >>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>>>> wrote: >>>>>>> > > > > > > >>>>>>> > > > > > > I think it’s better to OR >>>>>>> > between two >>>>>>> > > of the >>>>>>> > > > bucket >>>>>>> > > > > policies and >>>>>>> > > > > > OPA policies. So if one of them >>>>>>> reject >>>>>>> > > certain >>>>>>> > > > access the >>>>>>> > > > > request >>>>>>> > > > > > will reject as AWS do on its IAM >>>>>>> > and bucket >>>>>>> > > > policy. >>>>>>> > > > > > > Are you okay with this idea? >>>>>>> > > > > > > >>>>>>> > > > > > > On Wed, Jan 29, 2020 at 11:13 >>>>>>> PM >>>>>>> > Casey >>>>>>> > > Bodley >>>>>>> > > > > > <cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>> >>>>>>> > > > <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>> >>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>> >>>>>>> > > > <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx>>>> >>>>>>> > > > > <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>> >>>>>>> > > > <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>> >>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>> >>>>>>> > > > <mailto:cbodley@xxxxxxxxxx >>>>>>> > <mailto:cbodley@xxxxxxxxxx> >>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx >>>>>>> >>>>>> >>>>>>> > wrote: >>>>>>> > > > > > >> >>>>>>> > > > > > >> >>>>>>> > > > > > >> On 1/28/20 2:45 PM, Matthias >>>>>>> Muench >>>>>>> > > wrote: >>>>>>> > > > > > >> > Hi, >>>>>>> > > > > > >> > I think making Ceph special >>>>>>> > to what the >>>>>>> > > > rest of the clients >>>>>>> > > > > > in the >>>>>>> > > > > > >> > world would expect would >>>>>>> be a bit >>>>>>> > > off the >>>>>>> > > > idea of providing >>>>>>> > > > > > S3 like >>>>>>> > > > > > >> > service. >>>>>>> > > > > > >> > To my understanding, >>>>>>> setting >>>>>>> > OPA to be >>>>>>> > > > the source of >>>>>>> > > > > truth would >>>>>>> > > > > > >> > introduce latency (based on >>>>>>> > Casey’s >>>>>>> > > > comments) and will not >>>>>>> > > > > > allow to >>>>>>> > > > > > >> > set policies (based on >>>>>>> Seena). >>>>>>> > > > > > >> > The first one brings us >>>>>>> > towards harder >>>>>>> > > > latency and >>>>>>> > > > > especially >>>>>>> > > > > > >> > depending on extern systems >>>>>>> > resource >>>>>>> > > > capability (assume >>>>>>> > > > > central >>>>>>> > > > > > >> > resource as the idea is and >>>>>>> > > therefor not >>>>>>> > > > necessarily really >>>>>>> > > > > > “in reach” >>>>>>> > > > > > >> > within an acceptable >>>>>>> latency, >>>>>>> > > routing in >>>>>>> > > > addition, >>>>>>> > > > > etc.). The >>>>>>> > > > > > second >>>>>>> > > > > > >> > one says simply that this >>>>>>> would >>>>>>> > > break any >>>>>>> > > > existing >>>>>>> > > > > > compatibility with >>>>>>> > > > > > >> > clients and use cases. To >>>>>>> me it >>>>>>> > > looks not >>>>>>> > > > that good to >>>>>>> > > > > loose >>>>>>> > > > > > on both ends. >>>>>>> > > > > > >> >>>>>>> > > > > > >> Agreed. Even if one has to >>>>>>> > opt-in to this >>>>>>> > > > broken s3 >>>>>>> > > > > > compatiblity, I'm >>>>>>> > > > > > >> skeptical that users will >>>>>>> find this >>>>>>> > > to be a >>>>>>> > > > compelling target >>>>>>> > > > > > for their >>>>>>> > > > > > >> applications. >>>>>>> > > > > > >> >>>>>>> > > > > > >> The existing prototype of OPA >>>>>>> > integration >>>>>>> > > > sends this >>>>>>> > > > > authorization >>>>>>> > > > > > >> request to OPA -in addition >>>>>>> to- >>>>>>> > radosgw's >>>>>>> > > > own authorization >>>>>>> > > > > > logic, where >>>>>>> > > > > > >> we consult any of our >>>>>>> user/bucket >>>>>>> > > policies >>>>>>> > > > or ACLs that >>>>>>> > > > > apply. >>>>>>> > > > > > In this >>>>>>> > > > > > >> model, OPA is not the only >>>>>>> > source of >>>>>>> > > truth. >>>>>>> > > > It just has the >>>>>>> > > > > > opportunity >>>>>>> > > > > > >> to deny access that we would >>>>>>> > otherwise >>>>>>> > > > grant, so it doesn't >>>>>>> > > > > > require that >>>>>>> > > > > > >> we break compatibility with >>>>>>> any S3 >>>>>>> > > features >>>>>>> > > > that conflict >>>>>>> > > > > with >>>>>>> > > > > > OPA's >>>>>>> > > > > > >> view of policy. >>>>>>> > > > > > >> >>>>>>> > > > > > >> Were we to change this so >>>>>>> that OPA >>>>>>> > > was the >>>>>>> > > > only source of >>>>>>> > > > > > truth, then >>>>>>> > > > > > >> we'd be left with two bad >>>>>>> > options: either >>>>>>> > > > reject all requests >>>>>>> > > > > > to modify >>>>>>> > > > > > >> policy and break existing >>>>>>> > > applications, or >>>>>>> > > > send all >>>>>>> > > > > policy/ACL >>>>>>> > > > > > >> information to OPA and >>>>>>> require >>>>>>> > every OPA >>>>>>> > > > policy script to >>>>>>> > > > > implement >>>>>>> > > > > > >> s3-compatible enforcement of >>>>>>> > them. I also >>>>>>> > > > don't see any >>>>>>> > > > > benefit >>>>>>> > > > > > to this >>>>>>> > > > > > >> model - why, if an client >>>>>>> wants >>>>>>> > to use s3 >>>>>>> > > > policy to >>>>>>> > > > > restrict a >>>>>>> > > > > > certain >>>>>>> > > > > > >> access, would OPA want to >>>>>>> override >>>>>>> > > that and >>>>>>> > > > grant access >>>>>>> > > > > instead? >>>>>>> > > > > > >> >>>>>>> > > > > > >> > I could live more with the >>>>>>> > latency >>>>>>> > > issue >>>>>>> > > > but wouldn’t >>>>>>> > > > > like it. >>>>>>> > > > > > >> > For the second, I can >>>>>>> understand >>>>>>> > > the idea >>>>>>> > > > of having >>>>>>> > > > > > simplification for >>>>>>> > > > > > >> > auditing the access but I’m >>>>>>> > not that >>>>>>> > > > convinced to take the >>>>>>> > > > > > burden of >>>>>>> > > > > > >> > being “the special” one >>>>>>> that >>>>>>> > nobody >>>>>>> > > wants >>>>>>> > > > to work with. >>>>>>> > > > > So, I >>>>>>> > > > > > would >>>>>>> > > > > > >> > love to see the full >>>>>>> fledged >>>>>>> > support of >>>>>>> > > > setting the >>>>>>> > > > > policy by >>>>>>> > > > > > clients, >>>>>>> > > > > > >> > no matter what the result >>>>>>> > would be in >>>>>>> > > > terms of >>>>>>> > > > > implementing it to >>>>>>> > > > > > >> > interact with OPA. Instead, >>>>>>> > having an >>>>>>> > > > additional >>>>>>> > > > > requirement to >>>>>>> > > > > > >> > implement additional >>>>>>> handling >>>>>>> > to set >>>>>>> > > > policies different >>>>>>> > > > > from >>>>>>> > > > > > what S3 >>>>>>> > > > > > >> > actually provides would >>>>>>> > require special >>>>>>> > > > clients first and >>>>>>> > > > > > secondly an >>>>>>> > > > > > >> > additional path to OPA with >>>>>>> > all the >>>>>>> > > > additional burden >>>>>>> > > > > to tweak >>>>>>> > > > > > >> > security to allow this path >>>>>>> > to OPA. I >>>>>>> > > > feel that the first >>>>>>> > > > > > wouldn’t >>>>>>> > > > > > >> > happen (special clients) >>>>>>> and >>>>>>> > the second >>>>>>> > > > in practice not >>>>>>> > > > > > either because >>>>>>> > > > > > >> > of security constraints by >>>>>>> > the OPA >>>>>>> > > admin >>>>>>> > > > folks. >>>>>>> > > > > > >> > >>>>>>> > > > > > >> > G, >>>>>>> > > > > > >> > -matt >>>>>>> > > > > > >> > >>>>>>> > > > > > >> > —————————————————— >>>>>>> > > > > > >> > Matthias Muench >>>>>>> > > > > > >> > Senior Specialist Solution >>>>>>> > Architect >>>>>>> > > > > > >> > EMEA Storage Specialist >>>>>>> > > > > > >> > matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>> >>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>> >>>>>>> > > > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>> >>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>>> >>>>>>> > > > > > <mailto: >>>>>>> matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>> >>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>> >>>>>>> > > > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>> >>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx> >>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx >>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>>>> >>>>>>> > > > <mailto:mmuench@xxxxxxxxxx >>>>>>> > <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>> >>>>>>> > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>> >>>>>>> > > > > <mailto:mmuench@xxxxxxxxxx >>>>>>> > <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>> >>>>>>> > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>>> >>>>>>> > > > > > <mailto:mmuench@xxxxxxxxxx >>>>>>> > <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>> >>>>>>> > > > <mailto:mmuench@xxxxxxxxxx >>>>>>> > <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>> >>>>>>> > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>> >>>>>>> > > > <mailto:mmuench@xxxxxxxxxx >>>>>>> > <mailto:mmuench@xxxxxxxxxx> >>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx >>>>>>> >>>>>> >>>>>>> > > > > > >> > Phone: +49-160-92654111 >>>>>>> > > > <tel:+49-160-92654111> >>>>>>> > > > > > >> > >>>>>>> > > > > > >> > Red Hat GmbH >>>>>>> > > > > > >> > Werner-von-Siemens-Ring 14 >>>>>>> > > > <x-apple-data-detectors://2/1> >>>>>>> > > > > > >> > 85630 Grasbrunn >>>>>>> > > > <x-apple-data-detectors://2/1> >>>>>>> > > > > > >> > Germany >>>>>>> > <x-apple-data-detectors://2/1> >>>>>>> > > > > > >> > >>>>>>> > > > > > >>>>>>> > > > > >>>>>>> > > > >>>>>>> > > >>>>>>> > >>>>>>> _______________________________________________________________________ >>>>>>> > > > > > >> > Red Hat GmbH, >>>>>>> > http://www.de.redhat.com >>>>>>> > > > > <http://de.redhat.com/> · >>>>>>> > > > > > >> > Registered seat: Grasbrunn, >>>>>>> > Commercial >>>>>>> > > > register: >>>>>>> > > > > Amtsgericht >>>>>>> > > > > > Muenchen >>>>>>> > > > > > >> > HRB 153243 · Managing >>>>>>> Directors: >>>>>>> > > Charles >>>>>>> > > > Cachera, Michael >>>>>>> > > > > > O'Neill, Tom >>>>>>> > > > > > >> > Savage, Eric Shander >>>>>>> > > > > > >> > >>>>>>> > > > > > >> >> On Jan 28, 2020, at >>>>>>> 15:02, Seena >>>>>>> > > Fallah >>>>>>> > > > > > <seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>>> >>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>>>> wrote: >>>>>>> > > > > > >> >> >>>>>>> > > > > > >> >> >>>>>>> > > > > > >> >> Amazon AWS S3 has two >>>>>>> type of >>>>>>> > > policies. >>>>>>> > > > One from bucket >>>>>>> > > > > > policy and >>>>>>> > > > > > >> >> one form IAM. I think it >>>>>>> > could be >>>>>>> > > better >>>>>>> > > > to have two >>>>>>> > > > > > policies models >>>>>>> > > > > > >> >> in Ceph one from bucket >>>>>>> > policy and one >>>>>>> > > > form OPA if its >>>>>>> > > > > enable. >>>>>>> > > > > > >> >> So if you are okay we can >>>>>>> change >>>>>>> > > the PR >>>>>>> > > > to make bucket >>>>>>> > > > > > policy enabled >>>>>>> > > > > > >> >> when OPA is enabled, too. >>>>>>> > Because now >>>>>>> > > > bucket policies not >>>>>>> > > > > > working >>>>>>> > > > > > >> >> when OPA integration is >>>>>>> enabled. >>>>>>> > > > > > >> >> >>>>>>> > > > > > >> >> On Tue, Jan 28, 2020 at >>>>>>> 2:57 AM >>>>>>> > > Seena Fallah >>>>>>> > > > > > <seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>>> >>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>>>> >>>>>>> > > > > > >> >> >>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>>> >>>>>>> > > > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>>> >>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto: >>>>>>> seenafallah@xxxxxxxxx>> >>>>>>> > > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx> >>>>>>> > > <mailto:seenafallah@xxxxxxxxx >>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>>>>> wrote: >>>>>>> > > > > > >> >> >>>>>>> > > > > > >> >> Matt When OPA >>>>>>> integration is >>>>>>> > > enabled >>>>>>> > > > S3 policies >>>>>>> > > > > doesn’t >>>>>>> > > > > > work! If >>>>>>> > > > > > >> >> you want them to be >>>>>>> > worked we >>>>>>> > > should >>>>>>> > > > bypass S3 >>>>>>> > > > > policies >>>>>>> > > > > > to OPA >>>>>>> > > > > > >> >> for being applied and >>>>>>> > worked. >>>>>>> > > > > > >> >> Here we have conflict >>>>>>> in OPA >>>>>>> > > > integration with S3 >>>>>>> > > > > policies! >>>>>>> > > > > > >> >> >>>>>>> > > > > > >> >> On Tue, Jan 28, 2020 >>>>>>> at 2:52 >>>>>>> > > AM Matt >>>>>>> > > > Benjamin >>>>>>> > > > > > >> >> <mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>> >>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>>> >>>>>>> > > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>> >>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>>>> >>>>>>> > > > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>> >>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>>> >>>>>>> > > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx >>>>>>> >>> >>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>> >>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx >>>>>>> > <mailto:mbenjami@xxxxxxxxxx> >>>>>>> > >>>>>> >>>>>> > _______________________________________________ > Dev mailing list -- dev@xxxxxxx > To unsubscribe send an email to dev-leave@xxxxxxx _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
