In a code base (https://github.com/ceph/ceph/pull/32294/files#diff-bb1cc6889525611b45d6af6d5035ffa8) in rgw_process_authenticated function we have an IF that check for OPA authorization result after that if it was OK (200) it will call verify_permission() that check with radosgw policies like bucket policy, bucket acl, user acl, ....
Now if I create a policy in OPA that user A can access user B's bucket so OPA return OK (200) to radosgw and radosgw will call verify_permission() but in radosgw policies we don't have a policy that user A has access to user B so the request will be reject.
So OPA can just reject requests and if OPA accept a request radosgw should also has that policy for being accept to execute.
I think it's better to make OPA single source of truth and send bucket policy, acl, ... to OPA in OPA request to be authorized.
On Fri, Feb 21, 2020 at 2:09 PM Abhishek Lekshmanan <abhishek@xxxxxxxx> wrote:
Seena Fallah <seenafallah@xxxxxxxxx> writes:
> I have make this PR to make OPA single source of truth so OPA can get
> access to not owned bucket for user. And also in OPA request we send
> bucket policy, acl, ... to OPA so we can support them when OPA integration
> is enabled. We will authorized user based on bucket policy, acl, ... and
> OPA policies in OPA server.
> Can you please take a look at here?
> https://github.com/ceph/ceph/pull/32294
How does OPA decide on S3 conditionals wrt Policy here,
we already support a few bucket attribute or Object based conditionals
https://docs.ceph.com/docs/master/radosgw/bucketpolicy/#bucket-related-operations
which can only be evaluated after reading the object in question for eg.
These allow for eg. Bucket A would have access to User B to download
object iff the object tag has an attribute openaccess = true.
Additionally for buckets that are public
https://github.com/ceph/ceph/pull/30033 would prevent access when
IgnorePublicAcls are true.
>
> On Tue, Feb 18, 2020 at 9:52 AM Seena Fallah <seenafallah@xxxxxxxxx> wrote:
>
>> I have seen another bad scenario that we have two source of truth. If we
>> get access to user for a bucket that he/she doesn't own it in OPA we can
>> perform this action because op->verify_permission() will return -EACCES and
>> so in rgw_process_authenticated function rgw_opa_authorize will not check!
>>
>> I think it's better to have one source of truth when we enabled OPA
>> integration so we can send bucket policy, acl, ... to OPA on each
>> request to be authorized.
>>
>> Do you have any other suggestion?
>>
>> On Tue, Feb 18, 2020 at 1:01 AM Seena Fallah <seenafallah@xxxxxxxxx>
>> wrote:
>>
>>> I have changed the PR for sending bucket policy, bucket acl, iam policy
>>> and user acl as a field to OPA request so OPA can decision based on this
>>> parameters and we can have a external authorization for our organization
>>> like AWS IAM. So we can have an S3 service that authorize based on bucket
>>> policy and organizations IAM.
>>> https://github.com/ceph/ceph/pull/32294
>>> Could you please review this?
>>>
>>> On Mon, Feb 17, 2020 at 2:39 PM Seena Fallah <seenafallah@xxxxxxxxx>
>>> wrote:
>>>
>>>> Also we can add bucket policy result field to OPA request so in OPA
>>>> policies we can act based on bucket policy results.
>>>>
>>>> Are you agree with it?
>>>>
>>>> On Thu, Feb 13, 2020 at 12:17 PM Seena Fallah <seenafallah@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> In bucket policy we have an Effect::Pass in validation result so if we
>>>>> just put OPA authorization in case of Effect::Pass I think it will be close
>>>>> to what AWS do.
>>>>>
>>>>> On Fri, Feb 7, 2020 at 1:37 PM Seena Fallah <seenafallah@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> In organizations that has many services and want to have a centralized
>>>>>> authorization server this will be a good solution to have.
>>>>>>
>>>>>> I mean that when we just authorize user ReadOnly in OPA but give write
>>>>>> access via bucket policy, the user can’t write because OPA is rejecting.
>>>>>>
>>>>>> I think we can just weight bucket policy upper that OPA so bucket
>>>>>> policies that apply policies specific than OPA policies can accept and
>>>>>> reject at first then OPA would authorize that request. I mean bucket policy
>>>>>> specify policy more specific (On bucket or on object) than OPA (OPA can set
>>>>>> policy globally too like giving ReadOnly to all buckets) so it's better to
>>>>>> first check bucket policy then check for OPA. this could be easy solve this
>>>>>> problem.
>>>>>>
>>>>>> On Thu, Feb 6, 2020 at 10:11 PM Casey Bodley <cbodley@xxxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> On 2/6/20 11:58 AM, Seena Fallah wrote:
>>>>>>> > The main goal of using OPA like as AWS IAM is having an external
>>>>>>> > authorization so we can have out own management on policies from
>>>>>>> > external source of truth (OPA) too.
>>>>>>> >
>>>>>>> > I think it’s better to handle bucket policy with OPA as well as AWS
>>>>>>> > does so we can have a better S3 service :)
>>>>>>>
>>>>>>> Can you expand on why that's better than the model I suggested
>>>>>>> earlier
>>>>>>> in the thread, where a centralized policy service uses radosgw's
>>>>>>> existing IAM APIs to manage policy instead of requiring radosgw to
>>>>>>> call
>>>>>>> out to an external service for every request?
>>>>>>>
>>>>>>> I'd also like to clarify what you mean when you say "handle bucket
>>>>>>> policy with OPA" - my understanding is that it's not something that
>>>>>>> OPA
>>>>>>> itself does, but something very specific to your own product's OPA
>>>>>>> policy script. Am I getting that right? If so, it sounds like you're
>>>>>>> trying to re-engineer our OPA integration in a way that a) is not
>>>>>>> useful
>>>>>>> to OPA users in general, and b) duplicates functionality that radosgw
>>>>>>> already provides.
>>>>>>>
>>>>>>> For OPA users that just want the ability to write simple scripts to
>>>>>>> customize authorization for their environment, I think our current
>>>>>>> level
>>>>>>> of OPA integration is sufficient.
>>>>>>>
>>>>>>> >
>>>>>>> > On Thu, Feb 6, 2020 at 8:16 PM Casey Bodley <cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>> wrote:
>>>>>>> >
>>>>>>> > Is there an advantage to doing this in OPA over radosgw? Have
>>>>>>> you
>>>>>>> > looked
>>>>>>> > at using our PutUserPolicy[1] APIs instead? We support both
>>>>>>> user and
>>>>>>> > bucket policy, and (as far as I know) handle the intersection of
>>>>>>> > the two
>>>>>>> > as you'd expect.
>>>>>>> >
>>>>>>> >
>>>>>>> https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html
>>>>>>> >
>>>>>>> > On 2/5/20 9:55 AM, Seena Fallah wrote:
>>>>>>> > > I'm trying to implement AWS IAM with OPA so I can have
>>>>>>> external
>>>>>>> > > authorization for my S3 service and also have an active bucket
>>>>>>> > ACL and
>>>>>>> > > bucket policy.
>>>>>>> > >
>>>>>>> > > On Wed, Feb 5, 2020 at 6:11 PM Casey Bodley <
>>>>>>> cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > > I'm confused by your references to AWS IAM. Are you
>>>>>>> talking
>>>>>>> > about
>>>>>>> > > radosgw user policy? Or are you trying to implement IAM
>>>>>>> policy
>>>>>>> > > inside of
>>>>>>> > > OPA?
>>>>>>> > >
>>>>>>> > > On 2/5/20 8:54 AM, Seena Fallah wrote:
>>>>>>> > > > Hi all.
>>>>>>> > > >
>>>>>>> > > > Any updates here? :)
>>>>>>> > > >
>>>>>>> > > > On Sat, Feb 1, 2020 at 10:37 AM Seena Fallah
>>>>>>> > > <seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx> <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>>
>>>>>>> > > wrote:
>>>>>>> > > >
>>>>>>> > > > I should also mention that if we get access to
>>>>>>> bucket
>>>>>>> > via bucket
>>>>>>> > > > policy and reject it via AWS IAM, the request will
>>>>>>> > reject so I
>>>>>>> > > > think we should make a new behavior at what we
>>>>>>> should do
>>>>>>> > > with this
>>>>>>> > > > two source of truth?
>>>>>>> > > >
>>>>>>> > > > On Sat, Feb 1, 2020 at 10:33 AM Seena Fallah
>>>>>>> > > > <seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>
>>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>
>>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>>>>
>>>>>>> wrote:
>>>>>>> > > >
>>>>>>> > > > Yes but the main problem is when the policy
>>>>>>> isn't
>>>>>>> > set in AWS
>>>>>>> > > > IAM for example a user has
>>>>>>> only AmazonS3ReadOnlyAccess
>>>>>>> > > and we
>>>>>>> > > > give PutObject policy via bucket policy, user
>>>>>>> can
>>>>>>> > put object
>>>>>>> > > > to that bucket but in radosgw, OPA will deny
>>>>>>> this
>>>>>>> > process
>>>>>>> > > > because there is only ReadOnlyAccess to that
>>>>>>> > bucket for user
>>>>>>> > > > and radosgw will not check bucket policy that
>>>>>>> gave
>>>>>>> > access to
>>>>>>> > > > user.
>>>>>>> > > > I think we should weight bucket policy over OPA
>>>>>>> so
>>>>>>> > if bucket
>>>>>>> > > > policy accept that request it doesn't need to be
>>>>>>> > checked
>>>>>>> > > with
>>>>>>> > > > OPA BUT if there is no policy according to that
>>>>>>> > request it
>>>>>>> > > > should check by OPA because if the policy
>>>>>>> > according to that
>>>>>>> > > > request isn't set bucket policy will reject that
>>>>>>> > request
>>>>>>> > > so it
>>>>>>> > > > against failed!
>>>>>>> > > >
>>>>>>> > > > On Fri, Jan 31, 2020 at 1:30 AM Casey Bodley
>>>>>>> > > > <cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>>>
>>>>>>> wrote:
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > On 1/30/20 2:18 PM, Seena Fallah wrote:
>>>>>>> > > > > Hi Casey,
>>>>>>> > > > >
>>>>>>> > > > > The main problem now is when OPA
>>>>>>> integration is
>>>>>>> > > enabled
>>>>>>> > > > bucket
>>>>>>> > > > > policies aren’t work!
>>>>>>> > > > > I have checked AWS S3 that what is doing
>>>>>>> > when both
>>>>>>> > > > bucket policy and
>>>>>>> > > > > IAM policy (the policy is set with AWS
>>>>>>> panel
>>>>>>> > in IAM
>>>>>>> > > > section) is set it
>>>>>>> > > > > will OR between two of them so now in
>>>>>>> > radosgw S3 we
>>>>>>> > > > don’t have this
>>>>>>> > > > > feature and bucket policies won’t work
>>>>>>> when OPA
>>>>>>> > > > integration is enabled.
>>>>>>> > > > >
>>>>>>> > > > > So I think it’s better to active this
>>>>>>> > feature and
>>>>>>> > > > enabled bucket
>>>>>>> > > > > policy when OpA integration is enabled.
>>>>>>> > > > >
>>>>>>> > > > > There is two solutions here in this
>>>>>>> > discussion for
>>>>>>> > > > enabling bucket
>>>>>>> > > > > policy on OPA integration:
>>>>>>> > > > > 1. Send bucket policy on set/del actions
>>>>>>> to
>>>>>>> > OPA server
>>>>>>> > > > to be apply on
>>>>>>> > > > > OPA policy rules so in this case the
>>>>>>> source
>>>>>>> > of truth
>>>>>>> > > > will be OPA (the
>>>>>>> > > > > state that we have now in OPA integration)
>>>>>>> > and so
>>>>>>> > > these
>>>>>>> > > > policies that
>>>>>>> > > > > sent from bucket policy will be applied.
>>>>>>> > > > > 2. OR between bucket policy and OPA
>>>>>>> policy like
>>>>>>> > > AWS S3.
>>>>>>> > > > So there is
>>>>>>> > > > > two source of truth in this case and if
>>>>>>> any of
>>>>>>> > > them deny
>>>>>>> > > > the request,
>>>>>>> > > > > the request will be denied.
>>>>>>> > > >
>>>>>>> > > > What you described here in 2. is exactly
>>>>>>> how it
>>>>>>> > > currently
>>>>>>> > > > works.
>>>>>>> > > >
>>>>>>> > > > >
>>>>>>> > > > > Do have any other solutions we have here
>>>>>>> and
>>>>>>> > which of
>>>>>>> > > > these solutions
>>>>>>> > > > > do you prefer to have?
>>>>>>> > > > >
>>>>>>> > > > > On Thu, Jan 30, 2020 at 10:21 PM Casey
>>>>>>> Bodley
>>>>>>> > > > <cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx> <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>>
>>>>>>> > > > > <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>
>>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx
>>>>>>> >>>>>
>>>>>>> > > > wrote:
>>>>>>> > > > >
>>>>>>> > > > > Hi Seena,
>>>>>>> > > > >
>>>>>>> > > > > I think it would probably help if you
>>>>>>> could
>>>>>>> > > describe
>>>>>>> > > > your use case
>>>>>>> > > > > here,
>>>>>>> > > > > and what role you want OPA to play in
>>>>>>> the
>>>>>>> > > > interpretation of these
>>>>>>> > > > > bucket
>>>>>>> > > > > policies. In other words, what is it
>>>>>>> > that your OPA
>>>>>>> > > > policy is doing
>>>>>>> > > > > with
>>>>>>> > > > > these bucket policy documents that
>>>>>>> shouldn't
>>>>>>> > > be done
>>>>>>> > > > within radosgw?
>>>>>>> > > > >
>>>>>>> > > > > On 1/30/20 1:09 AM, Seena Fallah
>>>>>>> wrote:
>>>>>>> > > > > > So Matt what should we have done
>>>>>>> with
>>>>>>> > bucket
>>>>>>> > > > policy if we enable
>>>>>>> > > > > OPA
>>>>>>> > > > > > integration?
>>>>>>> > > > > >
>>>>>>> > > > > > On Thu, Jan 30, 2020 at 1:45 AM Matt
>>>>>>> > Benjamin
>>>>>>> > > > > <mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>>
>>>>>>> > > > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>>>>>> wrote:
>>>>>>> > > > > >
>>>>>>> > > > > > I think we should not be
>>>>>>> > introducing new
>>>>>>> > > > special case
>>>>>>> > > > > behavior, nor
>>>>>>> > > > > > sending policy documents to OPA,
>>>>>>> > which from
>>>>>>> > > > what we have
>>>>>>> > > > > heard and
>>>>>>> > > > > > read, intends to make no use of
>>>>>>> them.
>>>>>>> > > > > >
>>>>>>> > > > > > Matt
>>>>>>> > > > > >
>>>>>>> > > > > > On Wed, Jan 29, 2020 at 4:45 PM
>>>>>>> > Seena Fallah
>>>>>>> > > > > > <seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>>
>>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>>>> wrote:
>>>>>>> > > > > > >
>>>>>>> > > > > > > I think it’s better to OR
>>>>>>> > between two
>>>>>>> > > of the
>>>>>>> > > > bucket
>>>>>>> > > > > policies and
>>>>>>> > > > > > OPA policies. So if one of them
>>>>>>> reject
>>>>>>> > > certain
>>>>>>> > > > access the
>>>>>>> > > > > request
>>>>>>> > > > > > will reject as AWS do on its IAM
>>>>>>> > and bucket
>>>>>>> > > > policy.
>>>>>>> > > > > > > Are you okay with this idea?
>>>>>>> > > > > > >
>>>>>>> > > > > > > On Wed, Jan 29, 2020 at 11:13
>>>>>>> PM
>>>>>>> > Casey
>>>>>>> > > Bodley
>>>>>>> > > > > > <cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>
>>>>>>> > > > <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>>
>>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>
>>>>>>> > > > <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx> <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>>>>
>>>>>>> > > > > <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>
>>>>>>> > > > <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>>
>>>>>>> > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx>>
>>>>>>> > > > <mailto:cbodley@xxxxxxxxxx
>>>>>>> > <mailto:cbodley@xxxxxxxxxx>
>>>>>>> > > <mailto:cbodley@xxxxxxxxxx <mailto:cbodley@xxxxxxxxxx
>>>>>>> >>>>>>
>>>>>>> > wrote:
>>>>>>> > > > > > >>
>>>>>>> > > > > > >>
>>>>>>> > > > > > >> On 1/28/20 2:45 PM, Matthias
>>>>>>> Muench
>>>>>>> > > wrote:
>>>>>>> > > > > > >> > Hi,
>>>>>>> > > > > > >> > I think making Ceph special
>>>>>>> > to what the
>>>>>>> > > > rest of the clients
>>>>>>> > > > > > in the
>>>>>>> > > > > > >> > world would expect would
>>>>>>> be a bit
>>>>>>> > > off the
>>>>>>> > > > idea of providing
>>>>>>> > > > > > S3 like
>>>>>>> > > > > > >> > service.
>>>>>>> > > > > > >> > To my understanding,
>>>>>>> setting
>>>>>>> > OPA to be
>>>>>>> > > > the source of
>>>>>>> > > > > truth would
>>>>>>> > > > > > >> > introduce latency (based on
>>>>>>> > Casey’s
>>>>>>> > > > comments) and will not
>>>>>>> > > > > > allow to
>>>>>>> > > > > > >> > set policies (based on
>>>>>>> Seena).
>>>>>>> > > > > > >> > The first one brings us
>>>>>>> > towards harder
>>>>>>> > > > latency and
>>>>>>> > > > > especially
>>>>>>> > > > > > >> > depending on extern systems
>>>>>>> > resource
>>>>>>> > > > capability (assume
>>>>>>> > > > > central
>>>>>>> > > > > > >> > resource as the idea is and
>>>>>>> > > therefor not
>>>>>>> > > > necessarily really
>>>>>>> > > > > > “in reach”
>>>>>>> > > > > > >> > within an acceptable
>>>>>>> latency,
>>>>>>> > > routing in
>>>>>>> > > > addition,
>>>>>>> > > > > etc.). The
>>>>>>> > > > > > second
>>>>>>> > > > > > >> > one says simply that this
>>>>>>> would
>>>>>>> > > break any
>>>>>>> > > > existing
>>>>>>> > > > > > compatibility with
>>>>>>> > > > > > >> > clients and use cases. To
>>>>>>> me it
>>>>>>> > > looks not
>>>>>>> > > > that good to
>>>>>>> > > > > loose
>>>>>>> > > > > > on both ends.
>>>>>>> > > > > > >>
>>>>>>> > > > > > >> Agreed. Even if one has to
>>>>>>> > opt-in to this
>>>>>>> > > > broken s3
>>>>>>> > > > > > compatiblity, I'm
>>>>>>> > > > > > >> skeptical that users will
>>>>>>> find this
>>>>>>> > > to be a
>>>>>>> > > > compelling target
>>>>>>> > > > > > for their
>>>>>>> > > > > > >> applications.
>>>>>>> > > > > > >>
>>>>>>> > > > > > >> The existing prototype of OPA
>>>>>>> > integration
>>>>>>> > > > sends this
>>>>>>> > > > > authorization
>>>>>>> > > > > > >> request to OPA -in addition
>>>>>>> to-
>>>>>>> > radosgw's
>>>>>>> > > > own authorization
>>>>>>> > > > > > logic, where
>>>>>>> > > > > > >> we consult any of our
>>>>>>> user/bucket
>>>>>>> > > policies
>>>>>>> > > > or ACLs that
>>>>>>> > > > > apply.
>>>>>>> > > > > > In this
>>>>>>> > > > > > >> model, OPA is not the only
>>>>>>> > source of
>>>>>>> > > truth.
>>>>>>> > > > It just has the
>>>>>>> > > > > > opportunity
>>>>>>> > > > > > >> to deny access that we would
>>>>>>> > otherwise
>>>>>>> > > > grant, so it doesn't
>>>>>>> > > > > > require that
>>>>>>> > > > > > >> we break compatibility with
>>>>>>> any S3
>>>>>>> > > features
>>>>>>> > > > that conflict
>>>>>>> > > > > with
>>>>>>> > > > > > OPA's
>>>>>>> > > > > > >> view of policy.
>>>>>>> > > > > > >>
>>>>>>> > > > > > >> Were we to change this so
>>>>>>> that OPA
>>>>>>> > > was the
>>>>>>> > > > only source of
>>>>>>> > > > > > truth, then
>>>>>>> > > > > > >> we'd be left with two bad
>>>>>>> > options: either
>>>>>>> > > > reject all requests
>>>>>>> > > > > > to modify
>>>>>>> > > > > > >> policy and break existing
>>>>>>> > > applications, or
>>>>>>> > > > send all
>>>>>>> > > > > policy/ACL
>>>>>>> > > > > > >> information to OPA and
>>>>>>> require
>>>>>>> > every OPA
>>>>>>> > > > policy script to
>>>>>>> > > > > implement
>>>>>>> > > > > > >> s3-compatible enforcement of
>>>>>>> > them. I also
>>>>>>> > > > don't see any
>>>>>>> > > > > benefit
>>>>>>> > > > > > to this
>>>>>>> > > > > > >> model - why, if an client
>>>>>>> wants
>>>>>>> > to use s3
>>>>>>> > > > policy to
>>>>>>> > > > > restrict a
>>>>>>> > > > > > certain
>>>>>>> > > > > > >> access, would OPA want to
>>>>>>> override
>>>>>>> > > that and
>>>>>>> > > > grant access
>>>>>>> > > > > instead?
>>>>>>> > > > > > >>
>>>>>>> > > > > > >> > I could live more with the
>>>>>>> > latency
>>>>>>> > > issue
>>>>>>> > > > but wouldn’t
>>>>>>> > > > > like it.
>>>>>>> > > > > > >> > For the second, I can
>>>>>>> understand
>>>>>>> > > the idea
>>>>>>> > > > of having
>>>>>>> > > > > > simplification for
>>>>>>> > > > > > >> > auditing the access but I’m
>>>>>>> > not that
>>>>>>> > > > convinced to take the
>>>>>>> > > > > > burden of
>>>>>>> > > > > > >> > being “the special” one
>>>>>>> that
>>>>>>> > nobody
>>>>>>> > > wants
>>>>>>> > > > to work with.
>>>>>>> > > > > So, I
>>>>>>> > > > > > would
>>>>>>> > > > > > >> > love to see the full
>>>>>>> fledged
>>>>>>> > support of
>>>>>>> > > > setting the
>>>>>>> > > > > policy by
>>>>>>> > > > > > clients,
>>>>>>> > > > > > >> > no matter what the result
>>>>>>> > would be in
>>>>>>> > > > terms of
>>>>>>> > > > > implementing it to
>>>>>>> > > > > > >> > interact with OPA. Instead,
>>>>>>> > having an
>>>>>>> > > > additional
>>>>>>> > > > > requirement to
>>>>>>> > > > > > >> > implement additional
>>>>>>> handling
>>>>>>> > to set
>>>>>>> > > > policies different
>>>>>>> > > > > from
>>>>>>> > > > > > what S3
>>>>>>> > > > > > >> > actually provides would
>>>>>>> > require special
>>>>>>> > > > clients first and
>>>>>>> > > > > > secondly an
>>>>>>> > > > > > >> > additional path to OPA with
>>>>>>> > all the
>>>>>>> > > > additional burden
>>>>>>> > > > > to tweak
>>>>>>> > > > > > >> > security to allow this path
>>>>>>> > to OPA. I
>>>>>>> > > > feel that the first
>>>>>>> > > > > > wouldn’t
>>>>>>> > > > > > >> > happen (special clients)
>>>>>>> and
>>>>>>> > the second
>>>>>>> > > > in practice not
>>>>>>> > > > > > either because
>>>>>>> > > > > > >> > of security constraints by
>>>>>>> > the OPA
>>>>>>> > > admin
>>>>>>> > > > folks.
>>>>>>> > > > > > >> >
>>>>>>> > > > > > >> > G,
>>>>>>> > > > > > >> > -matt
>>>>>>> > > > > > >> >
>>>>>>> > > > > > >> > ——————————————————
>>>>>>> > > > > > >> > Matthias Muench
>>>>>>> > > > > > >> > Senior Specialist Solution
>>>>>>> > Architect
>>>>>>> > > > > > >> > EMEA Storage Specialist
>>>>>>> > > > > > >> > matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>
>>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>>
>>>>>>> > > > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>
>>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>>>
>>>>>>> > > > > > <mailto:
>>>>>>> matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>
>>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>>
>>>>>>> > > > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>
>>>>>>> > > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>
>>>>>>> > > <mailto:matthias.muench@xxxxxxxxxx
>>>>>>> > <mailto:matthias.muench@xxxxxxxxxx>>>>>
>>>>>>> > > > <mailto:mmuench@xxxxxxxxxx
>>>>>>> > <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>
>>>>>>> > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>>
>>>>>>> > > > > <mailto:mmuench@xxxxxxxxxx
>>>>>>> > <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>
>>>>>>> > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>>>
>>>>>>> > > > > > <mailto:mmuench@xxxxxxxxxx
>>>>>>> > <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mmuench@xxxxxxxxxx
>>>>>>> > <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>>
>>>>>>> > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mmuench@xxxxxxxxxx
>>>>>>> > <mailto:mmuench@xxxxxxxxxx>
>>>>>>> > > <mailto:mmuench@xxxxxxxxxx <mailto:mmuench@xxxxxxxxxx
>>>>>>> >>>>>>
>>>>>>> > > > > > >> > Phone: +49-160-92654111
>>>>>>> > > > <tel:+49-160-92654111>
>>>>>>> > > > > > >> >
>>>>>>> > > > > > >> > Red Hat GmbH
>>>>>>> > > > > > >> > Werner-von-Siemens-Ring 14
>>>>>>> > > > <x-apple-data-detectors://2/1>
>>>>>>> > > > > > >> > 85630 Grasbrunn
>>>>>>> > > > <x-apple-data-detectors://2/1>
>>>>>>> > > > > > >> > Germany
>>>>>>> > <x-apple-data-detectors://2/1>
>>>>>>> > > > > > >> >
>>>>>>> > > > > >
>>>>>>> > > > >
>>>>>>> > > >
>>>>>>> > >
>>>>>>> >
>>>>>>> _______________________________________________________________________
>>>>>>> > > > > > >> > Red Hat GmbH,
>>>>>>> > http://www.de.redhat.com
>>>>>>> > > > > <http://de.redhat.com/> ·
>>>>>>> > > > > > >> > Registered seat: Grasbrunn,
>>>>>>> > Commercial
>>>>>>> > > > register:
>>>>>>> > > > > Amtsgericht
>>>>>>> > > > > > Muenchen
>>>>>>> > > > > > >> > HRB 153243 · Managing
>>>>>>> Directors:
>>>>>>> > > Charles
>>>>>>> > > > Cachera, Michael
>>>>>>> > > > > > O'Neill, Tom
>>>>>>> > > > > > >> > Savage, Eric Shander
>>>>>>> > > > > > >> >
>>>>>>> > > > > > >> >> On Jan 28, 2020, at
>>>>>>> 15:02, Seena
>>>>>>> > > Fallah
>>>>>>> > > > > > <seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>>
>>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>>>> wrote:
>>>>>>> > > > > > >> >>
>>>>>>> > > > > > >> >>
>>>>>>> > > > > > >> >> Amazon AWS S3 has two
>>>>>>> type of
>>>>>>> > > policies.
>>>>>>> > > > One from bucket
>>>>>>> > > > > > policy and
>>>>>>> > > > > > >> >> one form IAM. I think it
>>>>>>> > could be
>>>>>>> > > better
>>>>>>> > > > to have two
>>>>>>> > > > > > policies models
>>>>>>> > > > > > >> >> in Ceph one from bucket
>>>>>>> > policy and one
>>>>>>> > > > form OPA if its
>>>>>>> > > > > enable.
>>>>>>> > > > > > >> >> So if you are okay we can
>>>>>>> change
>>>>>>> > > the PR
>>>>>>> > > > to make bucket
>>>>>>> > > > > > policy enabled
>>>>>>> > > > > > >> >> when OPA is enabled, too.
>>>>>>> > Because now
>>>>>>> > > > bucket policies not
>>>>>>> > > > > > working
>>>>>>> > > > > > >> >> when OPA integration is
>>>>>>> enabled.
>>>>>>> > > > > > >> >>
>>>>>>> > > > > > >> >> On Tue, Jan 28, 2020 at
>>>>>>> 2:57 AM
>>>>>>> > > Seena Fallah
>>>>>>> > > > > > <seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>>
>>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>> <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>>>
>>>>>>> > > > > > >> >>
>>>>>>> > <mailto:seenafallah@xxxxxxxxx <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>>
>>>>>>> > > > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>>
>>>>>>> > > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx <mailto:
>>>>>>> seenafallah@xxxxxxxxx>>
>>>>>>> > > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>
>>>>>>> > > <mailto:seenafallah@xxxxxxxxx
>>>>>>> > <mailto:seenafallah@xxxxxxxxx>>>>>>> wrote:
>>>>>>> > > > > > >> >>
>>>>>>> > > > > > >> >> Matt When OPA
>>>>>>> integration is
>>>>>>> > > enabled
>>>>>>> > > > S3 policies
>>>>>>> > > > > doesn’t
>>>>>>> > > > > > work! If
>>>>>>> > > > > > >> >> you want them to be
>>>>>>> > worked we
>>>>>>> > > should
>>>>>>> > > > bypass S3
>>>>>>> > > > > policies
>>>>>>> > > > > > to OPA
>>>>>>> > > > > > >> >> for being applied and
>>>>>>> > worked.
>>>>>>> > > > > > >> >> Here we have conflict
>>>>>>> in OPA
>>>>>>> > > > integration with S3
>>>>>>> > > > > policies!
>>>>>>> > > > > > >> >>
>>>>>>> > > > > > >> >> On Tue, Jan 28, 2020
>>>>>>> at 2:52
>>>>>>> > > AM Matt
>>>>>>> > > > Benjamin
>>>>>>> > > > > > >> >> <mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>>
>>>>>>> > > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>>>
>>>>>>> > > > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>>
>>>>>>> > > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx
>>>>>>> >>>
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> > > <mailto:mbenjami@xxxxxxxxxx <mailto:mbenjami@xxxxxxxxxx>>
>>>>>>> > > > <mailto:mbenjami@xxxxxxxxxx
>>>>>>> > <mailto:mbenjami@xxxxxxxxxx>
>>>>>>> >
>>>>>>
>>>>>>
> _______________________________________________
> Dev mailing list -- dev@xxxxxxx
> To unsubscribe send an email to dev-leave@xxxxxxx
_______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
