You beat me by a few days on this. I was looking at something similar
some time ago, but I was thinking a bit in another direction.
I'm a bit verbose today, if you missed that from the previous mails, but
please bare with me.
tl;dr
- 'negotiation' should be 'discovery' by initiator queries only
- check with Coraid for their own feature detection method(if applicable)
- I propose a feature string based discovery method
- your coalesced read suggestion sounds confusing, please document and
split to vendor command
------------------On feature discovery------------
Since Coraid basically controls the protocol and they make actual
commercial products against it, it would be only natural to synchronize
with them on this before making any changes to it.
I don't have access to traffic dumps or actual hardware, so I'd like to
first ask Ed Cashin if Coraid has implemented and/or uses any vendor
specific extensions that he can talk about.
Regarding 'feature negotiation', I believe that this cannot be used as
the proper term here. Since AoE is connectionless, it would incur that
the server should not care or have any state about a potential client.
As such, I believe the onus should be on the initiator to determine what
the target supports and adapt to that. I believe 'feature detection' is
probably a better term.
To this end, since there is no requirement that both initiator and
target be written by the same vendor, I believe that last point of AoE
r11 point 2.6. Commands (Command Codes 0xf0-0xff are reserved for vendor
specific use) should be considered invalid without a proper, sanctioned,
mechanism in place to prevent collision of vendor specific uses that
could have dangerous side-effects.
Looking at command 1 (Query Config Information) there is nothing that
would allow for safe detection of features.
Providing vendor identification inside a config string could be lost in
normal usage. Obtaining the vendor/feature list from the ATA IDENTIFY
DEVICE command (ECh) might not be reliable for devices that actually do
ATA pass-trough to the respective drives (I've read about some
enclosures that expose the drive over AoE and they might want to do that).
I was looking to propose the following change
1. define a feature as
- a string of lower-case ascii letters representing the feature
- vendors features must be prefixed with a consistent (for that vendor)
upper-case ascii letters identifier
- features must have at most 15 characters
2. define Feature String as a comma separated list of at most 32
features (with a total maximum length of 512 bytes)
4. define Feature Response as a string of at most 31 bytes formed by
dash joining of
- a feature
- a hex number representing the command code the target uses for above
feature
- an optional argument(maximum of 12 alphanumeric chars)
4. define a Feature Response String as a comma separated list of at most
32 feature responses (with a total maximum length 1024 bytes)
5. allocate Command 1. CCmd 0x5(or 0xF) as 'test feature string'
CCmd 5: test feature string
Respond with a Feature Response String containing features responses
for features valid on this target.
Response could be empty(and would return an error if CCmd 0x5 is
unsupported)
6. An initiator should use this as follows.
- iterate own supported features
- initiate a query for CCmd 5 with argument set to feature string
- parse response and configure itself to use discovered features on the
command codes provided in the feature response string
- use supported features
-------------On coalesced read requests------------
I'm quite interested in how you manage to do this in WinAoE (from a
previous mail, I understand you are working with that as initiator)
without changing quite a bit of the logic, given that the Tag is
generated in the network scheduling code, not in the disk code.
I would ask that you move this code to a vendor reserved command code,
unless it's included in the official protocol.
In the same style you propose here, it would be interesting to
concatenate several write responses into fewer ones, since they are all
fired towards the target at once by the initiator.
My concerns with this kind of code have less to do with cpu usage and
more to do with latency. Sending one packet instead of multiple would
reduce the waiting on both ends. Receiving responses for writes would
also benefit due to the spinlock on the tag list on reply.
As an alternative that may be slightly easier to implement and manage, I
was thinking of sending only one read request for several sectors(set
sectors to the original number in the Disk IO request), while making it
clear to the target that they should be sent back with the same Tag but
in separate frames. Responses should fill the IRP buffer until it is
complete and only one IRP would need to be completed. This could allow
for less memory usage for the initiator, as less memory would be used to
track requests. One side effect would be that the timeout routine
currently used by WinAoE to reschedule missed packets would not be aware
of what parts of the buffer are missing. This would either mean
resetting the received counter and rescheduling all requests (or storing
a bitmap of them) or else re-request the entire buffer. The second
option would cost quite a bit of time if the requested area is large.
--The end!--
I'd say that this was my 2 cents, but it looks more like 2 dollars..
Thanks for reading!
On 10/06/2014 12:18 AM, Killer{R} wrote:
> Hello aoetools-discuss,
>
> ..coalesced read requests. Its quite an expensive to send 60-bytes
> packet per every MTU-sized read. I did experimental read requests
> coalesting in my forked project and it noticable decreased
> packets-per-second rate and also reduced target's CPU usage.
> I did this by appending bunch of such structures prepended by uchar
> indicated their count just after normal Read command packet:
> struct AtaCoalescedRead
> {
> uchar tag[4];
> uchar lba[6];
> uchar resvd[5];
> uchar sectors;
> };
> + added feature negotiation command so initiator can negitiate with
> target using of that improvement without losing backward compatibility
>
>
------------------------------------------------------------------------------
HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions
Find What Matters Most in Your Big Data with HPCC Systems
Open Source. Fast. Scalable. Simple. Ideal for Dirty Data.
Leverages Graph Analysis for Fast Processing & Easy Data Exploration
http://p.sf.net/sfu/hpccsystems
_______________________________________________
Aoetools-discuss mailing list
Aoetools-discuss@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/aoetools-discuss
