On Tue, Mar 09, 2010 at 09:46:04AM -0500, Andy Adamson wrote: > > On Mar 8, 2010, at 8:46 PM, J. Bruce Fields wrote: > >> The Linux server's reboot recovery code has long-standing >> architectural >> problems, fails to adhere to the specifications in some cases, and >> does >> not yet handle NFSv4.1 reboot recovery. An overhaul has been a >> long-standing todo. >> >> This is my attempt to state the problem and a rough solution. >> >> Requirements >> ^^^^^^^^^^^^ >> >> Requirements, as compared to current code: >> >> - Correctly implements the algorithm described in section 8.6.3 >> of rfc 3530, and eliminates known race conditions on recovery. >> - Does not attempt to manage files and directories directly from >> inside the kernel. >> - Supports RECLAIM_COMPLETE. >> >> Requirements, in more detail: >> >> A "server instance" is the lifetime from start to shutdown of a >> server; >> a reboot ends one server instance and starts another. Normally a >> server >> instance consists of a grace period followed by a period of normal >> operation. However, a server could go down before the grace period >> completes. Call a server instance that completes the grace period >> "full", and one that does not "partial". >> >> Call a client "active" if it holds unexpired state on the server. >> Then: >> >> - An NFSv4.0 client becomes active as soon as it succesfully >> performs its first OPEN_CONFIRM, or its first reclaim OPEN. >> - An NFSv4.1 client becomes active when it succesfully performs >> its first OPEN, or a RECLAIM_COMPLETE. > > RFC 5661 in section 18.51.3 > > Whenever a client establishes a new client ID and before it does the > first non-reclaim operation that obtains a lock, it MUST send a > RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no > locks to reclaim. If non-reclaim locking operations are done before > the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned. > > So there will never be a 'first OPEN' (except for an OPEN reclaim) > without a RECLAIM_COMPLETE. There will be in the case of an entirely new client, or a client that missed the grace period completely. (But I should have specified "first non-reclaim OPEN" in the 4.1 case, not just "first OPEN".) --b. > > >> - Active clients become inactive when they expire. (Or when >> they are revoked--but the Linux server does not currently >> support revocation.) >> - On startup all clients are initially inactive. >> >> On startup the server needs access to the list of clients which are >> permitted to reclaim state. That list is exactly the list of clients >> that were active at the end of the most recent full server instance. >> >> To maintain such a list, we need records to be stored in stable >> storage. >> Whenever a client changes from inactive to active, or active to >> inactive, stable storage must be updated, and until the update has >> completed the server must do nothing that acknowledges the new state. >> So: >> >> - When a new client becomes active, a record for that client >> must be created in stable storage before responding to the rpc >> in question (OPEN, OPEN_CONFIRM, or RECLAIM_COMPLETE). >> - When a client expires, the record must be removed (or >> otherwise marked expired) before responding to any requests >> for locks or other state which would conflict with state held >> by the expiring client. >> >> Updates must be made by upcalls to userspace; the kernel will not be >> directly involved in managing stable storage. The upcall interface >> should be extensible. >> >> The records must include the client owner name, to allow identifying >> clients on restart. The protocol allows client owner names to consist >> of up to 1024 bytes of binary data. (This is the client-supplied >> long form, not the server-generated shorthand clientid; co_ownerid for >> 4.1). >> >> Also desireable, but not absolutely required in the first >> implementation: >> >> - We should not take the state lock while waiting for records to >> be stored. (Doing so blocks all other stateful operations >> while we wait for disk.) >> - The server should be able to end the grace period early when >> the list of clients allowed to reclaim is empty, or when they >> are all 4.1 clients, after all have sent RECLAIM_COMPLETE. >> - Will allow pluggable methods for storage of reboot recovery >> records, as the NFSv2 and NFSv3 code currently does (in order >> to support high-availability). >> >> Possibly also desireable: >> >> - Record the principal that originally created the client, and >> whether it had EXCHGID4_FLAG_BIND_PRINC_STATEID (see rfc 5661 >> section 8.4.2.1). >> >> Draft design >> ^^^^^^^^^^^^ >> >> We will modify rpc.statd to handle to manage state in userspace. >> >> Previous prototype code from CITI will be considered as a starting >> point. >> >> Kernel<->user communication will use four files in the "nfsd" >> filesystem. All of them will use the encoding used for rpc cache >> upcalls and downcalls, which consist of whitespace-separated fields >> escaped as necessary to allow binary data. >> >> Three of them will be used for upcalls; statd reads request from them, >> and writes responses back: >> >> create_client: >> - given a client owner, returns an error. Does not return until >> a new record has safely been recorded on disk. >> >> grace_done: >> - request and reply are both empty; rpc.statd returns only after >> it has recorded to disk the fact that the grace period >> completed. >> >> expire_client: >> - given a client owner, replies with an empty reply. Replies >> only after it has recorded to disk the fact that the client >> has expired. >> >> One additional file will be used for a downcall: >> >> allow_client: >> - before starting the server, statd will open this file, write a >> newline-separated list of client owners permitted to recover, >> then close the file. If no clients are allowed to recover, it >> will still open and close the file. >> >> Statd will use the presence of these upcalls to determine whether the >> server supports the new recovery mechanism. nfsd may use rpc.statd's >> open of allow_client to decide whether userspace supports the new >> mechanism. Thus allows a mismatched kernel and userspace to still >> maintain reboot recovery records. >> >> In addition, we could support seamless reboot recovery across the >> transition to the new system by making statd convert between on-disk >> formats. However, for simplicity's sake we plan for the server to be >> refuse all reclaims on the first boot after the transition. >> >> By default, statd will store records as files in the directory >> /var/lib/nfs/v4clients. The file name will be a hash of the >> client_owner, and the contents will consist of two newline-separated >> fields: >> - The client owner, encoded as in the upcall. >> - A timestamp. >> >> More fields may be added in the future. >> >> Before starting the server, and writing to allow_client, statd will >> manage boot times and old clients using files in /var/lib/nfs: >> >> If boot_time exists: >> - It will be read, and the contents interpreted as an >> ascii-encoded unix time in seconds. >> - All client records older than that time will be removed. >> - The current boot_time will be recorded to >> new_boot_time (replacing any existing such file). >> - All remaining clients will be written to allow_client. >> If boot_time does not exist, an empty /var/lib/nfs/v4clients/ is >> created if necessary, but nothing else is done. >> >> Statd will then wait for create_client, expire_client, and grace_done >> calls. On grace_done, it will rename boot_time to old_boot_time, and >> new_boot_time to boot_time. >> >> --b. >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" >> in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
