On Fri, Aug 24, 2018 at 1:19 PM David Gauthier <davegauthierpg@xxxxxxxxx> wrote:
Hi Everyone:I'm going to throw this internal customer request out for ideas, even though I think it's a bit crazy. I'm on the brink of telling him it's impractical and/or inadvisable. But maybe someone has a solution.
Reading below, I think you're right about it being inadvisable. It seems to me that your user, like most, just has "needs" and doesn't really want to worry about "details" or much of anything else. We have programmers where I would with this attitude: We list what we want -- you supply it and maintain it. Because we only code (we don't even design much because we're AGILE! ), we don't worry about those little details (availability, reliability, security -- not our concern!). One thing missing from your post is the OS involved. Linux? Windows? Other? Multiple different ones? E.g. some users are Linux while others are Windows. {ouch}.
He's writing a script/program that runs on a workstation and needs to write data to a DB. This process also sends work to a batch system on a server farm external to the workstation that will create multiple, parallel jobs/processes that also have to write to the DB as well. The workstation may have many of these jobs running at the same time. And there are 58 workstation which all have/use locally mounted disks for this work.
First question. You say DB. Do you need an SQL based data base. Or do you just need a shared data store which is easy to use in a script. Have you considered any NOSQL type data stores such as CouchDB, MongoDB
Second question, which scripting language and what programming language? Examples might be: Windows Powershell, Linux BASH, Windows BASH, Python, Perl, surely not some AWK variant, R, Google GO, Rush, C/C++, and so on.
At first blush, this is easy. Just create a DB on a server and have all those clients work with it. But he's also adamant about having the DB on the same server(s) that ran the script AND on the locally mounted disk. He said he doesn't want the overhead, dependencies and worries of anything like an external DB with a DBA, etc... . He also wants this to be fast.
So, if I understand, he wants a single shared data store for 58 clients. He also wants the disk holding the data to be "locally mounted" to every workstation. Depending on what "locally mounted" means to the user, I only way that I know of to do something like this is to have the actual disk / filesystem directly attached to a single server. All the workstations would need to use a communication protocol (IP?) to communicate their I/O to the "shared data" to this server to do the physical I/O. So we're talking some protocol like NFS or CIFS (Windows "share"). The easiest way that I know to do this sharing is to have a NAS box, such as NetApp, which really is a server+disk "in a box" and which implements these protocols in an easy to manage manner.
Or can each workstation have a separate, unrelated data store for its processing. The thought then would be some way to have the application write locally into its data store. Something else, perhaps "in line", would replicate the data to a central store on a server. That server would then distribute the changes back out to all 58 workstations and ??? servers so that each will have, eventually, an identical copy of the current data; but the stores might have inconsistencies until they synchronise.
My first thought was SQLite. Apparently, they now have some sort of multiple, concurrent write ability. But there's no way those batch jobs on remote machines are going to be able to get at the locally mounted disk on the workstation. So I dismissed that idea. Then I thought about having 58 PG installs, one per workstation, each serving all the jobs pertaining to that workstation. That could work. But 58 DB instances ? If he didn't like the ideal of one DBA, 58 can't be good. Still, the DB would be on the workstation which seems to be what he wants.I can't think of anything better. Does anyone have any ideas?
Yes, but the civil authorities would prosecute you if they found the body.
Thanks in Advance !
Between infinite and short there is a big difference. -- G.H. Gonnet
Maranatha! <><
John McKown
John McKown
