On Sun, Jul 21, 2019, at 10:54 AM, Rene Veerman wrote:
> Hi.
>
> I like JSON a lot better than the SQL data model,
> but now that i've been using it for a few years,
> i find myself combining file system storage of lots of JSON files in
> many sub-folders into a web app,
> like news gathering with commandline PHP per N minutes from 280+ RSS
> online sources,
> into a PHP web-app that loads the latest news, via a web-powered PHP
> filesystem search of the latest new-items JSON files, stored on a
> filesystem thats got folders like
> ./YYYY/MM/DD/hh/mm/ss/CATEGORY_SUBMENU_PATH leading to JSON files.
>
> so now i want something more flexible. something that can give me a
> web-based treeview of data spanning many JSON-like files (in more
> complicated ways than my news app) and even into other file formats
> stored somewhere on the filesystem of some server.
>
> i plan to license this right from the start under a LGPL typed license,
> because i believe input from other developers is key to making a DB
> system that is not only easy to use, but also very efficient to run.
>
> i'd like the code to stay pure PHP, not even using the object oriented
> functionality of PHP, just plain properly named functions and their
> return values.
>
> i'll name this DB architecture FolderDB, and to prevent web-apps from
> forcing the server to reload (and decode!) many .jsone files upon a
> simple request to change something in some referenced .jsone file, i
> plan to run a single PHP server daemon that will respond to a yet to be
> written ASCII/UNICODE protocol.
>
> the file format itself, has to differ from JSON somewhat i think, and i
> also believe we can lose some of the overhead.
> i post this all here to hear your opinion, especially when it points to
> insurmountable problems related to my designs.
>
> i'm also interested to learn about how to add cloud redundancy to this
> DB architecture, preferably with tiered backup servers, or at least be
> prepared for such expansions of the DB architecture later on.
>
> here's the string based format that i came up with so far :
>
> JSON-Extended file format (ASCII / UNICODE)
>
> ------------------
> {
> key1 : s|string value,
> key2 : 2,
> key3 : 3.4,
> key4a : t,
> key4b : f,
> key5 : [
> numbered array
> ],
> key6 : {
> named array
> },
> key7 : url|/siteData/bla/bla/file.jsone,
> key8 : key|key7/someKeyInsideFileBehindURLofKey7_key1/key1a,
> key9 :
> url|/siteMedia/backgrounds/someFile.png||thumb|/siteMedia/backgrounds/thumbs/someFile.png,
> key10 :
> url|/siteData/bla2/bla2a/file.odt||thumb|/siteData/bla2/bla2a/file.odt.thumb.png||thumbGenerator|URL
> }
> ------------------
>
> key4a = true
> key4b = false
>
> key1 would need it's value escaped, using \| and \,
1) "i'd like the code to stay pure PHP, not even using the object oriented functionality of PHP, just plain properly named functions and their return values."
OK, right there you're already lost. There is zero reason to avoid OOP in modern PHP. Objects are, in modern PHP versions, *faster* and *vastly more memory efficient* than array structures. (cf https://steemit.com/php/@crell/php-use-associative-arrays-basically-never) It's also the API style most current developers will be looking for. So you're already setting yourself up for failure with that sentence.
2) "the file format itself, has to differ from JSON somewhat i think"
So you're not even using real-JSON, which means you can't use the C-based JSON parser and need to make your own in user-space. That's going to be dog slow compared to even just using pure JSON, and as Ashley noted modern DBs are blazing fast at searching/sorting/filtering behavior. Doing that in PHP will never be able to compete.
3) " i'm also interested to learn about how to add cloud redundancy to this DB architecture, preferably with tiered backup servers, or at least be prepared for such expansions of the DB architecture later on."
If you're using a file system as your primary storage, Git is the obvious solution for data replication. I've toyed with that approach before and while it wasn't a completed project I think it's promising.
But seriously, this is not a good approach at all. If you don't like SQL, there are other options that may fit your data model better. MongoDB is the obvious one (JSON native), but MySQL, MariaDB, PostgreSQL, and SQLite *all* have (slightly incompatible of course) JSON data types and query-ability. You can store and query JSON in an SQL database just fine, and it's going to be waaaaaay faster and waaaaaay more flexible than just using the file system, especially when you start mixing and matching JSON and relational concepts to leverage both in the places where they make the most sense.
--Larry Garfield
