Is there a limit on too much globbing? and SQL?
I've basically got a LONG process that repeats monthly. Every month, task 1 is created.
task 1 >> creates 20 task 2 >> each creates 1-20 task 3 etc >> each creates task 4 etc ..
If there are no task 4s, it checks task 3, etc - going up until all tasks are completed...
every 5 minutes, 1 task is completed.
ISSUE: One of these jobs is a 'jobs' task these are stored in the jobs directory path/.../data/*/*/Jobs/(jobs info here).txt
PART 1 QUESTION:
Instead of checking the 'jobs' directory one at a time (5 minutes each), I'm thinking of doing a GLOB - to locate the 'jobs' directory and create a task ONLY if the directory exists. instead of creating a jobs check each and every time.
If I create a GLOB to locate the jobs directory and create a task only if the directory exists - (fopen,fwrite - create task check file)
Will that be more beneficial? or how will I know it's too much/many?
Part 2: QUESTION
I've got a second routine to loop through and find 4 'jobs' tasks & update the database via MYSQL. If I improve the 1st part (as above) there could be a potential of 100's or 1,000's of job tasks to do...
currently, ive got a loop to process 4 'jobs' tasks at once
- find job tasks
- Get raw data
- update database SQL.
- delete task file
- sleep(1)
- repeat 4 times.
If I remove the sleep(1), could I process 100 at a time? More? - without breaking anything?
I've basically got a LONG process that repeats monthly. Every month, task 1 is created.
task 1 >> creates 20 task 2 >> each creates 1-20 task 3 etc >> each creates task 4 etc ..
If there are no task 4s, it checks task 3, etc - going up until all tasks are completed...
every 5 minutes, 1 task is completed.
ISSUE: One of these jobs is a 'jobs' task these are stored in the jobs directory path/.../data/*/*/Jobs/(jobs info here).txt
PART 1 QUESTION:
Instead of checking the 'jobs' directory one at a time (5 minutes each), I'm thinking of doing a GLOB - to locate the 'jobs' directory and create a task ONLY if the directory exists. instead of creating a jobs check each and every time.
If I create a GLOB to locate the jobs directory and create a task only if the directory exists - (fopen,fwrite - create task check file)
Will that be more beneficial? or how will I know it's too much/many?
Part 2: QUESTION
I've got a second routine to loop through and find 4 'jobs' tasks & update the database via MYSQL. If I improve the 1st part (as above) there could be a potential of 100's or 1,000's of job tasks to do...
currently, ive got a loop to process 4 'jobs' tasks at once
- find job tasks
- Get raw data
- update database SQL.
- delete task file
- sleep(1)
- repeat 4 times.
If I remove the sleep(1), could I process 100 at a time? More? - without breaking anything?
