Re: Interactive task scheduling (system) ? How do I run a list of tasks consecutively on a server ?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 2022-02-04 09:04, Neal Becker wrote:
After this discussion, I needed a simple batch scheduling system.  I tried installing and starting condor on F35.  Never saw so many selinux problems.  Couldn't dnf remove it fast enough.

After a bit of searching, I found the system I wrote 11 years ago.  
https://pypi.org/project/batch-queue/

I just finished updating for py3 and a few more tweaks.

It's a very simple system that runs on the local host and allows you to submit jobs.  It will schedule them up to the #cpus.  There are commands to list the queue, kill jobs, suspend and continue them.  Does just what I need.

If it helps you too that's be great.

On Wed, Jan 26, 2022 at 2:04 PM Fred Erickson <fredferickson@xxxxxxxxx> wrote:
On Wed, 26 Jan 2022 12:59:23 -0500
Neal Becker <ndbecker2@xxxxxxxxx> wrote:

> I've needed this over the years but all the ones I've seen appeared
> much too complex for my simple use case.  I ended up writing my own
> using pyxmlrpc.  Unfortunately haven't used it for years and don't
> know if I could find it again (was uploaded to pypi at one time).
>
> Are any of these batch systems simple to install, use, and maintain?
>

I see batch was included with my f34 system and condor is provided in
the updates repo.


I'm confused.  Why do you feel the need for an overly-complicated job scheduler, emulating the mainframe scheduling mess?  Linux is a unix-like system.  Why don't you simply use the already-installed batch command?  It can easily handle thousands of simultaneously-scheduled batch jobs without bringing the system to its knees.

Unless your jobs are 100% cpu-bound, scheduling jobs by the number of cpus seems just wrong, leaving a lot of unused cpu cycles on the table.  If your jobs are i/o-bound, then disk or network load seems like it should be taken into account, not cpus.

Luckily, the overall system load is always calculated for you with no complicated mechanisms required. The batch command by default will not schedule a job if the load > 1.5 so that you do not impact foreground processes very much.  You can also renice your batch job as required to lessen the as-running impact even more.

If what you really need is a CI/CD system, then use the correct tools for the job.  Batch is generally not considered to be one of them.  Install Jenkins or CircleCI or any of a dozen tools that are built to do this right.

--

John Mellor

_______________________________________________
users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [EPEL Devel]     [Fedora Magazine]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Desktop]     [Fedora Fonts]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Fedora Sparc]     [Libvirt Users]     [Fedora ARM]

  Powered by Linux