Folks,
I tracked down the createrepo stalls to a fundamental bug in createrepo.
I have made the attached patch, a form of which will ultimately be
required for all Fedora architectures.
Currently deployed for production test on bahamas.
http://koji.fedoraproject.org/koji/taskinfo?taskID=4786005
Jon.
commit de9d6312e45d3283cae2ceffc11d50a45fe0dda7
Author: Jon Masters <jcm@xxxxxxxxxxxxxx>
Date: Thu Dec 13 04:37:27 2012 -0500
createrep: Fix pipe stalls caused by blocking IO in worker stdio monitoring
The existing logic in createrepo assumes that pipe-based IO of workers
is infinite without blocking. Unfortunately, neither of these are true.
Change the logic to have worker output collected during its run (rather
than after it has been terminated), and remove the blocking IO actions.
Incidentally improve interactive output of createrepo in the process.
Signed-off-by: Jon Masters <jcm@xxxxxxxxxxxxxx>
diff --git a/createrepo-fix-pipe-handling.patch b/createrepo-fix-pipe-handling.patch
new file mode 100644
index 0000000..6899c9e
--- /dev/null
+++ b/createrepo-fix-pipe-handling.patch
@@ -0,0 +1,42 @@
+diff -urNp createrepo-0.9.9_orig/createrepo/__init__.py createrepo-0.9.9/createrepo/__init__.py
+--- createrepo-0.9.9_orig/createrepo/__init__.py 2012-12-13 04:16:24.800994125 -0500
++++ createrepo-0.9.9/createrepo/__init__.py 2012-12-13 04:21:02.365980802 -0500
+@@ -26,6 +26,7 @@ import tempfile
+ import stat
+ import fcntl
+ import subprocess
++import select
+
+ from yum import misc, Errors
+ from yum.repoMDObject import RepoMD, RepoData
+@@ -662,15 +663,22 @@ class MetaDataGenerator:
+ while gimmebreak != len(worker_jobs.keys()):
+ gimmebreak = 0
+ for (num,job) in worker_jobs.items():
++
++ if job.poll() is None:
++ readyio = select.select([job.stdout.fileno(), job.stderr.fileno()], [], [])
++
++ if (job.stdout.fileno() in readyio[0]):
++ line = job.stdout.readline()
++ if line:
++ self.callback.log('Worker %s: %s' % (num, line.rstrip()))
++ if (job.stderr.fileno() in readyio[0]):
++ line = job.stderr.readline()
++ if line:
++ self.callback.errorlog('Worker %s: %s' % (num, line.rstrip()))
++
+ if job.poll() is not None:
+ gimmebreak+=1
+- line = job.stdout.readline()
+- if line:
+- self.callback.log('Worker %s: %s' % (num, line.rstrip()))
+- line = job.stderr.readline()
+- if line:
+- self.callback.errorlog('Worker %s: %s' % (num, line.rstrip()))
+-
++
+ for (num, job) in worker_jobs.items():
+ if job.returncode != 0:
+ msg = "Worker exited with non-zero value: %s. Fatal." % job.returncode
+Binary files createrepo-0.9.9_orig/createrepo/.__init__.py.swp and createrepo-0.9.9/createrepo/.__init__.py.swp differ
diff --git a/createrepo.spec b/createrepo.spec
index 2a7e7e3..ba92a03 100644
--- a/createrepo.spec
+++ b/createrepo.spec
@@ -3,12 +3,13 @@
Summary: Creates a common metadata repository
Name: createrepo
Version: 0.9.9
-Release: 11%{?dist}
+Release: 11%{?dist}.jcm1
License: GPLv2
Group: System Environment/Base
Source: %{name}-%{version}.tar.gz
Patch0: createrepo-head.patch
Patch1: ten-changelog-limit.patch
+Patch2: createrepo-fix-pipe-handling.patch
URL: http://createrepo.baseurl.org/
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
BuildArchitectures: noarch
@@ -24,6 +25,7 @@ packages.
%setup -q
%patch0 -p1
%patch1 -p0
+%patch2 -p1
%build
@@ -47,6 +49,9 @@ rm -rf $RPM_BUILD_ROOT
%{python_sitelib}/createrepo
%changelog
+* Thu Dec 13 2012 Jon Masters <jcm@xxxxxxxxxxxxxx> - 0.9.9-11.jcm1
+- Fix pipe stalls caused by blocking IO in worker stdio monitoring
+
* Thu Feb 16 2012 James Antill <james at fedoraproject.org> - 0.9.9-11
- update to latest head
- fix for lots of workers and not many rpms.
_______________________________________________
arm mailing list
arm@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/arm
